Are you sure you're talking about the Sesame Street Treasury and not the Sesame Street Library? The difference is simple, in the following respects other than just the titles.
1 The Sesame Street Library was divided up into fifteen volumes, with the last three dedicated to featuring their respective numbers.
2 The books in the Sesame Street Library featured hand-drawn images of the characters, the Sesame Street sign at the top of the book, and the volume number in a little square at the top corner of the book above the sign.
3 The books in the Sesame Street Library featured a combination of letters and numbers, not just one each; and had stories for both the letters and numbers. The volumes were the following:
Volume 1: Featuring the Letters A and B, and the Number 1. Big Bird holding block letters A and B and block number 1.
Volume 2: Featuring the Letters C and D, and the Number 2. Ernie holding the letters, while Bert drove the bicycle built for two and the number shows up on the front plate.
Volume 3: Featuring the Letters E and F and the Number 3. Cookie Monster resting atop two blocks showing the letters, and munching on a cookie number 3.
Volume 4: Featuring the Letters G, H and I and the Number 4. Grover's piloting a plane with the number on the tail and the letters made out of clouds before him.
Volume 5: Featuring the Letters J and K and the Number 5. Big Bird, Ernie, Betty Lou, Cookie Monster, and Bert performing as the Sesame Street 5 with the number or band logo on Ernie's drums and the letters on the side of Bert's piano.
Volume 6: Featuring the Letters L and M and the Number 6. Sherlock Hemlock's examining a few Twiddlebugs who hold up signs with the appropriate letters and/or number.
Volume 7: Featuring the Letters N, O and P and the Number 7. The Count's playing hopscotch on numbered squares with the 7 lighted up, as a few bats watch from above a fence post where the letters are painted.
Volume 8: Featuring the Letters Q and R and the Number 8. The Amazing Mumford pulls the letters out of his hat, while a rabbit holds the number below the table.
Volume 9: Featuring the Letter S and the Number 9. Oscar, as an artist, holds two paintings he made; one of a letter S made of swirling paint in all the colors of the rainbow, and a 9 made out of pieces of trash.
Volume 10: Featuring the Letter T and the Number 10. Grover wearing a letter T on his shirt jogs by a Twiddlebug lemonade stand selling drinks for 10 cents a glass.
Volume 11: Featuring the Letters U and V and the Number 11. Herry Monster, Frazzle, and a few monsters are fishing in a leaky boat marked by the number whereas Herry catches the letters on his line.
Volume 12: Featuring the Letters W, X, Y and Z and the Number 12. Prairie Dawn, Guy Smiley, Bert and Ernie are taking a train ride on their way to the letter-marked signposts, with the number on the front of the train.
Volume 13: Featuring the Number 13. Bert and Ernie racing in a go-cart downhill, with the number painted on the cart's side.
Volume 14: Featuring the Number 14. Little Bird, perched at Big Bird's feet, waves a banner with the number.
Volume 15: Featuring the Number 15. Super Grover flies by holding a weight marked by the number.
The only time I saw the books from the Sesame Street Library was about ten years ago, during my last trip to Baltimore for an eye examination. Back then, my sight was still better to the point that I could discern images on book covers and read text (no more nowadays though.) The thing that made these stand out was the fact that their covers featured the actual Muppet characters from Sesame with some sort of item for the letter featured in the volume, which wasn't announced right off the bat. The other thing was the rully nice inside cover art showing the characters and letters and number in a sort of spreadsheet fashion. There were only seven books back then, and I haven't seen them since.
Sorry for the long reply, but hope this helps and you have a good night.