Caroll Spinney spoke of an actual cab driver in real life, who pretty much resembled stereotypical cab drivers of the day, in appearance and mannerisms. The cab driver had a cigar in one corner of his mouth, and spoke out of the other in a gruff voice, which he based Oscar's voice on; this guy is obviously an actor.
Admittedly, that guy's voice does sound kind of like Caroll's earlier Oscar voice. Reminds me of a bit from Homeward Bound: The Incredible Journey, where Sassy sneaks through the dog pound to bail out Shadow - various dogs are making cat-calls at her, one of which says "Heh, nice whiskers, baby!", who I thought sounded a lot like Oscar as well.