Paintball is the most fun ever. You can get a pump action gun with 2 C02 cartridges (approximately forty shots a cartridge) and a mask for about $20 - $35 at K-Mart or a sport's store. Then just buy a box of 20 C02 cartridges for like $5. So you pump the gun, fire one round and then pump again to fire another.
The other way to go is get an automatic (very very expensive) or semi-automatic gun (usually start between $30 - $70) which requires an air tank (should be included) which needs to be filled at a retailer. Usually stores that sell air canisters for paintball guns fill them too. You can get anywhere from 70 to 500 shots from a canister depending on how big of a tank you buy. Automatic means you just hold down the trigger and the gun'll unload paintballs one after the other. Semi-automatic means you just pull the trigger for every ball you shoot.
Your best bet is to get the pump guns if you decide to go run around in the woods or something because otherwise you'd have to go back to the store to get your tanks refilled or if you were at a paintball arena, back to the pro shop.
Otherwise; to just go in the woods and play you'll need:
• Pump action paintball gun w/ mask - $20
• Additional CO2 cartridges - $5
• Large box of paintballs - $40 (you can get the cheap painballs, so that's like 3000 for that price. They may break in your gun so you might have to clean them out every once in awhile to clean the barrel of the gun)
• Barrel cleaner - (sometimes inluded w/ pump gun) $3
• Painball holster canister(s) - $8 (This is to hold your spare paintballs while you're running around. It's a hard plastic tube you clip on your belt. You could also just stick 'em in a bag and carry 'em around, but you risk breaking them or just run back and fourth to the box every time you need to reload)
• Hopper - $6 (A hopper is what holds your rounds. Most pump guns come with a hopper that'll hold ten rounds/paintballs. If you're trigger happy like most beginners, you'll empty that real fast, so you may want to pick up a hopper that'll hold like a hundred or so balls)
If you get the pump, one little tid bit you should know is that after you use up all the air in a C02 cartridge, be careful when you take it out because it'll be colder then ice. And speaking of cold, if you play this time of year, the cold air will harden the paintballs so when you get hit it's gonna' hurt more!
EDIT - As far as where to buy this stuff, if you have a flea market around you, check out there first. You can get better deals and the staff is usually more inclined to give you some tips.