Originally Posted by M.C
I have both a 3DS and a Vita, so I have a good understanding of both. (long post ahead.)
The Vita is a fantastic handheld... on a technical level. The screen is beautiful, it makes PSone classics look fantastic cause of the smaller resolution (they stretch really wide on PS3/HDTV), it has very neat features that definitely beat out the 3DS, just in general it's better than the 3DS on every single technical level. It also can be used with your PS3/PS4 for cross-game play.
The problem is it only has 4 games on the whole system that are any good. You have Uncharted: Golden Abyss, Persona 4 Golden, Muramasa Rebirth, and Soul Sacrifice. The future of the Vita looks kinda good, you have a few solid games coming out for it, but overall it is lacking a lot on games. It can play PSP games, but it lacks the UMD drive (nowhere to put the PSP physical games), so you have to do it via downloading games onto a card or some other method, pain in the ass.
The 3DS on the other hand while technically inferior in most areas and the only thing it has going for it on that side is 3D (which is cool btw), has a ridiculous amount of games and honestly I would say at this very moment is the single best console a person can own if you are looking for great games.
Literally off the top of my head - Luigi's Ghost Mansion, Fire Emblem: Awakening (best game this year on any platform so far, IMO), Kingdom Hearts 3D, Animal Crossing: A New Leaf, Donkey Kong Country Returns 3D, Ocarina of Time 3D, StarFox 3D, Monster Hunter 3 Ultimate, Kid Icarus, Paper Mario Sticker Star, Zero Escape, Super Mario 3D Land, Mario Kart 7, Project X Zone, Tales of the Abyss, Professor Layton, Etrian Odyssey IV, Castlevania: Lords of Shadow - Mirror of Fate, Shin Megami Tensei: Devil Summoner: Soul Hackers, THEATRHYTHM Final Fantasy, Super Street Fighter IV 3D edition, etc.
Games coming out for the 3DS soon - Zelda: A Link Between Worlds (A Link to the Past sequel), Bravely Default, Smash Bros 3D, Shin Megami Tensei IV, Phoenix Wright: Ace Attorney - Dual Destinies, Pokemon X/Y, so many great games.
It is also 100% backwards compatible with the DS, you can fit physical DS games into the same slot, and the DS library is mind-blowingly amazing on its own, so you get two extremely full libraries of great games in one system.
So... in my opinion if you have to choose one the 3DS is easily the best choice, in fact if you were saying which is better a PS3 or a 3DS I'd probably still say 3DS, there's just so many damn great games on it.
Still, a Vita is great to have for the few games it does have and if you are huge into PS1 classics, it can play all PS1 games in a better resolution than on your TV so they look a lot better, and in the future you can bet it you will find it worth the purchase price.
I'm glad I own both, but the 3DS is superior than any console on the market right now IMO, I'd own that over a PS3 or 360 or Wii U or anything right now.
edit - if you are going to get a 3DS, make sure you get the 3DS XL. It's a larger 3DS so the screen is bigger and it's far more comfortable to hold. It's not as portable cause it's larger but if you don't have the hands of an 11 year old boy, you will want the bigger XL.
Long post but I wanted to go into some details for you.