I would recommend the GP2X for "on the go" emulation. It runs everything and is very easy to code for. NES games in particular run incredibly well with the emulator I'm using (believe its a port of fceux). It's a bit pricer, but it can do so much more. PSP emulation is nice, provided that you can find one for on the cheap, but for whatever reason games don't look very good on that screen. Wrong ratio I think.
Its been discontinued though so keep in mind that prices may rise/fall as well as some vendors no longer carrying the unit.
On the subject of ease of development I should point out that despite having easier tools to dev with the psp is far more widespread. Its easier for things to get ported over to the psp because its more likely for it to be used more. At least that's how I see it.
Even though I have other options I still find myself wanting to get one. It seems so interesting and its potential makes it seem like getting one now would save me the headache later if only for the fact that it suffers from the same homebrew shortcomings of the gp2x. The most developed software for handhelds are ports and applications to simulate a clock.
If you are in the market for a handheld for homebrew check out the Game Park touchscreen unit. I don't think its out yet and the price is higher but it will probably have a better user base than this, maybe.