I would go with something simple, a Nice colour fade that compliments the blue would be good. 1 or 2 characters would be fine I think but no more.
4 buttons for P3 & P4 is good for console games, If it's just mame then only about 3 games need a 4th button. Cardboard mockup's are definitely the way to go as you can try out a number of things.
As an aside do you think that it will be often you will be playing 4 player games, If not then I would look at having a 2Player setup with trackball, spinner and possibly a dedicated 4way joystick.
Anyway keep us all updated as you progress.