So I guess I would only need like 6-8 buttons on player 1 & 2 for fighting games and only 4 buttons for Players 3 & 4?
Not trying to talk you into any particular number of buttons, but some points to consider:
No "normal" game in MAME uses more than 6 buttons. (you have mahjong panels and the Thayer's quest keyboard, etc)
When you map globally in MAME, it assumes you have a 6 button layout arranged as so:
123
456
Stick with that, and 99% of the individual games will be mapped correctly.
If you do add an extra button or two, keep those core 6 in that arrangement:
1237
4568
All
current PC fighting games can be played with 6 buttons. The two extra are always post a combination of the others.
Some modern PC games like Castle Crashers have every little action mapped to a different button & require 8 (might be able to get by with 7).
I've found that having to remember the function of 8 different buttons annoys me to the point of not playing these games.
Some people like a 7th button so they can have the Neo Geo layout of 4 in a row.
I did this (pretty much had to as my cab is Neo Geo themed), but found that I prefer to use the top three in the first row and then hit the first one in the second row with my thumb for the D button.
In MAME, the only 4 player games that use 4 are the few (couple?) Dungeons and Dragons games. The rest of the 4 player games only use 3.
Outside of MAME, the only arcade game to use more than 4 buttons for player 3 and 4 is Guilty Gear Isuka which had 5 buttons per player.
It is playable on Demul emulator if you have a good enough PC and graphics card. The game is the weakest of the series and not worth adding an extra button for IMO though.
Some people add the extra buttons to player 3 and 4 citing console games like Super Smash Bros, but I'm unsure how many those games actually require.