Ok, lets assume and simplify the issue by saying you have 2 buttons per player.
First, grab NoNameMame. He incorporated my hacks. If you don't want to use that you are going to have to compile your own mame.
ok, in thi diagram players are labelled, from left to right, player 1, player 3, player 4, player 2.
P1J would be the player 1joystick, P1B1 is player 1 button 1.
|----------------------------|
P1J | | P2B2
| |
P1B1 | | P2B1
| |
P1B2 | | P2J
|----------------------------|
P3J P3B1 P3B2 P4J P4B1 P4B2
Now let's assume the buttons are assigned to these keys:
P1B1 KEYCODE_A
P1B2 KEYCODE_S
P2B1 KEYCODE_Z
P2B2 KEYCODE_X
P3B1 KEYCODE_D
P3B2 KEYCODE_F
P4B1 KEYCODE_C
P4B2 KEYCODE_V
Note these are the phyisical representations on how they are hooked up. Not how we will be using them in mame. you actually gotta think of the buttons on your cabinet by their keycode, not by which is player 1 button 1.
If you don't know how to use ctrlr files is the quick and dirty way of doing this. To learn more about ctrlr files read the doc folder in mame.
Since you will be using my hack (or NoNameMame which has my hack) you will do this:
1) in the ctrlr folder of mame make a folder called cabinet
2) in the mame.ini file edit these two lines
ctrlr cabinet
ctrlr_directory ctrlr (make sure this is pointing to the mame/ctrlr folder)
3) in the cabinet folder you just made create a horizont.ini and vertical.ini. (actually you could create a default.ini which will be the default vertical setup and a horizont.ini also)
4) vertical.ini will look like this (at least for the buttons, put in joysticks simularly)
P1_BUTTON1 "KEYCODE_A"
P1_BUTTON2 "KEYCODE_S"
P2_BUTTON1 "KEYCODE_Z"
P2_BUTTON2 "KEYCODE_X"
P3_BUTTON1 "KEYCODE_D"
P3_BUTTON2 "KEYCODE_F"
P4_BUTTON1 "KEYCODE_C"
P4_BUTTON2 "KEYCODE_V"
4) horizont.ini will look like this (at least for the buttons, put in joysticks simularly)
P1_BUTTON1 "KEYCODE_D"
P1_BUTTON2 "KEYCODE_F"
P2_BUTTON1 "KEYCODE_C"
P2_BUTTON2 "KEYCODE_V"
I assume we don't need to list player 3 and 4 here as they probably won't be used in this orientation.
Now, whenever you play a horizontal game the horizontal side will be mapped to player 1 and player 2. When you play vertical the vertical sides will be player 1 and player 2 while the horizonal side will be player 3 and player 4.