I'm actually in the planning stages of swappable panels myself, and asked a similiar question of myself on the topic of 4-way panels.
Looking through MAWS, there is a number of 4-way games that used 4 buttons, not a huge number, but a few. So I'm going with 4 per player.
My plan is though, in the center I'm going to place a diagonally rotated 4-way with a single button, to give me a Q-bert/Congo-Bongo joystick without configuration hassals.
There are also a few dual 4-way games, but I've decided there are so few that I'll use that 8-way that switches to 4-way from the top on my 8-way panel and handle them.
Hope this is of interest/relevance!