You're probably over engineering this problem.
I'm not an expert on this so take it with a grain of salt. My numbers might be way off too. I don't have my aluminum notes right here with me. I also didn't confirm any of this with the local metal shop yet.
Based on my research on aluminum you can replace sheet steel with the equivalent sheet aluminum that's about 40% thicker. In other words, if you decided to use sheet steel that's .18" thick, replace it with 1/4" thick aluminum. You lose in thickness, but aluminum is something like 30% lighter than steel. But since you intend to sandwich aluminum and steel or wood, you might actually have a net gain in thickness, instead of a loss.
It's just a rule of thumb I guess. I think it does depend on the grade steel vs grade aluminum. For instance, I guess you can get away with using 5052 aluminum but you'll need thicker 3003 and really really thick 1100 to replace basic steel.
Point being, if you select the right aluminum. You won't need to back it with steel or wood.
That said and with brushed aluminum. I would go with plastic washers if they go on top or move to washers underneath if you want metals. Either way, you're going to get some hefty wear. Either on the washer itself or on the panel.