This was a great discussion/brainstorming of Modular panel ideas. I'd love to hear more about this stuff as it's the direction I'm leaning for my first project. I've seen Doc's panel and I think I like that one the best although my concern is the panel widths, dowel holes and dowels all have to be so precise otherwise you get gaps or other problems. So, I'm still looking for another idea.
I shared many of the same concerns you had when I started. In fact, I almost stopped several times and went back to my static panel because I thought this whole thing would never work. I also thought I was never going to be able to secure the panels without mounting screws. However, once I had the dowel rails installed and fit the first 5-6 panels into the rails, I realized that this crazy idea actually might work. It did.
I have come up with a modification that can make the modular panels easier to build. If you put a front dowel rail in that is perpendicular to the panels (just like the back one is) it makes it much easier to drill the panels since all panel holes are now vertical into the panel.
On my original case, I could not implement this idea because I had pre-existing pinball buttons that interfered.
As for the rest, you do need access to a table saw to cut the panels, but once you set the table saw up for a given panel size, you can run them all through in a few minutes. If you can borrow a table saw and mark the wood in advance you could probably cut all of the panels in an hour or two at most. Obviously you cut the panel just shy of the desired width (say 1/16" less than the desired width).
I know that the holes can be cut without a drill press, since I had to do so for several of the larger panels (my tiny drill press was not deep enough). The key here is to carefully measure the holes and then start with your smallest drill size to drill an accurate pilot hole. Then move to a medium one, and finally the full drill size. Use some tape wrapped around the drill bits to make sure you don't drill too deep.
Fitting the panels is also worth some discussion. The vast majority of the panels don't fit perfectly the first time you pop them in. I used a pencil to mark the holes that did not match well and then a drill bit to auger out the holes a bit until I arrived at a good fit. It takes a little bit of patience at first, but by the end I could fit a panel very quickly with no slop.
I'm sure there are many other ways to do this - (there are 5-6 other examples cited in this thread) but I wanted wood panels that I could make in the garage and remove from the system with no tools. Hopefully someone can take the basic idea and simplify it even further.