You can get little Atom or AMD E-350 type boards that are passively cooled and pretty cheap. Slap an SSD on them (some boards even have provisions for mSATA cards), and they have no moving parts. For Windows, enable enhanced write filter (EWF) or, for Linux, use either a flat-out read-only root filesystem or use a copy-on-write overlay (which is how EWF works), and you can safely hard power the system at will. If you want to save high scores, savestates, and other preferences, you'll have to redirect those to a read/write partition which means you may, at some point, lose that data due to a power loss though it's unlikely with modern filesystems (worst case is that you should end up with old data).
Obviously this is a bit more DIY than something like a 60-in-1, but it comes with full flexibility that people expect from a traditional PC home arcade setup. They're probably capable of running higher end stuff that those low-end setups, too, but don't expect to run the late-model 3d accelerated stuff on such a puny CPU.