The biggest concern is to make sure they are mounted properly. If the clamp parts aren't located properly, then you won't get good friction and that's what holds the panel in place.
Using more than two would make the panel technically more solid, but in practice, 2 should suffice. It really depends on the size and construction of the panel. Some panels, like the one on my Millipede cabinet, are metal and hinged, while my Defender cabinet panel is wood and completely removable. Both use only two clamps, but the Defender panel is small and "captured" on 3 sides.
So my suggestion when building a panel that is captured on one or fewer sides is to use some narrow framing to capture the panel and prevent lateral movement, then rely on the clamps only to hold it down. Some thin weather stripping or rubber securely affixed to the surface you are pulling your panel into contact with may also help to keep it from moving.
RandyT