Nope, not swappable. One large 4 player unit.
Depending on the Velcro and storage of the swappable cps, would really dictate the longevity of the velcro.
I wouldn't recommend getting some of the cheapo velcro that you find in 1 inch strips but look for at leat 2-3" wide stuff.
Look at it this way, if it wears out, you can always replace it.
Although, if you're talking about just swapping out individual joys rather than a fully build CP, you might not find the solution very viable since there would be some cushion to the board in that scenario.