Put a small jeweler's screwdriver in and turn it to pry the e-clip open.
Yes! This is the way.
I used to use a pair of pliers to push the e-clip off, and catch it in a box so it wouldn't scoot under a workbench or otherwise disappear. It worked well, but was complicated and annoying. Then I looked closely at the clip and noticed those two nice little slots between the clip and the joystick handle. Game over -- a little twist with a jeweler's screwdriver, and it's off and under control.
Wish I had noticed that a year ago.

As for removing a quick disconnect, make sure you get the pliers' jaws down low enough to pull the entire disconnect off the terminal -- not just the wire or the colored sleeve. Grab it too high, pull hard, and you might tear the thing apart. Then you're right back at square one, but with a broken disconnect!