I'm assuming that you are talking about the two pins that go through perpendicular to each other.
The other two (one just above those two, and one just below them that interfaces with the grommet) are permanent.
The top one keeps the wires headed out the wire hole, and the bottom one keeps the stick from spinning on the grommet.
These two don't connect the two pieces together, so there is no need to remove them.
That said, once you get the two middle ones OUT with a pin punch, you can replace them with hitch pins, or cotter pins.
That's what I've done on most of mine.
That makes them easy to get apart the next time.
The only issue I've run into with that is that occasionally you will have alignment problems with the holes on the two pieces, if you swap the shafts out between sticks.
I've always been able to get at least 1/2 of a cotter pin into the misaligned holes though.