You can remove the 'pin', but it can be a pain.
The paint often comes off because its not really bonding to the metal (or more specifically the coating on the metal that was there first). You need to sand them down first (light sand blast if you can, but scuffing with 80 - 150 grit sand paper works too) to give the primer something to stick too, coat them with an etching primer (so it'll "bite" in), then paint them with a good paint made for metal. I often use Rustoleum rattle cans, but it takes a long time to dry / cure.
Basically, the factory coating on that hinge is not letting anything stick to it.