Agreed, If you haven't, have yourself allergy tested.
That said, I wouldn't try latex gloves, as latex allergy is much more common and more severe than plastic allergy.
In a similar way, metals (especially inexpensive ones that could be plated onto a button) are also allergenic.
The possibility of a solvent or cleaner causing this is very real, so I'd check that out also.
Plastics are so common in our world, do you also have problems typing on a computer keyboard, driving your car (possibly leather coated wheel though), writing with pens (plastic), using a mouse, etc.?
I think you may find that it's something other than the plastic itself... look for anything else that causes similar problems, and strongly consider allergy testing if it's not been done.