kiddk1, that yellow piece is a restrictor plate used exclusively on Seimitsu sticks - if someone sold this to you as a Sanwa, you've been lied to.
The plate "restricts" the actual movement of the shaft:
using the top circular hole will make it an 8-way, the middle groove will
only allow 2-way movement, and the diamond shape at the bottom will make it a 4-way by making diagonals impossible.
Sanwas and Seimitsus use the same pin-out and are interchangeable,
so Orclord's post should apply to you as well.