You are probably hitting some combination involving CTRL or ALT
First I'd try disabling hot keys in windows, but if those keys have some function in Nestopia, it won't help.
If possible, change your encoder to not use those keys.
Why those are included as default keys and people continue to use them is beyond me.