Whelp, nevermind, figured it out!
When mapping this control using Retroarch, I mapped the directions up, down, left right and used trigger and thumb buttons as A and B.
There are no other buttons on this joystick, so I could not map anything for the hotkey.
Turns out, when you play a game and hit the A button without a hotkey, it was mapped to exit the game.
I manually mapped the hotkey button in the .cfg file to a random number, and that seems to have fixed it.