I can write more later (need to get to work), but basically for any key that you want to have double-duty (shifted and non-shifted behavior), you need to specifically map that input in one case to both buttons, and in the other case to one button and not the other one. Here's an example:
<?xml version="1.0"?>
<!-- This file is autogenerated; comments and unknown tags will be stripped -->
<mameconfig version="10">
<system name="pacman">
<counters>
<coins index="0" number="71" />
</counters>
<input>
<port tag="IN0" type="COIN1" mask="32" defvalue="32">
<newseq type="standard">
JOYCODE_1_BUTTON3 NOT JOYCODE_1_BUTTON2
</newseq>
</port>
<port tag="IN1" type="START1" mask="32" defvalue="32">
<newseq type="standard">
JOYCODE_1_BUTTON2 JOYCODE_1_BUTTON3
</newseq>
</port>
</input>
</system>
</mameconfig>
In this case, JOYCODE_1_BUTTON3 by itself is START1, where JOYCODE_1_BUTTON3 with JOYCODE_1_BUTTON2 held down is COIN1.
I can write up a tutorial later on how to set this up.
To set it up the "not" for anything keyboard/keyboard encoder based, all you have to do is hit the key that you want to be "notted" twice. Unfortunately for joypad devices like the Ultrastik 360, this method doesn't work. I manually changed my pacman.ini above to test this.
YAY! I can use my U360 as a button encoder without losing this functionality and I can continue to be lazy and write no code!