You almost got it, but a little backwards. The part you changed is telling mame what player in the game. So your change told mame about the game's player 3 inputs. But your clip didn't give a new input even for player3.
If you have your 4-way wired to mame's default player 3 keyboard buttons like it sounds:
<port type="P1_JOYSTICK_UP" mask="1" index="1" defvalue="1">
<newseq type="standard">
KEYCODE_UP OR KEYCODE_I </newseq>
</port>
You want to change the bold part (or add it with the lines above and below if not existing yet). The above tells mame that for pacman's player 1 input for the game's up dirrection, use either the up key or the I key. You can put it to whatever you want; I gave the above example to show the OR, but you might just want KEYCODE_I.
Or you can start the game, tab, input (this game), up/down with player 1 stick to the input you want to change, enter, then move the 4-way stick in that direction. Repeat for each direction.
Things get a little confusing with all the different things "joystick 3" can mean. There's the original game's joystick 3, the default keys mapped to that game's joystick 3, windows third joystick installed, and in your case the encoder's "joystick 3". Add that mame can map any input to (basically) any key, and most keybaord encoders can be programmed to send different than default values for each port. Usually they all line up, but when you want to use the joystick second from the left on the control panel (that 4-way stick) that's wired to the encoder's joy3 to player 1's inputs...
Even the second method I gave is a simplification that works with the default settings, but since they can be changed it doesn't always work.