Just to check, you still have the numbers 5 and 6 registered in Mame as your coin buttons? So you want when you press joystick button 17 or 20 to pretend to be keyboard numbers 5 or 6.
Try changing Joy17 to 1Joy17, that assumes it is the first joystick attached to system, and if that doesn't work keep changing the number one at a time until it does work, then do the same for other button.
You could try the Joystick test script to help figure out joystick number and button number
https://lexikos.github.io/v2/docs/scripts/index.htm#JoystickTestScroll down the page a bit and find the correct script for checking joysticks.
Good luck
Ps just checked this on mine and it work
In Mame coin input must be 5 and 6 from keyboard NOT your joystick buttons
Script needs to be as below - but with YOUR joystick button numbers and whatever other changes you want/need to make
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn ; Enable warnings to assist with detecting common errors.
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
#SingleInstance Force
key5 := 0
key6 := 0
~esc::
key5 := 0
key6 := 0
return
1joy9::
key5++
if key5 <= 5
{
Send {5 down}
Sleep, 50
Send {5 up}
}
return
7joy9::
key6++
if key6 <= 5
{
Send {6 down}
Sleep, 50
Send {6 up}
}
return