Edit: okay I downloaded another mame version ' MAME 0.149 NoNag, Hiscores, CAVE & DirectInput' and now I don't get that error message so problem solved, post is being left here to help other people in case they get this error message and cannot fix it.
I get
'unable to create the direct3D device'
in mame when starting up a game, however this is only when I am using windowed borderless gaming application to make the game run in a borderless window, if I don't enable the program it wont do that.
so I am basically looking for a way to ignore the message, or a better way to solve it, I have tried changing video types but that leads to really bad performance although the error message is gone if I do that.