I'm guessing that you are using Windows XP and an older Groovymame build, because you're using a HD4350 card ("legacy") and talking about d3d9? Getting the HD4XXX cards working properly under Win7 is not easy, if you've managed that then well done.
Later mame builds (which GM is based on) don't support XP, and 32bit platforms generally, and require 3D rendering capabilities beyond d3d9 (which is as far as XP goes with d3d).
I'm using a CRT arcade monitor with an ati 4350 connected VGA to scart rgb bypass trought the JAMMA pins, the ps1 uses an official sony RGB to Scart cable if it matters.
Most arcade monitors will require a video amp, otherwise the picture will be dim. Surprised you don't have one, but the pics look bright enough anyway.
I just booted GM_0.249 and see that Tekken 3 reports imperfect emulation of graphics. You won't see this if you have the "info" or "nag" screens switched off (and this I why I leave mine switched on). Guessing this hasn't changed yet in latest versions.
The infor/nag screens also report Tekken3 (arcade version) as being "640x480". Even though Butters advises us this was 480p (progressive) originally, it would be displayed, on your 15khz arcade monitor, in 640x480i (interlaced). This is the best it can do with that version of the game. Your PS1 probably displays Tekken3 at 720x480i (NTSC standard), because it was designed for 15khz NTSC TVs/displays, which means you'll get a few more horizontal pixels. However, you are comparing apples and oranges as the games are actually different (arcade vs PS1), having been designed for different platforms and display technologies.
An extra fact is that when I change from d3d9 to open gl or gdi the image looks even a little more pixelated and has a small flicker... This problem is with all games, some more other less but still noticiable
Anyone experienced the same?
Help or some tip is appreciated
I think that, over time and different versions, the devs/pro users etc. are pretty consistent about recommending d3d before open_gl or gdi rendering, except in special cases or bug-testing. Search the site and see for yourself (you can start with the GM release threads pinned at top of Groovymame forum branch).
GM (and mame) have made many advancements, and various changes to video rendering from time-to-time, responding to developments and bugs encountered. To compare meaningfully, you will need to use the latest versions on appropriate platforms.