Unexplained Lag testing result

[jimmer]

I've been doing some lag testing on my Dell 1913 TN panel.  System one is JROK FPGA hardware emulation. System 2 is  groovymame171 with framedelay 9.  The game is Defender and I'm testing the shot response.

I'm recording video at 120fps and I've got multiple LEDs mounted vertically  to give a better estimate of when the button was pressed.  I count in 'half-frames' of 8ms and then adjust. I've counted over 300 groovymame shots and over 100 JROK shots.

I take readings at two ship positions: High (approx 1/4 down the screen) and Low (approx 3/4 down the screen). So these positions are 8ms apart in a 17ms scanout.

The average result for JROK is:                High = 30ms, Low = 29ms
The average result for groovymame is:     High = 44ms, Low = 35ms

On original hardware Defender polls inputs every 8ms and the expected response time is the same in both halves of the screen, so the JROK result is as I would expect (but maybe a bit slower than I expected)

For groovymame, I was expecting it to be faster than JROK at the top of the screen and slower at the bottom. Looking at the results above I would expect the groovymame High result to be 27ms (35-8) and I would have been happy with that.

So... That's the odd result. And now I'm trying to explain it (and then maybe fix it / or improve the emulation). I'm working my way through the MAME documentation and trying to understand the code but it's all new to me so this could be a long project. 

I'm mentioning it here in the hopes of finding someone knowledgeable and interested enough to help me out. If I manage to work it out myself I'm sure that will be fulfilling but I would be even happier if someone gave me a solution tomorrow

[1500points]

I doubt here is where you are going to find the precise mamedev info you need since sean riddle is the expert that has done all the wms mame work since aaron giles early work around 106 era. Sean didn’t think he could take mame any further after the mame 15x updates.  He said there would have to be core advancements in how mame itself functioned before it could move forward so he ended his exploration.

You know his email address.


Sent from my iPhone using Tapatalk

[Calamity]

@jimmer,

I find your results logical. Anyway, before any further conclusions, I think you should test Defender with the frameslice implementation, here is a build based on 0.195.

Usage: mame64 defender -frame_slice 9 -vsync_offset 18 -priority 1

The -vsync_offset value needs to be found manually so that the emulation is stable. Press f11 to see the bands, they shoudn't jump around. Frame slice 9 should be fine if your system is decent enough, otherwise try a lower value.

[jimmer]

I doubt here is where you are going to find the precise mamedev info you need since sean riddle is the expert that has done all the wms mame work since aaron giles early work around 106 era. Sean didn’t think he could take mame any further after the mame 15x updates.  He said there would have to be core advancements in how mame itself functioned before it could move forward so he ended his exploration.

You know his email address.

Yeah, Sean is always my first stop   But he's not done anything on MAME Defender and says he doesn't know the MAME code well.  But he had a quick look and couldn't see anything to support one of my speculations.

[jimmer]

@jimmer,

I find your results logical. Anyway, before any further conclusions, I think you should test Defender with the frameslice implementation, here is a build based on 0.195.

Usage: mame64 defender -frame_slice 9 -vsync_offset 18 -priority 1

The -vsync_offset value needs to be found manually so that the emulation is stable. Press f11 to see the bands, they shoudn't jump around. Frame slice 9 should be fine if your system is decent enough, otherwise try a lower value.

Good idea. I haven't been following this development but if there's a chance it might improve my defender response then I'll try it.

[jimmer]

I've just got it set up, and just about to take some video.

Fastest arcade PC I've got here is I3-2120 and it can only do  frame_slice 4 at 100%.  everything is default in my .ini except: monitor lcd.

Should I be able to get more slices?  my monitor is 1280x1024  and I see  super_width  2560 (I don't know what that is though)

[Calamity]

I'd say 4 sices is pretty good for that cpu. It's enough anyway to tell us if it makes any difference. If MAME's current emulation of Defender allows the machine to poll inputs in middle frame, then you should see an improvement. Otherwise, frame slice won't make a difference as compared to frame delay.

[jimmer]

First test. 60 shots spaced at 18.7ms  position= High (1/4 screen).   Average 36ms lag.  The max/min range is very slightly tighter than previous results ( min 20ms, max 54ms, SD=9ms).

The average result for JROK fpga is:            High = 30ms,  Low = 29ms
The average result for GM 171 FD9 is:         High = 44ms,  Low = 35ms
The average result for GM frameslice 4 is:    High = 36ms,  Low = ??


I haven't measured the low position yet, and I don't understand the frame slicing. But it looks promising.

[NaNo1982]

Hello!! lately I was testing the frame_slice option with CPS1, CPS2 and NEOGEO games, is there any difference with the frame_delay option? For example for NEOGEO games use that command line: -srf -frame_slice 5 -vsync_offset 14 garou, the imput lag is practically unperseptible, but I wanted to know if I'm doing something wrong, thank you very much! Calamity, you are the best !!

[jimmer]

I've counted some more shots. And also added 2ms to the low results (to adjust for recording method)

Updated:
average response JROK fpga is:            High = 30ms,  Low = 31ms
average response GM 171 FD9 is:         High = 44ms,  Low = 37ms
average response GM frameslice 4 is:    High = 37ms,  Low = 36ms

Still working on understanding how all this works (Defender machine, MAME, groovymame, frameslice).

I started out a few years ago happy that groovymame used the fast processing to mitigate against slow LCD response and hence get closer to CRT response. Then learning about the 120hz polling threw a spanner in the works because I thought MAME was tied to a 60hz poll/calculate/display cycle. But now I'm confident again that it's possible to beat eg the JROK LCD performance and hence get closer to true (CRT) response, and that things are moving in that direction.

I'm contemplating changing the Defender poll rate to 240hz (and maybe overclocking the cpu slightly to make up for the lost cpu cycles) on the assumption that would reduce input lag by average 4ms (4x 4ms polls vs 2x 8ms polls)

[Calamity]

Thanks for testing.

My understanding based on your results is that MAME's Defender emulation is accurate with regards to input->screen. At least it matches the description you wrote about somewhere. This can only be exposed by a feature like frame slicing (beam racing).

Frame delay and frame slice will provide similar results for frame-buffered systems, like the cps1, etc. Although it might improve input granularity even for frame-buffered systems (it's still not demonstrated this has an effect).

In systems that do sub-frame screen updates is where frame slice is the only way to match or get close to real hardware behaviour.

In your tests, the first half of the screen is always too late with plain frame delay, this is because input is only polled once, at the bottom of the screen.

On the other hand, with frame slice, inputs are polled in middle frame too, giving time to the emulated cpu to process that input on time.

There's still a gap of 6 ms as compared with the fpga. This must be roughly your host system latency. You could cut some extra ms by using a more powerful cpu (finer frame slicing) and specially by overclocking your USB port or simply using a higher speed input device (if the device is slow, it won't help overclocking the port).

[jimmer]

I was going to ask about whether frameslice 8 would improve things, I assumed not by much as the driver is still polling at 120hz, but I've probably missed something.  How much do you think it should improve in theory?

My input is good I think: 1MHz usb polling, zero delay on button press (coded that myself).

About higher performance PC. My system is i3-2120 onboard HD2000 graphics. I've just checked the cpu usage and one thread is near 100% another is around 15% and the other 2 are 0%, so it's effectively using just 1 core. I can't overclock these Dells so best I could reasonably do is get a 20% faster cpu eg i5-3570 HD2500.  Is it linear ie that would give me one more slice ? 

PS. I've just been playing the frameslice version and it feels nice. It's very hard to be sure of anything at this sort of small change but my performance at shooting baiters in top half of the screen was better than it has been. But that's a very one time anecdotal test.  In previous testing I felt like I could tell the difference between framedelay 1 and framedelay 7 (ie 10ms average lag difference) but I wouldn't swear to it. One day I'll do some statistically significant blind testing.

[1500points]

Did you ask james r himself. He surely has details about this kind of thing sussed already.

So is this work for the defender mws arcade cabs you sell?  A high end pc vs an fpga sounds like the money isn’t much different.


Sent from my iPhone using Tapatalk

[jimmer]

Did you ask james r himself. He surely has details about this kind of thing sussed already.

So is this work for the defender mws arcade cabs you sell?  A high end pc vs an fpga sounds like the money isn’t much different.

JROK obviously knows the original hardware inside out, but I don't know if he knows anything about MAME emulation. In any case, he rarely replies to my questions and suggestions about his own product, so seems unlikely to be a fruitful line of enquiry

This is my ongoing quest to get the best Defender emulation I can on a flatscreen, not sure if it's a hobby, work, or an obsession at this point.

Why don't I just use the JROK?  Well, for one thing, I'd still need a PC for other games and combining them makes for a less tidy solution.

[1500points]

Ahh.

With the thousands of hours the fpga folks put into dissecting code and schematics and the real hardware cross testing, you should probably still tread lightly out of respect.

Putting your layman observations on fpga performance is likely not very accurate scientific results.

Didn’t James get involved in mamedev stuff in past. I thought all the gurus did at one point or another.


Sent from my iPhone using Tapatalk

[jimmer]

I think you've misinterpreted what I'm saying in this thread, so let me expand:

I'm using the JROK as my benchmark because it is accepted as arcade accurate hardware.

But, if JROK+CRT is arcade perfect, then JROK+LCD is slower than arcade.

And what I'm pursuing is  SOMETHING + LCD that isn't slower than arcade.

In other words, because half my system is not original (slower) the other half has to be not original (faster).


Whether differences of 5ms,10ms mean anything in game playing reality is another question.

[1500points]

Ah so you are chasing a unicorn.

So here is a thought experiment for you since robo is a faster busier game than defender per hw needs.

I have spent many hours in past beta testing effort with real hardware, jrok of different revs, and mame of various tweaks.

Jrok w/crt vs jrok w/lcd.  You can’t feel the difference. 
Jarvis and I discussed this briefly in past, he is old school hardcore that crt is better than lcd....yet raw thrills is stuck with lcd too...but you can’t feel the pedantic timing that you might see with a scope.  Humans aren’t that granular.


Sent from my iPhone using Tapatalk

[1500points]

There is a way you can see detectable lag by human observation though.  Run the jrok with cga output into a gonbes cga-vga then play on lcd.   It is sketchy and will even display some odd behavior like ship that’ll glitch a few pixels off randomly.


Sent from my iPhone using Tapatalk

[jimmer]

Ah so you are chasing a unicorn.

Not really. I'm pretty sure that with mods to the MAME Defender emulation and maybe also mods to groovymame to cope with the MAME changes, that there could be a better emulation (120hz polling) that can also be framedelayed. 

The end result then depends on the actual magnitude of the lag in the PC system and the LCD. Using framedelay Groovymame can claw back up to 15ms of average lag on a 60hz poll, but only up to about 7ms on a 120hz poll rate.

Jrok w/crt vs jrok w/lcd.  You can’t feel the difference. 

I'm a long way from any conclusions on what sort of delta is noticeable or important (those could be 2 different things) in general or for any particular game.  That's one reason for wanting to remove all the lag relative to arcade, much better to say there is no lag than to say there is some lag but it's OK it doesn't make any difference.

I didn't manage it this year, but next year I hope to test some of the best players to see what they can feel. 

hmmm, come to think of it, I can write a script to blind test myself (that's not a claim that I'm one of the best!). I can add up to 13ms average lag to the Groovymame171 FD9 system.

[Paradroid]

And what I'm pursuing is  SOMETHING + LCD that isn't slower than arcade.

Mate, just get a CRT already! And get an original Defender PCB while you're at it.

Anything less is going to prioritise convenience over performance... which is fine trade-off in most cases.

I love GroovyMAME but I also have a few choice PCBs (e.g. Golden Axe, Metal Slug, Strider, SF2 Hyper, Raiden, etc.) for games that were dear to me growing up.

No point making it your life's mission to overcome emulation limitations when it's just not that hard to acquire the original article in most cases.

[1500points]

It isn’t quite that easy in the UK though is it?  Plus isn’t this for selling mame cabs? 


Sent from my iPhone using Tapatalk

[Paradroid]

It isn’t quite that easy in the UK though is it?  Plus isn’t this for selling mame cabs?

True: anywhere outside Japan and the US makes PCBs more expensive. However, CRTs should be workable since the UK and Europe have SCART.

Don't get me wrong, I love MAME and I'm no PCB collector. Just pointing out that the real thing isn't out of the question.

Sent from my SM-G955F using Tapatalk

[jimmer]

I had a jrok+crt system but it got dismantled when I was short of space.

What you (paradroid) don't know is that I want a menu with about 15 Defender versions/settings. ArcadePCB and JROK don't meet that requirement.  So groovymame+CRT would be my CRT solution, but then there's still the emulation problem I'm highlighting in this thread (I assume it also applies to CRT ?).

So as it stands now I'd be looking at building a frameslice+CRT system and then testing it against JROK+CRT benchmark to see how close it was to perfect.

Or I persuade James R to produce a JROK firmware with configurable slots/menu.


And then after all that, when I have a perfect Defender CRT system, I might still prefer to save the space and cash and just play near perfect Defender on my all-in-one LCD cab.

[jimmer]

It isn’t quite that easy in the UK though is it?  Plus isn’t this for selling mame cabs? 

I've altered my signature to clarify who jimmer represents on this forum.

I'm pursuing the overall best game system for myself. But then obviously everything I learn about arcade gaming systems informs the advice that customers of jbgaming get.

[Paradroid]

What you (paradroid) don't know is that I want a menu with about 15 Defender versions/settings.

Right! Well, that explains your persistence then. Emulation would be the only way to achieve that outcome.

Sorry, I wasn't meaning to criticise your approach... I wouldn't have commented at all but when I saw the words "flatscreen" and "obsession" in the same sentence a few posts back my own LCD prejudice couldn't help itself.

[jimmer]

Sorry, I wasn't meaning to criticise your approach... I wouldn't have commented at all but when I saw the words "flatscreen" and "obsession" in the same sentence a few posts back my own LCD prejudice couldn't help itself.

That's OK, and groovymame is (was?) a CRT thing  (I wonder what screen type most new users are using )

And just because I have some reasons, doesn't means it's also not an obsession!

[Paradroid]

And just because I have some reasons, doesn't means it's also not an obsession!

Haha! I never doubted your level of obsession... just your choice of fetish.

Each to their own though. That's what makes the world go around.

[faybiens]

I tried Frame slice
Did lots of slowmo, and I think my jamma usb is not a fast polling device

Sometimes, I can get a fantastic input reaction to the next frame at 40/50% of the screen and at another time I will not get any reaction to next frame when at 20/30%
Smells like 125hz polling... I did changed to 1000hz and using rawinput option, but seems this is not working

I will give a try to the Jamma JPAC, Calamity gives very positive feedback on it

[faybiens]

Got my JPAC, and it definitely makes a difference vs the JammASD
Slow-motions show that I can get many next frames reactions
Definitly a good move, better feeling in games I used to play on hardware, like Garou Mark of the Wolves.

Just understood of the advantage of frame slice.
How can we know which hardwares/games do sub frames updates?

I think I need to play with defender to understand
Shouldn't these type of games be non-sync as well? Read for example that Atari 2600 draws in real-time and takes in account any action happening anytime

[1500points]

I think I need to play with defender to understand
Shouldn't these type of games be non-sync as well? Read for example that Atari 2600 draws in real-time and takes in account any action happening anytime

I really like the jpac/ipacs for quality and dependability. 

2600 pitfall is a game where you can easily feel lag. Try a running jump over 3 alligators on a pi.

If you aren’t well versed at playing defender it will be hard to feel what you are trying to observe.  Some people think the 19-1 defender is decent but if you compare to real/jrok you can feel sluggish lag.


Sent from my iPhone using Tapatalk

[faybiens]

Yes the JPAC is really reactive

So if I understand well, with frame slice, games that do take in account actions (button press) during a frame can get a reaction in the same frame?
Like if each slice is an emulator recalculation, compared to frame delay which recalculate once at the next frame
So for frame_slice set at 7, 7 recalculations will occur during a frame (1 for each slice) compared to one for frame_delay, 7 peaks of CPU ressources instead of 1

We could imagine a recalculation at each action received by the emulator

[jimmer]

Just understood of the advantage of frame slice.
How can we know which hardwares/games do sub frames updates?

I think I need to play with defender to understand
Shouldn't these type of games be non-sync as well? Read for example that Atari 2600 draws in real-time and takes in account any action happening anytime

Possibly it's a new thing that you can interrogate the video driver to find out where the scan line is, and that now opens up the possibility for the emulators to do the same things that the original games were doing.

Defender testing might confuse you more than it helps though. I've done lots more tests and found behaviour that changes depending on how many enemies there are, and behaviours that change over time (maybe flip-flop periodically). On top of that there is the correct slow-down behaviour when things get busy.

I've done enough testing. Now I'm going to focus on the code.

[faybiens]

Yes, you are right for defender, I just read your base line :-), I should focus on things I know like the Neogeo.

See how the hardware behaves when action is received in the middle of a frame, if the action comes 2 frames after or 3 frames after as suggests Shift+P in GM, and what happens when action is received at the bottom of the frame to see if there is a kind of frame delay behaviour...
Then compare with frame slice if necessary

Globally if actions are queued at end of a frame (like calamity called confirmed for the cps1) or 2x times or more during a frame

That's all I can do, but so far that JPAC and frame_delay is a winning combo
I'll take time to do that

[Calamity]

Yes the JPAC is really reactive

So if I understand well, with frame slice, games that do take in account actions (button press) during a frame can get a reaction in the same frame?
Like if each slice is an emulator recalculation, compared to frame delay which recalculate once at the next frame
So for frame_slice set at 7, 7 recalculations will occur during a frame (1 for each slice) compared to one for frame_delay, 7 peaks of CPU ressources instead of 1

We could imagine a recalculation at each action received by the emulator

Frame slice and frame delay implementations are mutually exclusive.

Normal emulation runs the emulated CPU for a whole frame, say 224 scanlines, then it draws the screen and polls inputs.

With frame slice 1, the same cycle is done for scanlines 0-111, then for 112-224. Inputs are polled in the middle and in the end.

The higher you set frame slice, the finer the CPU slices and the input granularity. In theory you could slice at the scanlines level or even more but this is still science fiction on nowadays computers.

[faybiens]

With frame slice 1, the same cycle is done for scanlines 0-111, then for 112-224. Inputs are polled in the middle and in the end.

At frame slice 1, if a game is a one frame reactive game and do polls input several times in a frame,
The same second emulator cycle for scanlines 112-224 will take in account the inputs happened in scanlines 0-111 and then show the reaction in these 112-224 scanlines?

[Calamity]

At frame slice 1, if a game is a one frame reactive game and do polls input several times in a frame,
The same second emulator cycle for scanlines 112-224 will take in account the inputs happened in scanlines 0-111 and then show the reaction in these 112-224 scanlines?

If you mean "current frame" response, yes it's possible, check this video:

However, by "one frame reactive" game I'd understand something different: action will only happen on next frame at best, and frame slice 1 would be equivalent to frame delay 5, meaning that the probability to get input ready for next frame would be 50% (assuming zero host system latency).

Frame slice 2 would be equivalent to frame delay 6.67 (2/3)
Frame slice 3 would be equivalent to frame delay 7.50 (3/4)

and so on...

[jimmer]

Time to resurrect this thread!


Inspired by oomek, my TEMT6000 photosensor arrived on Saturday and I can now take a lot of measurements quickly. I made a modified Defender ROM with 1 pod and no baiters so nothing interrupts my measuring except the pod drifting through the screen now and again. My arduino sends a Fire keypress and then times how long it takes for the laser to emerge. It delays by an extra 1ms every loop so it works it's way through the polling window. If it doesn't spot the laser after 6 frames it calls a miss and fires again (I monitor the output and move the sensor around until I find a good spot). 

On my work machine (i5-2500 with 3 screens) I can't have lots going on if I want clean results, but my arcade machine (i3) always gives clean results. However the times were similar so I'm doing most testing on my work machine because it's quicker to analyse.

JROK hasn't been wired up yet, but I've found plenty to look at in the meantime.

I should say that contrary to what I thought earlier, Defender polls the gameplay inputs at 60hz. However for fun I made a modified rom with 120hz polling.



Some findings/thoughts:

Note: 60pc means 60% down the screen ie just below centre.

The plots show what we expect with the longest response being 17ms slower than the fastest. And therefore trying to find averages isn't needed, it's easier to just find the fastest response.

I've made a summary which shows the fastest responses against screen height.

groovy195_frameslice5 is considerably more responsive than groovy171_framedelay8 (by about 10ms)

I tested my 120hz Defender ROM, it doesn't alter the 17ms pattern but it does improve the response everywhere by about 6ms. I'm not sure what that tells me yet.