Measuring lag with High Speed Video

[rCadeGaming]

By the way, I'm interested on your high speed camera method. May I ask you which model are you using? Price?

The camera isn't any high-end video camera.  I just happened to stumble upon an interesting option in my photo camera, a Nikon Coolpix S9100.  It has some basic video recording capabilities, and I noticed that it has a 240fps mode.  When trying to run the camera this fast, the resolution is very low and the picture quality is pretty poor, BUT the timing is actually 240fps and it's very steady and accurate. 

You can confirm this by recording video of a CRT running at 60Hz.  When you play the video back, advancing frame by frame, you can see that the CRT takes 4 frames of the camera's video to finish scanning a frame.  This makes sense, as the camera should be running exactly four times as fast as the CRT, and the observable effect is steady and consistent.

Here's an example of a test I did using this, where I wanted to compare the lag of various video processing equipment that I was considering for my cabinet:

I recorded the TV with a camera in 240 fps. I got the TV screen and a closeup of an arcade stick in the same shot. I recorded pushing the light punch button in Third Strike, and counted the number of frames between the button being depressed and Ken beginning his punch animation. That number of frames in the video, divided by four, equals the number of frames taken in the game (240 / 4 = 60). I can confirm that the camera is running at 240 fps and the game is running at 60 fps because I can see the TV's vertical scanning in the video, and I can see it completes a frame every four frames, if you know what I mean. The results were consistent with repeated trials.

PS3 (480i) -> TC1600 -> TV = 3 frames until punch animation starts, defined here as 0 frames of lag

PS3 (480p) -> Super Emotia -> TC1600 -> TV = 3 frames to punch, 0 frames lag

PS3 (480p) -> RGB 202xi w/ADSP -> Super Emotia -> TC1600 -> TV = 4 frames to punch, 1 frame lag

PS3 (480i) -> Andora -> Super Emotia -> TC1600 -> TV = 4 frames to punch, 1 frame lag

PS3 (480i) -> Andora -> RGB 202xi w/ADSP -> TC1600 -> TV = 5 frames to punch, 2 frames lag

As you can see, it seems that the Super Emotia does not add any lag, but the Andora 202 actually add a frame of lag each. I'm guessing the 202 needs to use some video memory to do that position shifting, I'm not entirely sure why the Andora would add lag if the Super Emotia doesn't.

The conclusion of the Super Emotia adding no lag technically depends on the assumption that the PS3 itself doesn't add any lag when interlacing. I can check this by adding some NAND gates to my LM1881 circuit, which will allow me to get H and V sync from the LM1881's C and V sync. This will allow me to connect the PS3 directly to a CRT computer monitor in 480p for comparison.

The results of the 202 and Andora wouldn't be affected by this.

A TC1600 is an RGB to component transcoder that I use (a transcoder converts colorspace only, no scaling).  That and the CRT TV are theoretically considered to be lagless, so the benchmark of 3 frames lag is considered to be the minimum time taken between the controller and the system and the way the game is programmed.

An Extron Super Emotia is a scan converter that I use to force 480p to 240p.   An Extron RGB 202 is an interface mainly used here for picture shift.  An Extron Andora is a rudimentary line doubler than can be used to convert 240p/480i to 480p.  Sorry if anyone already knows all this, just want to clarify for context.

The end results of the test were that I will use the Super Emotia to force 240p in certain modern console games (like Third Strike Online or Mega Man 9, the latter scales cleanly only on the Wii), but I don't think I'll be using the RGB 202 or Andora anywhere in the cab.

You can test lag from just about anything with this method, provided that you can provide a reasonable benchmark to compare it to.

I tested an LCD TV I have against the CRT TV with this same setup, and found the TV lagging by about 2 frames.  However, the TV is supposed to be a 1 frame TV.  The discrepancy could be due to the fact that I had to run a PS3 in 480i to measure the CRT, whereas it was running in 1080p to measure the LCD.  This certainly could introduce at least a frame of difference within the PS3, and this difference among resolutions could even vary among games (was using MVC2 online for that one).  So, providing a universal benchmark is important, and it isn't always simple.

If I continue testing things this way, I plan to wire an LED to the button being pressed, which would give a better indication of the start of a button press, instead of visually judging at which frame the button has been depressed.

[Calamity]

Well that's very interesting, I'm not very keen on cameras, I had read about a Casio model with high speed recording but didn't know if this is a common feature to the non-professional sector.

So you didn't test this with MAME did you? This seems the proper method to measure lag. I guess that wiring a bulb or a led is a must if you want a reliable visual mark on the recording. It could be interesting to create a custom MAME build too, that prompted something on the screen when button 1 is pressed, for instance, as soon as the input is read by the emulator's osd layer (not by the emulated pcb). This would provide a measure of the overhead associated to the OS, and that one that's inherent to the emulated system.

[rCadeGaming]

Well that's very interesting, I'm not very keen on cameras, I had read about a Casio model with high speed recording but didn't know if this is a common feature to the non-professional sector.

Me either, just happened to notice it on mine and thought I could have some fun.

So you didn't test this with MAME did you?

Not yet, it's on the to do list though.  I may even make an excel sheet to compare lag among versions of all my commonly played games, whether comparing in different MAME builds, or different video settings, or comparing a console port.

I'd be willing to help test the effects of your frame delay option in GroovyMAME, as I'd love to help reduce lag in MAME.

This seems the proper method to measure lag.

Yes, as long as you can provide a proper benchmark.  This is no problem in MAME, as you can stick to the same PC and monitor.

It becomes messy when trying to measure LCD's if you can't use the same resolution or connection method on the LCD as you do on the CRT.  I used to have a Dell CRT monitor that could do 1080p, which would be really useful for this.

I guess that wiring a bulb or a led is a must if you want a reliable visual mark on the recording. It could be interesting to create a custom MAME build too, that prompted something on the screen when button 1 is pressed, for instance, as soon as the input is read by the emulator's osd layer (not by the emulated pcb). This would provide a measure of the overhead associated to the OS, and that one that's inherent to the emulated system.

Yeah, that would be great for benchmarking and optimizing your MAME PC.  That sounds like a job for you!

[MonMotha]

There are several variants on this technique that can work even if you don't have a 240fps capable camera.  They're still a bit uncommon but starting to show up in prosumer and even high end consumer space.

If you only want to know the latency of the monitor to within one 60Hz progressive (240p) frame time, you can output frame accurate timecodes and hook the signal up to both a known lagless (i.e. no digital scalers) CRT monitor + the DUT and just take a picture of the output with a short shutter time (less than 1/60th).  The difference between the timecodes is the latency of the DUT in frames.  This just needs a camera with which you can adjust the shutter time and a PC with suitable software to generate the timecodes.

Another trick is to build a video generator that toggles between a totally white frame and a totally black frame e.g. once per second.  A logic signal is output from the generator to indicate which state it is in, and the video is attached to the DUT.  You put a light sensing device (e.g. photodiode) in front of the TV and attach the output from that (amplified through a comparator to give an on/off logic indication) and the state indicator from the generator to either a timer/counter or an oscilloscope.  Let 'er run, and the time from the signal transition from the generator to the transition on the photodiode is the lag of the monitor.  On a scanned device like a CRT, you can actually even measure the placement of the photodiode with this trick (it's a rudimentary lightgun).  This is crazy accurate, but a fair bit of work, especially if you want to test digital (DVI/HDMI/DisplayPort) inputs on the DUT.

Anything involving response time of software on a PC is a bit suspect.  The timing just isn't guaranteed to be that accurate on a multitasking OS.  However, averaging together several runs can provide decent indication.

[rCadeGaming]

If you only want to know the latency of the monitor to within one 60Hz progressive (240p) frame time, you can output frame accurate timecodes and hook the signal up to both a known lagless (i.e. no digital scalers) CRT monitor + the DUT and just take a picture of the output with a short shutter time (less than 1/60th).  The difference between the timecodes is the latency of the DUT in frames.  This just needs a camera with which you can adjust the shutter time and a PC with suitable software to generate the timecodes.

Yes, this is the method used for the sub 1 frame monitor database that I always reference when someone starts talking about buying an LCD:

http://shoryuken.com/forum/index.php?threads/sub-1-frame-hdtv-monitor-input-lag-database.145141/

Edit:  That link seems to have stopped working, here's a new one

http://forums.shoryuken.com/discussion/145141/sub-1-frame-hdtvmonitor-input-lag-database/p1

The thing is, to properly test the minimum lag of an LCD you need to be feeding it it's native res on it's best input.  This will usually be 1080p on the HDMI input, but I think there could be some debate about the VGA input being faster if available.  This signal has to be split and sent the lagless CRT as well, so you need one capable of accepting it.  So you'll usually need a lagless CRT that accepts 1080p, if you're using HDMI you might be able to use an HDMI to DVI adapter.  You might have to confirm the lag of the adapter first though.

Anything involving response time of software on a PC is a bit suspect.  The timing just isn't guaranteed to be that accurate on a multitasking OS.  However, averaging together several runs can provide decent indication.

I would always average several results on any method, just to be safe. 

For the button method, there will always be a partial frame variation based on the random nature of at which point during a frame the button is actually pushed.

[adder]

hey all due to all the chat of lag/input lag i have read lately, i got paranoid and thought i would try some tests of my own.
using the game galaga'88 in mame i tried the advance-frame method and noticed that galaga'88 has an input lag of 4 frames.
what i also noticed though was that this number did not change depending on if i had triple buffer turned on or off.

i tried some other games also, eg. original galaga, 1-9-4-2, armed police batrider ... and triple buffer on/off didnt affect those results either

i dont get it, i thought triple buffer is supposed to introduce lag when turned on?
it would be nice to know your thoughts, eg. what video settings (or other settings/factors) in mame do you recommend to keep lag/input lag to a minimum?
kind of lost with all this atm

[rCadeGaming]

If you understand how triple buffer works, it seems there's no way to implement it without introducing lag. 

However, while this effect should be guaranteed in real-time gameplay, maybe it doesn't show up in the frame advance method.  Or maybe your hardware, or drivers, or DirectX is incompatible or has a problem and it wasn't actually working.  Or maybe something else in your video settings was overriding it and it wasn't turning on.

Could you make another thread for this and ask Haze and Calamity to look at it?  I'd be interested to know.

Post your mame.ini's, one for each of the settings used to compare, and your system specs, including what MAME and OS.

[adder]

If you understand how triple buffer works, it seems there's no way to implement it without introducing lag. 

However, while this effect should be guaranteed in real-time gameplay, maybe it doesn't show up in the frame advance method.  Or maybe your hardware, or drivers, or DirectX is incompatible or has a problem and it wasn't actually working.  Or maybe something else in your video settings was overriding it and it wasn't turning on.

Could you make another thread for this and ask Haze and Calamity to look at it?  I'd be interested to know.

Post your mame.ini's, one for each of the settings used to compare, and your system specs, including what MAME and OS.

ok, i better sleep now as it is nearly 4am here but will come back tomorrow
ps. i ran these tests here on my laptop (lcd), but tomorrow i will run them again but this time on my mame cab, with vga to scart -> crt tv, to see if that makes a difference .... will report back

ps. perhaps if you get the chance, you could also try the advance-frame method and galaga'88, see if you get the 4 frames delay with triple buffer on/off on your setup...