Encoder benchmark question

[PL1]

Toodles performed this test in response to an erroneous lag claim from another manufacturer of encoders.

It appears to be a solid test of which encoder is faster, but doesn't indicate how much faster.

If encoder X takes 2 mSec longer to process a button push than encoder Y, it's probably not enough to make a major difference in play, but could still give results that look very impressive.

If the difference is 16.7 mSec -- one frame @ 60 Hz -- it is much more likely to be an issue, even though either way Y is faster than X and will win in Toodles' setup.

Is there a combination of hardware and software that can be used to objectively test encoders for speed/lag?
(No human judgement involved, per Toodles' example)

What are the best practices/configurations for doing "apples-to-apples" benchmark testing of encoders?


Scott

[Gatt]

His test is about as close as you can get without a really complicated set of custom software.  To do better,  you'd need something to read the electrical signal as it is returned to the encoder,  software to mark the time,  and software/hardware to mark when it was output to the USB port.  It would likely not be cheap.

His test is *extremly* problematic though.  There's alot of room in there for major error,  such as...

-How does the hardware he used process USB inputs on the same hub,  is it queued?  Random?  I thought USB priority went to whichever port had the lowest ID when contention occurrs. 
-The CPU is going to pick one of the inputs to process first,  even if both arrived at the exact same ms.  So his test is highly subjective since it's prey to the design of the PC's architecture.
-Worse,  it's entirely possible that the PC might insert some other command for execution in between the two inputs,  even if they arrived at the same time.

So while interesting,  his test isn't valid,  because he hasn't eliminated the architectural issues from the test.  To actually test the latency of encoders will require hardware reading at the point of input and the point of output,  otherwise alot of other variables are inserted into the test that are completely unknown.

[PL1]

One minor correction about Toodles' test setup:  He didn't use a PC.  Not sure if that changes anything you brought up. 

Some of your points touch on the concerns that led me to ask about this.

I figure if you can create a testing configuration that keeps everything but the encoder the same, it will remove many of the variables that you mention.

My first thought on how to approach this was to use a storage o'scope.

Setting a start time is easy.  Trigger a storage o'scope on the negative slope of the voltage on the NO tab of the microswitch when it is pushed.   

The hard part is what do you put on the second trace to measure when the encoder sends an output and how do you ensure that you can check the same point on the various encoders tested?   


Scott

[Gatt]

I can think of two things you could try...

1.  Strip the USB cable and attack a volt meter there,  there has to be some kind of rise/fall when the encoder sends input
2.  Write software to monitor the USB port on the board and see when data reaches it,  there's software out there that'll let you look at what comes across the USB port.

Option 1 would probably be easiest.  I'm not a pro on volt meters by any means,  but couldn't you use the same volt meter to monitor NO and the USB port?  That would make getting a time easy wouldn't it?

[RandyT]

I agree with Gatt.  There are a lot of system variables in the equation which have not been accounted for.

At very minimum, another identical test should have been conducted with the controller ports swapped, and the results averaged.

There is also the possibility that the algorithms in the game will favor the stationary character for a short period of time, much like the way in which defensive attacks take a bigger toll on the opponent in a number of fighting games.  At minimum, the approaching character should have been allowed to rest in position for at least a couple of seconds before striking to eliminate this possibility.  I didn't see too many instances in the video where this was allowed to happen, and a "tie" wasn't registered.

But the plain fact of the situation is that the system is responsible for wrangling the incoming data, and if it does this in a specific order, which it must, then that ordering will be the thing which tells the tale when both encoders are equal.

[PL1]

I agree with Gatt.  There are a lot of system variables in the equation which have not been accounted for.

The question still is how to remove those variables to compare different encoders and objectively determine
  1.) are they unequal, and
  2.) to what extent are they unequal.

1.  Strip the USB cable and attack a volt meter there,  there has to be some kind of rise/fall when the encoder sends input

Sorry to burst your bubble, Gatt, but using a volt meter for this application is like using an hourglass to compare 100 meter sprint times between two runners.

A volt meter does not allow accurate (mili- or nanosecond) time measurement or comparing time differences between two events.  That's what the oscilloscope, specifically a digital storage model, is good for.

I guess I need to research what format the outgoing data packets will be in.  If the outgoing packet for a common keystroke can be translated into a specific series of 1s and 0s, that will translate into a waveform that can be found, verified, and the time difference measured on a storage scope.

2.  Write software to monitor the USB port on the board and see when data reaches it,  there's software out there that'll let you look at what comes across the USB port.

If you know of a USB packet sniffer that does something similar to what the ping command does, a lead would be greatly appreciated.


Scott