USB problem isolated!!

[JoyMonkey]

I got 1-14 hooked up and I installed test3 on my Via and 2 Nvidia's.  The results are wonderful!  All 3 systems run the program 100% the same...

Southpaw

Woohoo!

[MikeQ]

Mike,
What was the difference between the first test and the second?

It worked with my hub, and just realized I should probably of put that test under the analyzer as well.  Need to find a 2.0 PCI card and compare my results with that as well.

The test that worked is test3.  Test1 did a write-through of any hardware cache that may exist between the write command and hardware.  Test2 was an attempt at overlapped asynchronous I/O with notification of each report finishing.
Test3  I forced a flush and wait after each HID report is sent.

[MikeQ]

I have another test I'd like someone to run on one of the offending Nvidia chipsets.

You can download the zip file here: www.unappliedbraincells.com/test/LEDWizTest.zip

It is only about 45k.

This appears to fix the problem on my Athlon64 4800+ with nforce4 chipset.

I just tested this new one (I'll call it testbench3).
Without a clean boot, all my outputs saw quickly seven times, then once a little slower, then it repeats the sequence.

I'll try doing a clean boot now...

With a clean boot, it performs the same sequence (all my outputs saw quickly seven times, the once a little slower), but not as fast as without a clean boot.

Again, I've only got nine outputs hooked up (4,5,6,7,8,9,30,31,32).

What you should see is a fade in from 0 intensity to full intensity and then a couple of sawtooths.

[JoyMonkey]

Now that I look at it again, it does start with a fade in (what I thought was a slow sawtooth), then six or seven sawtooths.

[MikeQ]

Now that I look at it again, it does start with a fade in (what I thought was a slow sawtooth), then six or seven sawtooths.

Sweet.  This really excesses the PBA command that has been gaging hubs and Nvidia Cripsets.

[southpaw13]

So can you give us any insight as to where you are on the fix, or is it something Randy needs to address...

Thanks for your continued support of this project,
Southpaw

[MikeQ]

So can you give us any insight as to where you are on the fix, or is it something Randy needs to address...

Thanks for your continued support of this project,
Southpaw

No, Randy won't need to get involved.  I have a working knowledge of the hardwares interface so I can make the fix.  Randy may want to make the same fix to the control panel however.  PowerMAME will require a new Groovy.dll.  It will be included in the next release. 

The fix sucks so I'm putting in some extra logic to only issue the PBA command when it truly is different than the last time the command was sent.  A lot of times before sending a new sequence, a PBA is sent to initialize things to a know value.  If however the hardware was already in that state, the command doesn't need to be sent.  So I'm going to keep a shadow of the hardware registers and when a PBA command is issued, I'll check it against the shadow and only update if they differ.  Since the fix requires some idling, I only want to issue the command when truly necessary.

[JoyMonkey]

I have no idea what you just said, but I think in laymans terms what you said is probably "blinky lights more harder, but more betterer now"

Nice job!

[mccoy178]

Thanks guys, there are lurkers watching progress.

[MikeQ]

I have no idea what you just said, but I think in laymans terms what you said is probably "blinky lights more harder, but more betterer now"

Nice job!

That's getting awfully technical....but yes...I think.

[2600]

I'm not sure what Randy is going to do and not sure if I should put in my $02 on the problem, but we do it with everyone else so as long as I don't give away his secrets should be fine.  I don't want to give out too much of how Randy's controller works on the USB level so it's dumbed down.  The important thing is how to avoid the problem.


I believe the problem is do to the way the OHCI USB controller and some 2.0 hubs work.  The controller and hubs can make some commands happen faster compared to a UHCI controller.  This is perfectly USB Spec legal.  Speculating, but usually when this happens the firmware of the device is not fast enough to deal with it and the buffer is getting overwritten or some other incompatibility with the OHCI controller.

To avoid this problem for now in software, you can do like Mike has done and add delays, flush buffers, etc and *hope* you don't see it again.  So that means not only a new dll, but new ocx, and new applications that needs the new ocx or dll.  Applications can also minimize the use of commands that use multiple stages.

In hardware, if you have an Intel and Via chipset or use a Via USB card you are fine if you don't use a 2.0 hub.  Everyone else including ATI chipsets(winkie winkie Mike) then you may get this problem.

For hubs, I imagine it can happen with a lot of USB2 hubs on USB2 systems.  Maybe not on Via USB2 hubs but I don't know what their marketshare is and don't have a VIA USB2 hub.

My $.02 and I could be completely wrong, but it would be just plain wrong and misleading to put the blame on Nvidia chipsets.

Randy, or anyone else for that matter, if there is any information that you think is incorrect let me know and if it is I will remove it.  If you need any of the USB analyzer captures let me know, not sure how much use they would be though.

[MikeQ]

If each report is sent synchronously and each report fits in the devices input buffer, how could the buffer get overwritten? 

There is an obvious timing issue involved here but I'm confused as to why synchronous writes appear to have an asynchronous quality to them.  The offending command consists of 4 hid reports of the same size.  I send each of the 4 reports synchronously.  It appears that when I write the 4 reports one after the other, they clobber each other.  In theory, I should not be getting control back in my driver to send the next report until the first report has finished. 

What component is performing the blocking and reporting to the OS that the write has been performed?

Just out of curiosity, I'd like to see the analyzer dumps.  This is my first venture into USB devices so you've got me interested in fully understanding this.  The workaround will suffice for now though.

[2600]

If each report is sent synchronously and each report fits in the devices input buffer, how could the buffer get overwritten?

Here's one way:  Some of the reports are received back to back.  The report includes a setup and data packet.  Once the report is received, the device starts decoding the setup packet.  During this time, the next report is received which overwrites the previous data packet before the device has decided what to do with the data.

There is an obvious timing issue involved here but I'm confused as to why synchronous writes appear to have an asynchronous quality to them.  The offending command consists of 4 hid reports of the same size.  I send each of the 4 reports synchronously.  It appears that when I write the 4 reports one after the other, they clobber each other.  In theory, I should not be getting control back in my driver to send the next report until the first report has finished. 

What component is performing the blocking and reporting to the OS that the write has been performed?

Ok, before I get to the next questions.  I'm not a windows programmer.  Although, we may use windows a bit, we really use a real time linux usb stack that we wrote.      Second, I'm on the other side.  Designing the device and FW for it. I've helped the application programmers a bit, but am standing on less firm ground.

Your reports are being sent out fine.  All four of them  are there.  Even get back an Ack saying it was received.  But some of that is really low level and done in HW.

Let's say your application is really low level.  As far as your application is concerned, everything is fine.  Data was sent out and received fine.  The device is where the probelm is.  It received the data just fine.

With Windows though are you even that low enough of a level.  Aren't you merely sending it to the next buffer or part in the OS or even the stack that says, I'll take it from here.  The HW or Stack would probaby do a retry a couple of times before it complains to you that there is a problem. 
Kind of like file I/O.  Just because you write a file out, doesn't mean it is actually on the disk.  It could be in some cache or temp storage getting ready to be written, but the OS or HW is taking care of that for you.

Some of that is simplified to get my point across, but should be correct in general.

Just out of curiosity, I'd like to see the analyzer dumps.  This is my first venture into USB devices so you've got me interested in fully understanding this.  The workaround will suffice for now though.

Sorry that last part was meant for Randy.  I'm assuming he'd give you permission, but just want to make sure.  The part I'll clip out isn't too terribly exciting, but will show the offending part.  I'm going to see if I can borrow an OHCI card to get a capture of that as well.

[MikeQ]

It is hard for me to say what is going on under the hood since I'm using a Microsofts API.  It is bascially FILE I/O mapped to the hardware.  However, I have the ability to set (probably only request) flags that allow me to write through any cache and to perform non-buffered I/O.  Neither of these seemed to help.  However, I can't say what is really happening once I set these parameters.

So what is the solution for the device?  I've asked Randy to give permission to see the analyzer dumps.  He may chime in on this thread.  We could also take this to email if you want.  mikeq@unappliedbraincells.com

[RandyT]

Please feel free to give Mike any information about the device he needs.  He's a trusted party

As for the device not being fast enough to pick up the data, this doesn't really sound correct to me.  If this were the case, then simply adding gross delays between reports should have corrected the problem.  It was my understanding that it didn't.  It was also my understanding that there were spec limits on how fast a hub or controller would send reports to a device that has identified itself as a "low-speed" device.   

Have I missed something?

BTW, the LED-Wiz is doing a *lot* of stuff.  Providing 48 levels of PWM and HW modulation FX on 32 individual outputs without flicker takes a lot of work.  But the data coming in should be serviced in plenty of time.  We are only talking 8-bytes of data per packet and the receiver is interrupt driven.

And whether this is an Nvidia thing or not, I can't say.  But the only other chipset that has reported problems so far is an outdated Intel chipset.  The later chipset from Intel works fine.  So whatever Intel was doing that caused problems on the early one was changed in a newer version.  Coincidence?  Maybe.

Thanks, and please keep me updated.

RandyT

[2600]

Please feel free to give Mike any information about the device he needs.  He's a trusted party

Pics sent.  Should of CC'ed Randy, but don't have time to look up his email right now.

The two scenarios are using a PC with an 865g.  In one case, the LEDWiz is directly connected.  In the Second, a USB 2 hub with an NEC IC is being used.

As for the device not being fast enough to pick up the data, this doesn't really sound correct to me.  If this were the case, then simply adding gross delays between reports should have corrected the problem.  It was my understanding that it didn't.  It was also my understanding that there were spec limits on how fast a hub or controller would send reports to a device that has identified itself as a "low-speed" device.   

Have I missed something?

An OHCI controller can group multiple Control Transfers together in a frame.  I'm not clear on the size of the gross delays.  If large enough, your right it would have to work.  Speed was only one scenario.  It could also be HW implementation, which I would doubt, or something simple like not expecting multiple packets.  On the outside looking in so you would know more about your device.

BTW, the LED-Wiz is doing a *lot* of stuff.  Providing 48 levels of PWM and HW modulation FX on 32 individual outputs without flicker takes a lot of work.  But the data coming in should be serviced in plenty of time.  We are only talking 8-bytes of data per packet and the receiver is interrupt driven.

I was thinking the other day about how much it is doing.  It is a lot.
Also, remember that it is 16 byte's.  8 byte's for the setup packet.  8 byte's for the data.

And whether this is an Nvidia thing or not, I can't say.  But the only other chipset that has reported problems so far is an outdated Intel chipset.  The later chipset from Intel works fine.  So whatever Intel was doing that caused problems on the early one was changed in a newer version.  Coincidence?  Maybe.

Thanks, and please keep me updated.

RandyT

Hadn't heard of the other Intel Chipset being affected.  Which one?  I think I may be able to test another scenario tonight.  We'll see if I can get to it.

[MikeQ]

Hadn't heard of the other Intel Chipset being affected.  Which one?  I think I may be able to test another scenario tonight.  We'll see if I can get to it.

The Intel system was an 82801 AB or EB.  Can't remeber which now.  I tried it on both the AB and EB and it works on one fails the other.  I think it failed on the AB but can't say for sure.  Also, this doesn't fail on either system when the LEDWiz is plugged directly into the system.  It only fails when the HUB is connected and the LEDWiz is connected to the hub.  This hub also gives my ASUS motherboard fits.  The computer won't boot with the hub plugged in.  I have to boot the system and then plug the hub in.  The hub very well could have a problem.

However, the same workaround that fixed the Nvidia problem fixed the hub problem on the 82801.

I don't know off hand what chipsets these systems are running.

[RandyT]

An OHCI controller can group multiple Control Transfers together in a frame.  I'm not clear on the size of the gross delays.  If large enough, your right it would have to work.  Speed was only one scenario.  It could also be HW implementation, which I would doubt, or something simple like not expecting multiple packets.  On the outside looking in so you would know more about your device.

I believe a delay of roughly 1000ms (1 second) between the reports was used and the problem still existed.  The SIE is a very slightly altered reference design from a very respected and oft utilized USB chip producer.  That doesn't seem like it would be the culprit either (but I'm not ruling anything out.) 

I was thinking the other day about how much it is doing.  It is a lot.
Also, remember that it is 16 byte's.  8 byte's for the setup packet.  8 byte's for the data.

I understand.  8 bytes per packet, but there seem to be 2 packets coming in rapid succession.  What I have read is that when dealing with a "low-speed" device, there should not be more than 1 packet transmitted by or intended to be received by the device but once every 10ms.  Is this information in error?  Obviously, the setup and data packets are being sent much closer together.  The question is:  why is it only a couple of chipsets that appear to be doing this?

Thanks for the help, BTW!

RandyT

[RandyT]

Hadn't heard of the other Intel Chipset being affected.  Which one?  I think I may be able to test another scenario tonight.  We'll see if I can get to it.

The Intel system was an 82801 AB or EB.  Can't remeber which now.  I tried it on both the AB and EB and it works on one fails the other.  I think it failed on the AB but can't say for sure.  Also, this doesn't fail on either system when the LEDWiz is plugged directly into the system.  It only fails when the HUB is connected and the LEDWiz is connected to the hub.  This hub also gives my ASUS motherboard fits.  The computer won't boot with the hub plugged in.  I have to boot the system and then plug the hub in.  The hub very well could have a problem.

FWIW, I have a 98SE system here with the Intel AB/EB (the later model) and it works just fine, even with a hub.

RandyT

[Howard_Casto]

Yes, that is the driver that is giving everyone the problems.  FWIW, I decided to use NVIDIA chipsets because my cabs double as a Daphne emulator also.  Daphne and ATI don't mix well together.  I have not run into any major problems with NVIDIA and Mame...
Maybe I should switch to Via chipsets with a NVIDIA video card?

Southpaw

Sorry to respond to this so late but. 

This is a complete and utter falsehood. 

Ati cards at one point (not anymore) had an issue with hwaccel in daphne.  The thing nobody mentioned though (because the daphne team, unfortuantely are nvidia fanboys) is that ati's 2d support is so much better than the average geforce card that daphne STILL performed better with an ati card with acceleration turned off than a similar nvidia card with it turned on.  How do I know?  Cause I tested it of course. 


To answer your second quesiton.. no you should switch to via chipsets with an ATI video card. Ati cards have better tv out and arguably use the mame filters better.  So you'll get a better picture regardless of the monitor you choose to use.  While daphne used to have issues with ati cards it doesn't anymore.  Even if it did that's only 20 games and it's not a emulator crippling issue, vs the 5000 in mame that will look better on an ati.  Not to mention the fact that the nvidia drivers still totally screw up visual pinball, making the tables unplayble so that's 500+ games with issues on the nvidia side vs the 20 in daphne that are now (and always were) a non-issue. 


Again... we haven't preached the benefits of ati and the evils of nvidia for the past 9 YEARS just for our health.  Nvidia sucks, only buy ATI!!!

(Howard_Casto awaits all the free stuff ati owes him for that winning endorsment.)

[southpaw13]

Thanks for the feedback, I will get with the program on my next build 

Southpaw

[2600]

Ok, first a primer so we are on the same page.  I just want to make sure the failing conditions are exactly known if it needs to go to a higher power.

There are three types of USB controllers:
UHCI - Universal Host Controller - Designed by and Licensenable by Intel.  Via is the only? one who licensed it.  For use with full and low speed devices.
OHCI - Open Host Controller - Used by everyone else. For use with full and low speed devices.
EHCI - Enhanced Host Controller - Open for everyone to use.  For use with High Speed devices.  Every EHCI has a couple companion controllers.  Either UHCI or OHCI depending on whose chip.

If you plug a LEDWiz device directly into a PC it will use either the UHCI or OHCI controller.  If you use a 1.1 HUB, it attaches to the UHCI or OHCI controller.  If you use a 2.0 hub on a system that has an ECHI controller(USB2), it attaches to the EHCI controller.  The 2.0 hub is responsible for the down conversion (transaction translator) and basically has a UHCI or OHCI in it.  If you attach the 2.0 hub to a system that doesn't have an ECHI controller the it attaches to the UHCI or OHCI contrller and doesn't have to do the down conversion.

Know when an when it does not work is crucial to see the pattern and debug it.  If it helps, go to device manager.  View -> devices by connection.  Then find which one it is connected to.


In Mikes example, when the LEDWiz is directly connect to the PC it attaches to the UHCI controller.  Using the hub, previousily mentioned 2.0 hub, one system (AB) does not have EHCI controller and connects to the UHCI.  I would think this would work.  The other example, (EB) does have have an EHCI controller.  The hub attaches to it and is running High Speed to the hub.  The hub converts it down to low speed for the LEDWiz.  I have this set up and it does not work.  The hub IC is made by NEC.  Not sure if Mikes is like this or is it reversed.

In Randy's example, 1.1 or 2.0 hub?  Who makes the hub IC?  Can you see what controller the LEDWiz is connected to?  Win98 with USB2, never tried it.

[2600]

I believe a delay of roughly 1000ms (1 second) between the reports was used and the problem still existed.  The SIE is a very slightly altered reference design from a very respected and oft utilized USB chip producer.  That doesn't seem like it would be the culprit either (but I'm not ruling anything out.)

1 second seems like long enough.  I dunno though.  SIE could also cause the problem, but that's out of my league.  The USB.org dev forum has people much smarter that could help and give other clues into it.  Is that test still available?  I could capture that to see what's going on.

I understand.  8 bytes per packet, but there seem to be 2 packets coming in rapid succession.  What I have read is that when dealing with a "low-speed" device, there should not be more than 1 packet transmitted by or intended to be received by the device but once every 10ms.  Is this information in error?  Obviously, the setup and data packets are being sent much closer together.  The question is:  why is it only a couple of chipsets that appear to be doing this?

Tired so hopefully my terminology is correct.
Control transfers consist of a setup and data packet.  In the capture I did, when it works you are getting this in one frame.  When it doesn't, you are getting 2 control transfers in one frame, 32 bytes.

A low-speed device device is guarenteed a maximum latency of 10ms.  The host can choose to poll faster.  This is for interupt transfers.

Control transfers are different and are periodic, kind of like bulk transfers.  A common problem is not counting on a feature of the OHCI controller which will group multiple control transfers into a single frame.  Latency is a different issue. 


Let me see if I can get this other box to fail and I'll do a capture of that as well.

[RandyT]

In Randy's example, 1.1 or 2.0 hub?  Who makes the hub IC?  Can you see what controller the LEDWiz is connected to?  Win98 with USB2, never tried it.

The chip in the hub is the GeneSys Logic GL650USB.  It's a 1.1 4-port hub (the little purple ones I sell at the store)

I am using this hub with the LED-Wiz on a Win2k system with a VIA USB2.0 card.  I'm assuming it is using the "VIA Universal Host Controller", but I cannot verify this.  Based on what you wrote above, and the fact that it is a 1.1 hub, it's probably a safe assumption.

98SE system: Intel 82371 AB/EB PCI to USB Universal Host Controller.  Same hub, no problems.

Perhaps sticking to USB 1.1 hubs on 2.0 cards and 1.1 USB cards when used without a hub would be a pretty safe way to avoid any of these issues.  Though,  the LED-Wiz seems to work fine connected directly to the VIA 2.0 and perhaps select other 2.0 chipsets.

RandyT

[2600]

The chip in the hub is the GeneSys Logic GL650USB.  It's a 1.1 4-port hub (the little purple ones I sell at the store)

I am using this hub with the LED-Wiz on a Win2k system with a VIA USB2.0 card.  I'm assuming it is using the "VIA Universal Host Controller", but I cannot verify this.  Based on what you wrote above, and the fact that it is a 1.1 hub, it's probably a safe assumption.

98SE system: Intel 82371 AB/EB PCI to USB Universal Host Controller.  Same hub, no problems.

Perhaps sticking to USB 1.1 hubs on 2.0 cards and 1.1 USB cards when used without a hub would be a pretty safe way to avoid any of these issues.  Though,  the LED-Wiz seems to work fine connected directly to the VIA 2.0 and perhaps select other 2.0 chipsets.

This is what I thought.  When you use a 1.1 hub on a 2.0 system, you are connecting to the companion controller of the ECHI.  This can be OHCI or UHCI depending on who's chip it is.  With Intel and Via you are fine and would be fine without the 1.1 hub as well.  If it is someone else's 2.0 chip, then you would connect to the OHCI and could have problems.



Also, an addition for the part where I say Control Transfers are more periodic as it can be read a couple of different ways.  Control Transfers are scheduled by best effort, which means not synchronous and not guaranteed latency.

[RandyT]

Perhaps sticking to USB 1.1 hubs on 2.0 cards and 1.1 USB cards when used without a hub would be a pretty safe way to avoid any of these issues.  Though,  the LED-Wiz seems to work fine connected directly to the VIA 2.0 and perhaps select other 2.0 chipsets.

This is what I thought.  When you use a 1.1 hub on a 2.0 system, you are connecting to the companion controller of the ECHI.  This can be OHCI or UHCI depending on who's chip it is.  With Intel and Via you are fine and would be fine without the 1.1 hub as well.  If it is someone else's 2.0 chip, then you would connect to the OHCI and could have problems.

Well, it appears that the nature if the beast has been identified so it can at least be easily accommodated.  I'll take a look at some more things on my end, but folks who aren't having problems, should continue not to have problems unless they change computers / USB hardware and end up connected to the OHCI. 

People just need to understand the limits to which the LED-Wiz is pushed from a hardware standpoint.  Maybe the addition of a double-buffering FIFO would help the situation, but there probably isn't enough memory or cycles left over to do that without disturbing other functions of the device. 

Also, an addition for the part where I say Control Transfers are more periodic as it can be read a couple of different ways.  Control Transfers are scheduled by best effort, which means not synchronous and not guaranteed latency.

I went through my documentation again last night and read this as well.  This would perfectly explain the "random" behaviour as seen with the affected chipsets.  I'm guessing the UHCI handles these transfers in a much less aggressive manner. 

But I am still curious as to why the addition of the aforementioned gross between transaction delays did not seem to fix the problem.  Unless, as I think Mike had surmised, the transactions were being held in a buffer longer than they perhaps should have been and then released 2 at a time.  If this is the case, even though it would technically be "spec legal", it wouldn't sound like a very good way to do things.

RandyT


*edit* typo

[2600]

Woops, typed this up before Randy's post above re-edited a little, but not a lot.

Ok, ran my other box with a test last night.  The OHCI failed as expected, but transfers were a little different.  I switched the view in my analyzer from grouping transfers to grouping transactions so you can easily see what is going on per frame.

Then realized where I think Randy and I were mixing up our bits and bytes while talking and saw why and may of been also because of intermixing terminology, I tend to do that.  Also, I was wrong about you getting 32 bytes during the hub example.  You are getting 24.  setup, data, then the next setup in the hub example.

In the OHCI, example. The transactions are still grouped, setup and data.  But the next transfer is not grouped with it.  Instead it is just close to the last transfer.

So, I'm not sure if it's one or two problems.  Grouping of multiple transactions or transfers together and multiple transfers so close to each other.  The UHCI doesn't exhibit this because it doesn't group anything i.e. Much less aggressive.  You'll see in the pics.


I'll send Randy and Mike the latest captures with it grouped on transactions so you can see the problems easier.  I'll also include a capture of what I think is test 2 and test 3 so you can see what the Mike's fix did.  Luckily, I haven't seen any problems during enumeration which is also control transfers.  So if that keeps holding true, and the software updated for the accommodations everything should be good.

[arzoo]

Just as an FYI, NewEgg.com has a VIA based USB/PCI card for $5.99 (plus $4.99 shipping). It solved my LEDWiz problem!

http://www.newegg.com/product/product.asp?item=N82E16815124007