Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Lightguns Arcade1Up --- Bug Reports --- Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news

  

Author Topic: DIY projects with other than IR led tracking  (Read 715 times)

0 Members and 1 Guest are viewing this topic.

sniiki

  • Trade Count: (0)
  • Jr. Member
  • **
  • Offline Offline
  • Posts: 9
  • Last login:September 26, 2020, 02:35:30 pm
  • I want to build my own arcade controls!
DIY projects with other than IR led tracking
« on: August 10, 2020, 02:40:23 pm »
Not that the Sinden gun price would be bad at all, but just wondering if, even just for the joy (and rage) of tinkering anyone else has been experimenting with RGB/BW cameras using something else than IR leds to track the monitor?
I did some crude tests with OpenCV and the basics were easily there but not really sure if OpenCV would allow fast enough algorithms out of box.

With 176x144 RGB video the tracking test takes ~15ms on my old i7 Ivy Bridge, which is a probably too much (no idea of the delays in other tracking systems). This time includes grapping the pic from an URL, preprocessing it, finding the screen rectangle, sorting the corners and calculating perspective transformation matrix to map the center point of camera on that found screen plane. Now, it's a sucky test code and I actually know nothing about OpenCV but I thought it was at least somewhat promising and therefore thought there must be some other tests around this forum?

Here's a vid just for the fun of it. Huge delay was mostly from using old cell phone as an IP camera over WiFi as I couldn't find any non integrated webcam and also for this retarded demo I used a larger 1080p source video so the total processing time went up to 250ms or so.



« Last Edit: August 10, 2020, 02:44:26 pm by sniiki »

Howard_Casto

  • Idiot Police
  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 18764
  • Last login:Today at 03:11:01 pm
  • Your Post's Soul is MINE!!! .......Again??
    • The Dragon King
Re: DIY projects with other than IR led tracking
« Reply #1 on: August 10, 2020, 03:57:18 pm »
If one is using a camera, IR filtering is probably the best way to go.  Basically it filters off a buttload of image data before the camera even starts processing, which increases responsiveness.  We were discussing trying IR lasers and a stationary camera last year but we had trouble sourcing a ir laser powerful enough that wouldn't destroy our retinas. 

JayBee

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 320
  • Last login:Today at 11:14:34 am
  • If I can't buy it, I build it.
    • Github
Re: DIY projects with other than IR led tracking
« Reply #2 on: August 11, 2020, 02:48:46 am »
Very nice tests you did there.
Yeah the amount of data to process and all the issues that can come from this solution aren't easy to overcome I guess.
If one is using a camera, IR filtering is probably the best way to go.  Basically it filters off a buttload of image data before the camera even starts processing, which increases responsiveness.  We were discussing trying IR lasers and a stationary camera last year but we had trouble sourcing a ir laser powerful enough that wouldn't destroy our retinas. 
But would IR filtering be really useful here without any IR emitting source?
I also tried the laser stuff a bit, and true, it's not working very well, especially with non glossy screen that don't reflect the laser enough.

sniiki

  • Trade Count: (0)
  • Jr. Member
  • **
  • Offline Offline
  • Posts: 9
  • Last login:September 26, 2020, 02:35:30 pm
  • I want to build my own arcade controls!
Re: DIY projects with other than IR led tracking
« Reply #3 on: August 11, 2020, 05:44:14 am »
Thanks for the comments. Initial driver to experiment was my earlier experience on IR tracking a long, long long time ago with FreeTrack and with some issue with environment light. That said, I don't think these IR light guns have the same issue as there should not be any light sources shining at the screen anyway. Still, fun to just try new stuff and I think I will experiment more but need to get a better camera (maybe one that provides non Bayer filtered image to minimize the delay). Using gyros along with the camera tracking could help with the response times but then again if current tracking options have no significant delay, why bother.

I found that laser discussion of yours and it was indeed an interesting idea but yeah, LCD is a nasty surface to detect the beam.


JayBee

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 320
  • Last login:Today at 11:14:34 am
  • If I can't buy it, I build it.
    • Github
Re: DIY projects with other than IR led tracking
« Reply #4 on: August 11, 2020, 06:36:27 am »
Thanks for the comments. Initial driver to experiment was my earlier experience on IR tracking a long, long long time ago with FreeTrack and with some issue with environment light. That said, I don't think these IR light guns have the same issue as there should not be any light sources shining at the screen anyway. Still, fun to just try new stuff and I think I will experiment more but need to get a better camera (maybe one that provides non Bayer filtered image to minimize the delay). Using gyros along with the camera tracking could help with the response times but then again if current tracking options have no significant delay, why bother.

I found that laser discussion of yours and it was indeed an interesting idea but yeah, LCD is a nasty surface to detect the beam.
Having various solutions is always a good thing for innovation  ;D
Yeah parasite lights is one of the most annoying thing in the IR led solutions. If we can find a way to remove this issue without losing the overall stability and speed, that'd be great.
I'd recommend a camera of at least 120fps to minimize the latency.

Howard_Casto

  • Idiot Police
  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 18764
  • Last login:Today at 03:11:01 pm
  • Your Post's Soul is MINE!!! .......Again??
    • The Dragon King
Re: DIY projects with other than IR led tracking
« Reply #5 on: August 11, 2020, 01:57:11 pm »
Very nice tests you did there.
Yeah the amount of data to process and all the issues that can come from this solution aren't easy to overcome I guess.
If one is using a camera, IR filtering is probably the best way to go.  Basically it filters off a buttload of image data before the camera even starts processing, which increases responsiveness.  We were discussing trying IR lasers and a stationary camera last year but we had trouble sourcing a ir laser powerful enough that wouldn't destroy our retinas. 
But would IR filtering be really useful here without any IR emitting source?
I also tried the laser stuff a bit, and true, it's not working very well, especially with non glossy screen that don't reflect the laser enough.


I'm not exactly sure what you are asking.  Blocking out everything but IR light essentially makes it a black and white camera that seldom has any white in the image but the targeted tracking points.  So yeah, on paper at least that really makes it easier to process than, let's say a full color image where a very specific color is the tracking color.  It's just less stuff to parse.   

sniiki basically asked why IR was being used.... well amongst other reasons that's why. 

JayBee

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 320
  • Last login:Today at 11:14:34 am
  • If I can't buy it, I build it.
    • Github
Re: DIY projects with other than IR led tracking
« Reply #6 on: August 11, 2020, 11:42:07 pm »
Very nice tests you did there.
Yeah the amount of data to process and all the issues that can come from this solution aren't easy to overcome I guess.
If one is using a camera, IR filtering is probably the best way to go.  Basically it filters off a buttload of image data before the camera even starts processing, which increases responsiveness.  We were discussing trying IR lasers and a stationary camera last year but we had trouble sourcing a ir laser powerful enough that wouldn't destroy our retinas. 
But would IR filtering be really useful here without any IR emitting source?
I also tried the laser stuff a bit, and true, it's not working very well, especially with non glossy screen that don't reflect the laser enough.


I'm not exactly sure what you are asking.  Blocking out everything but IR light essentially makes it a black and white camera that seldom has any white in the image but the targeted tracking points.  So yeah, on paper at least that really makes it easier to process than, let's say a full color image where a very specific color is the tracking color.  It's just less stuff to parse.   

sniiki basically asked why IR was being used.... well amongst other reasons that's why.
Just quoting what Sniiki was saying:
[...]using something else than IR leds to track the monitor?
He wasn't talking about IR solution at all, but instead screen tracking solution. Your suggestion of adding a filter only works with IR tracking points. Not in this case.

lightgungamer

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 57
  • Last login:September 24, 2020, 02:43:28 am
  • I love playing Light Gun Games
    • LightgunGame.com
Re: DIY projects with other than IR led tracking
« Reply #7 on: August 12, 2020, 10:51:03 am »
Hey. That looks really good! Iím not sure that using a banana for a gun will catch on but what you have on the screen looks excellent.

I would guess that the lag could be reduced but how consistent is it at finding the screen accurately?

Would it work well on different monitors or does it need to be coded for the one you are using?
https://LightGunGamer.com - every Lightgun game and how to play them
Dolphin Emulator

caiom

  • Trade Count: (0)
  • Jr. Member
  • **
  • Offline Offline
  • Posts: 1
  • Last login:September 25, 2020, 01:24:42 am
  • I want to build my own arcade controls!
Re: DIY projects with other than IR led tracking
« Reply #8 on: September 23, 2020, 01:34:09 am »
One of the biggest challenge in this case is the latency - the time from when you hit the trigger to the time the mouse position is received. It is really hard to find a USB camera with low latency, I don't think there is one easily accessible.
After that you still have the update rate (camera FPS), image processing time and so on...

Your processing time is 15 ms, which is nothing compared to the total latency (~1000 ms). The Sinden processing time is 7 ms, but I guess the Sinden's total latency should be in the range 50-100 ms. So I don't expect good latency from Sinden.

I also thought of using a IMU (accel, gyro) to estimate the moviment, so that the camera gives an absolute position and the IMU gives a relative one. This is analogous to a car positioning system (GPS gives the absolute position). I guess it is not hard to find a 1000hz IMU. In this cenario, due to the image grabbing delay, we would never use the actual image information to update the current position, rather we will use it to correct a past position and propagate this correction to the current positon using past reading of the IMU sensor. What do you guys think?

One, question, how bad the IR based solutions latency?

Best regards,

« Last Edit: September 23, 2020, 01:58:26 am by caiom »

JayBee

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 320
  • Last login:Today at 11:14:34 am
  • If I can't buy it, I build it.
    • Github
Re: DIY projects with other than IR led tracking
« Reply #9 on: September 26, 2020, 06:26:08 am »
One of the biggest challenge in this case is the latency - the time from when you hit the trigger to the time the mouse position is received. It is really hard to find a USB camera with low latency, I don't think there is one easily accessible.
After that you still have the update rate (camera FPS), image processing time and so on...

Your processing time is 15 ms, which is nothing compared to the total latency (~1000 ms). The Sinden processing time is 7 ms, but I guess the Sinden's total latency should be in the range 50-100 ms. So I don't expect good latency from Sinden.

I also thought of using a IMU (accel, gyro) to estimate the moviment, so that the camera gives an absolute position and the IMU gives a relative one. This is analogous to a car positioning system (GPS gives the absolute position). I guess it is not hard to find a 1000hz IMU. In this cenario, due to the image grabbing delay, we would never use the actual image information to update the current position, rather we will use it to correct a past position and propagate this correction to the current positon using past reading of the IMU sensor. What do you guys think?

One, question, how bad the IR based solutions latency?

Best regards,
Not sure about the other solutions, but my IR solution has a total processing time of around 7ms thanks to the very fast IR cam we use and the calculation I do ;)
I still have to find way to measure the total latency accurately, but from my tests it's under 100ms.

One of the few advantages IR/camera based solutions have compared to common crt tech is that the position is refreshed all the time from the cam, so whenever a button is pressed both button press and position are sent right away.
With crt tech once the button is pushed, you need to wait for next full frame to become white and check when the tv beam hits the sensor. So depending of the tech, it can take as long as 3 frames (not including the game and controller latency), and is harder to handle because it relies on screen resolution and timing.

You're right about the normal cam solution, it's quite hard to have a good enough camera, and be able to process as many frames at it would need to get good latency.

It's funny you're talking about IMU, I'm actually trying to add it to my solution too, to make detection faster and more accurate.