VR lighthouse based tracking for lightguns

[Toasty833]

Works great, I upped the X borders to 480 so shots in the black borders at 4:3 would be counted as off-screen too and that works fine. The only edge case issue I could find is if you hold down the trigger, enter the border/exit the screen, then return to the screen, it'll send a reload. But this is so minor it isn't even worth fixing, I doubt anyone would run into it and it doesn't cause any problems. No issues with forced scaling, maybe I'll wind up using DS more now, I think HotD3 might have some slight control issues for route selection without it.

[greymatr]

Awesome, glad to hear it works.


I was anticipating a problem like that. But it would've been when clicking in off-screen and re-entering screen area before releasing the click could've caused reload to be stuck on.

But I figured out a way to handle that by releasing both left and right click. Which was a better way than how I first planned to handle it.

I didn't think of what you found but I agree probably not worth fixing atm.

Any more things to fix while I'm on a roll? lol 😆

[ThatOneSeong]

By the way, thanks for that reply ThatOneSeong. It was enlightening. I think choices like this get made earlier on in making a program and then get harder and harder to fix later on. So it just becomes the way it is.

Right.
Keep in mind that I'm not trying to be too harsh (even though I did just call it "dumb") - all the lightgun emulation code was, in all likelihood, developed all the way back in the mid-2000's - when most people already were running 4:3 displays/didn't have a PC lightgun solution at all at the time, so the issue wasn't as prevalent for the person implementing it to take action. I wouldn't be surprised to hear if there's some underlying tech debt left in MAME's code that makes it difficult to resolve--and/or just not enough interested persons around to fix the issue, probably from complacency in the PC light gun community... coughGun4irCough.

Regardless, glad to hear that RetroArch's been working for y'all!

(And for those of you who haven't used DemulShooter yet, yes, it relies on hooking directly into the individual mouse object - that's how it gets multi mice to work after all. So I'm guessing you'll want to load it up while your software is running? The -useSingleMouse option effectively just has DS use the general Windows cursor. I only know this because of having to use it as a workaround on Linux, since Wine currently does not expose individual mouse devices yet.)

[JayBee]

probably from complacency in the PC light gun community... coughGun4irCough.

So now the Mame "flaws" are our fault? If you get an issue with my community, you can say it to my face, I'm right here 

You are misunderstanding the "issue" with mame, which isn't an issue at all but just the way it was designed to work.

Mame uses real "raw" inputs, where the gun coordinate is directly sent/converted to the game input without any kind of modification.
It allows much lower latency than retroarch, but means of course if the screen coordinate isn't the same as the game video coordinate, the aim won't be correct.
Reason why I implemented fix directly in the GUN4IR that does not add any latency.

Retroarch on the other end, while technically using raw input too, doesn't send the input data directly, but instead does various conversions first depending on your video/screen modes.

That means the Mame way must be "dumb" and outdated right?
Yes and no.
Yes because I am sure Mame could do it better, and we already tried to talk to the devs about it and bring interest to them, the same way we did with so many other emulators.
No because fixing the aspect ratio in the gun itself allows for far less input latency than the method used by retroarch, and allows native offscreen reload on the black bars.

[Migcar]

Hello everyone, I had already started a similar project in Unity and posted it in a topic here on the forum, but I abandoned it for various reasons. I then started a different project, for now, I won't say what it is, but it contains useful code related to the topic of this thread and which I used to create this small program, this time using OpenVR and DX11. If you want to take a look, here is the executable, and there is a readme in the archive.

https://mega.nz/file/LN8XkBrQ#5-wU8IidRlo8BArTf7sfHuE8M1FchpOM_SnBOHhm9dY

[Migcar]

Ok, so no one has tried it... but I understand. Almost certainly, I too would be cautious about running an executable on my PC posted by a stranger with 3 messages who pops up out of nowhere and doesn't include anything, not even a demonstration image. So, I apologize and introduce myself... I'm Migcar . In my 'curriculum' there's not much on the internet. A few years ago, I was credited in a release of DoomVR for helping get the weapons working in Heretic: https://www.reddit.com/r/WindowsMR/comments/92zrbk/doom_vr_new_release_now_with_heretic_support/. I also made a tutorial for modifying weapon positions in DrBeef's Half-Life VR mods. That's about it, not much. To make up for it, I'll first post a video of the program in action and then reveal my 'secret' project: it's simply a program like Virtual Desktop, but designed to control a light gun in VR. I have quite a few cool ideas in mind, including playing games in stereoscopic 3D, using Geo11 or Dolphin's 3D mode and anything in SBS. I'll upload 4 videos on Mega featuring Point Blank, Time Crisis in 3D using DuckStation and Geo11, House of The Dead, and Gunblade New York, both in 3D. In the Wii games, you'll notice that it's possible to achieve 1:1 aim in all games. I'm putting them on Mega and not on YouTube because they need to be viewed in VR, ideally with a program like Virtual Desktop or DeoVR in dome mode at 20m. If you feel like testing the program, there's no virus, don't worry. Besides, I wouldn't be capable of creating a virus that antivirus programs can't detect, nor am I interested in doing so. Criticisms and suggestions are welcome.



Point Blank Mame: https://mega.nz/file/vAFzxIjb#uL5hFYXmBKX4al611J2Ju2APD0dSBDXuhjYiavOPLrA
Gunblade New York Wii: https://mega.nz/file/OccDhYyK#v-uMCxtlU1ADFvMwL0SK4NX1VMHTTqUBRmPuaft3scU
House Of The Dead Wii: https://mega.nz/file/HI8WkZpB#1ofr-4EAbqejbebNthmeiAqDZmyS2VAa-djfVlyc5-o
Time Crisis PSX : https://mega.nz/file/vB9EkC4A#NBxHQ3dz7YP0nU5RtyxyAgpj6WGjtzhnVBI6fssQu-A

[greymatr]

Hi Migcar,

So after seeing your video and seeing what looked like a Quest 2 controller (I guess it was a Pico controller actually). I thought I'd give it a go with my Quest 3 (I originally thought it was a Vive Lighthouse thing).

Anyway, I can't get past calibration, or rather get a decent calibration. The virtual plane never looks right in the headset and I'm not sure setting the corners is working right either, I seem to just end up highlight text on a web browser or something when I pull the trigger. (I do get the controller vibration when pushing left or right though). The mouse pointer won't line up where I'm pointing and sometimes it's even moving in the wrong directions.

I think part of the issue is not knowing exactly how to align the controller exactly to the monitor. Maybe it is easier with your controller and your monitor but I have no guide to know I'm pointing exactly in line with it.

Some other software do their calibration differently, like the Vive Mouse dos program uses a controller position in each of the 4 corners of the screen. Or with Virtual Desktop it is possible to get the virtual screen in the same position as the real screen and adjust it's size so it all overlays, and peeking till you get it right.

I think the virtual desktop way would be easiest if you could also use passthrough AR mode on the headset. I can see what you are trying to do with this method as it is less work but for me it doesn't seem accurate enough with only the one reference point.

[Migcar]

Hi Greymatr,

First of all, thanks for taking the test. I probably didn't explain the calibration phase well. The first thing to do is to aim the right controller, making sure the shooting line is aligned with the TV/Monitor screen, and pull the trigger. This way, the plane on which your TV resides is "captured."

Now, if you look in the headset, you'll see a rectangle. It won't seem to be in the same position as your TV, but that's normal; the space you see in the headset is not 1:1 with the real space. This is shown by the fact that you physically went next to your TV and created the rectangle exactly there (because the rectangle, when created, starts exactly from the position of the controller, and you can verify this by creating it with the headset on). Yet, if you look in the headset, it seems closer.

So, don't worry about the position of the rectangle you see in the headset; just make sure it has the same orientation as the TV. This is important. One thing I forgot to write in the readme is that to create the plane, you need to aim the controller at the side of the TV that corresponds to your left when you are looking at it. Basically, the right side of the controller points towards the viewer. I hope I've explained this clearly; anyway, in the video, you can see where I place the controller. If you place it on the other side, it's as if you have the plane inverted. This might explain why your mouse was moving in the wrong direction.

OK, now you can set the top left corner and the bottom right corner. You can do this in any order and at any time. Don't look at the rectangle in the headset anymore because it is no longer updated; as I said, it is only for checking orientation rather than position. To set the top left corner, push the analog stick to the left, you'll feel the controller vibrate, aim at the top left corner of the TV, and shoot. To set the bottom right corner, push the analog stick to the right, you'll feel the controller vibrate, aim at the bottom right corner, and shoot. The concept, in the end, is that for calibration, you only use the real space.

I hope I've clarified well how to set the two corners because maybe the main problem you had was there. Yes, I have a Pico 3, but all controllers should work fine because they all have the same "forward."

Another thing I forgot to mention in the readme: the program assumes you have a desktop resolution of 2560x1440. In the next version, if it's worth continuing development, I will ensure that you can select the resolution, or better yet, that the program automatically detects it.

And Yes.. it was a Vive Lighthouse thing, but since the program uses SteamVR, we're not completely off-topic, at least I hope so.

[greymatr]

I just gave it another go.. Aiming the right controller at the left side of the screen does make a big difference. That was definitely the problem I was having.

The second issue I had though was that I wasn't doing the corner calibrations properly. I thought it was that you pushed left or right, it vibrated and that was like an indication it was ready for the trigger. I.e. I thought you let go of the stick and did the trigger afterwards.

So I worked out you need to pull the trigger while holding the stick left or right.

But the third problem I have is I need to put the headset somewhere. It gets in the way and it seems like moving it around affects how the mouse moves or is positioned. It also seems to it probably needs to be in the direction of the screen.

So I'd probably have to come up with some kind of mounting solution. I'm just using a screen on a desk with a chair.

[Migcar]

Great! I'm glad we've clarified how the calibration works. As for the headset, it should simply be positioned to detect the controller. I place mine on the desk facing me. Personally, I'm fine with the Pico 3 why even when it's not worn—it doesn't go into standby completely and continues to receive controller position data. However, with other headsets, you might need to place something near the proximity sensor to make it think it's being worn. Interestingly, the Oculus Rift S, at the firmware level, so doesn't allow any workaround, after a certain period of inactivity, it goes into standby, so occasionally you have to move it.
 I'm achieving good results—I played Point Blank on Mame yesterday and even beat some scores I had achieved with the Lightgun. In my opinion, with a well-set pistol-shaped cover for accurate aiming during calibration, excellent results can be obtained. One issue I'm encountering with the motion controller is the physical feedback when pressing the trigger compared to something like a GunCon—it's an entirely different experience. Now that the adapter for PSVR2 is coming out, I'm curious to see if we'll be able to programmatically control the functions of the Sense Controller's trigger.

[greymatr]

I tried it again. I can confirm that putting the headset on the desk facing me does work. I didn't have much luck using it in game though. I think it may be configuring issues but I was getting frustrated and cut testing short.

Two questions though, is there a way to send a right click so that we can use reloading in games and is there a way to pause the tracking? It would be good to pause the tracking and regain the normal mouse function when using windows or menus.

As far as I've heard about PSVR2 the adaptive triggers won't be available on PC but that may just be on launch or maybe hackers will find a way to get it going.

[Migcar]

Hello, I’ve made the changes you requested. Now, if you leave the controller idle for a while, it loses control of the mouse. And if you shoot off-screen, it simulates a right-click. What issues have you encountered and with which games/emulators?

https://mega.nz/file/qcVGCapb#I5gDme6f3FUT-Qs_a88lxzMwu5sgl4MaMa5Xe7UwXdU

[greymatr]

Thanks for doing the changes. Unfortunately, it doesn't seem to work for me, I don't know if it's the Quest perhaps or something to do with my setup. It will track for a bit and then just seem to start being really erratic. I'm just trying to see the tracking on the desktop with the mouse cursor.

So I try to pick different spots and then aim where they are with my hand and the cursor will move toward it but often not go in a straight line and will arc or pull back and it feels like I'm trying to fight it.

I tried moving further away from my desk and that seemed a bit better so maybe I'm too close to my monitor. Or the headset on the desk is too close to the monitor. Or some combination of those things.

I tried Virtual Desktop out, usually I'm using the app version on the Quest. But I had got the desktop version to do the Vive things in this thread. It seems to work with the Quest in Airlink mode which is good and I can do the same thing with putting the HMD on the desk facing me. The tracking on that is a lot smoother but it is more fiddly to setup as you need to hit Reset Calibration with the headset lined up to the monitor and then adjust the size and distance of the virtual screen while peaking back and forth to try and match it to the real monitor.

The biggest problem though is the one you mentioned which is that the headset goes to sleep after a while, maybe 10 minutes. I thought maybe I could trick it with a vibration timer on my phone and put it under the headset but it didn't work.

[Migcar]

Yesterday, I tried the Quest 2 with Airlink and remembered why, instead of buying a Quest 3, I bought a Pico 3 Neo Link... latency... there's terrible latency. I also tried it with the cable; it was a bit better but still not enjoyable at all. Maybe that's the problem, my code isn't very optimized, and it works fine on my headset that has a direct connection to the DisplayPort, so no latency and data compression. I tried something; I disabled all the drawing in the headset and made some optimizations. I tested it with the Quest 2 + Air Link, and it's much better. I'll give you just the executable; you can overwrite the file you already have. As for the headset going into standby, one solution (not very good) would be to keep the headset on but lifted above the eyes so that you can see.

https://mega.nz/file/6INTxLBR#IJchCDpe8jug-qyQz0VDScncNmqTwY6sn74W-g8iXSU

[Migcar]

I think I understand what might cause some problems: the two-point calibration assumes that the TV is horizontally aligned with the horizontal axis of the headset space, which may not be the case. Therefore, I should implement a calibration with at least 3 points, the first 2 at the top left and top right to establish the screen's length and its tilt relative to the horizontal axis, and a third point at the bottom (right or left) to determine the screen's height. In the coming days, I'll try to make these changes.