VR lighthouse based tracking for lightguns

[Toasty833]

Lighthouse tracking, used with the HTC Vive and Valve Index VR systems, has an accuracy of within a few mm, and they come with 2 controllers out of the box. It wouldn't be too hard to 3D print or strap some gun sights to the top of a controller, define your screen area and positional offsets, and create a lightgun with no distance requirements by just ignoring the headset.

There was an attempt made here: https://www.reddit.com/r/Vive/comments/751j8d/vive_controller_as_a_mouse_pointer_v03/ to do something similar, but unfortunately SteamVR has since updated and I can't get the software to work to see if the accuracy is good enough for use as a light gun. Current LCD systems all require a minimum distance of about 1.5x screen size to work, which isn't so bad on smaller screens, but if you want to play on newer 4k screens you might have to stand very far away. With a tracked gun, it doesn't matter where the gun is in relation to the screen, you can be an inch away and get the same performance.

[Toasty833]

Went looking around for this kind of thing again today, and found https://imgur.com/gallery/vBaXSkW, using openVR (https://github.com/asingh57/openvr_lightgun). Very user unfriendly, but I managed to get it working with my Vive wand. In terms of tracking accuracy, it's fantastic, dead on the same mark every time, very low latency. The problem is with calibration, the tool is very basic and doesn't really let you dial in the screen position properly, a proper GUI could offer crosshairs and multi-angle shooting to define the screen much more accurately. And this was clearly early software. I also couldn't get it to compile so I had to use the debugging version with clicks disabled, but I wasn't really expecting it to be feature complete, I just wanted to know what kind of accuracy was possible.

https://imgur.com/a/ZVay1zS is a quick example of the tracking quality from about a foot away, which isn't possible on any existing LCD based setups to my knowledge. The alignment issues are all down to the shoddy calibration, what's important is how well it follows the movement.

[Toasty833]

Well, it's been another 2 years, and I went looking around again. Nothing has changed in that timeframe, I've seen a couple of WIP projects for this kind of thing pop up and then fizzle out without any release.

But today I read a comment somewhere about how you could just use Virtual Desktop, align it to your real screen, and then use a VR controller as a light gun on your monitor. After about 20 minutes of fiddling with distance, taking the headset on and off, I honestly got it pretty close. Not perfect, but it was about 5mm variance for most of the screen. The main problem is my 40" monitor is shiny and reflective, which throws off the tracking, I imagine other screens would work better. More than responsive enough to play with a cursor though, so I played through a bit of Blue Estate, which worked great. Only issue is that one of the buttons in VD seems to be mapped to moving the screen around if you hold it by accident, and obviously you don't get anything fancy like off-screen reloading.

I might try and mess with this concept a bit more. I've got a tracker puck, I might try and attach it to the bottom of a guncon, use a bit of card in front to block reflections, and hook up the buttons via the pogo pins. It'd be really nice if I could somehow calibrate the virtual mouse that VD creates, but that might be beyond the scope of the software, and find a program that takes any left click inputs near the edge of the screen and converts them into an alternate button press for quasi off-screen reloads. Maybe it's something AHK can do with a macro?

EDIT: Got the system half working with a vive wand, which seems to be a lot more robust for tracking than the index controller I was using yesterday, tracking is much more stable even with a reflective monitor. Unfortunately, I couldn't work out a way to remap any of the vive buttons, so you're stuck with just the left click trigger. In order to get the vive wand at a usable angle (by default VD has it coming out straight like a lightsaber), I used an updated version of OpenVR input emulator found at https://github.com/Louka3000/OpenVR-InputEmulator-Fixed to adjust the angle to be more gun like. Sadly, I couldn't get the input remapping element of this tool to work, and attempts to use the SteamVR remapper to add control options proved fruitless, so I used a keyboard button to reload instead. My hope is that with a vive tracker I get similar tracking performance and usable iron sights after attaching it to a gun shell, and maybe I work out a way to get the pogo pin inputs as usable buttons.

[greymatr]

Hi Toasty,

I tried the vive tracker as a light gun route myself and used the precompiled vive controller as a mouse pointer as well. I ended up getting this gun (Hyperkin Hyper Blaster) to attach the vive tracker to:



https://unboundxr.eu/hyper-blaster-vr-gun

https://www.amazon.com/Hyperkin-Hyper-Blaster-Vive-Tracker-pc/dp/B077Y9WBDZ

It did all work in the end but it was a lot of mucking around. I ended up getting an off the shelf solution that uses 4 IR emitters around the TV / Monitor.

[Toasty833]

I tried the vive tracker as a light gun route myself and used the precompiled vive controller as a mouse pointer as well. I ended up getting this gun (Hyperkin Hyper Blaster) to attach the vive tracker to:

How did you get the calibration to work? For me it's always all over the place, and there's no good way to adjust it.

I saw the hyperkin guns, but importing one would've been overly expensive, and they mounted the tracker on the top so you can't use the iron sights without heavy modification anyway. I found it easier to just mount to a guncon, you can put a tracker in the hole for the cable with minor modification to fit a tripod mount, and then you can use the iron sights and wrap a couple of wires around the pogo pins for a trigger and A/B. I've tried some alternatives, but the required minimum distance is just too much for my setup, even with fish eye lenses. I could use a smaller screen or move further away, but I want to play on a big screen, and that requires both a large screen, and a low minimum distance. Right now I'm playing with my gun barrel about 9 inches from the display, I think even camera based laser detection would have problems here due to occlusion.

As a basic example, at this distance on a 4:3 game I have a horizontal view angle of 50 degrees. If we assume you can get about 1x diagonal resolution away from a screen with a fish eye, and you're playing with an outstretched arm instead of shooting from the hip, you would need to be around 60 inches away (40 inches due to screen, 20 due to distance from barrel to eye), giving you a view angle of 26 degrees, or almost half as small a picture. With a much larger screen, barrel to eye distance gets smaller proportionally, but even with a 200" display (which would give you a 163" 4:3 screen), you'd still only manage a 33 degree horizontal view angle.

Obviously, there's an argument to be made about what view angle is good enough. Time Crisis 3 deluxe had 50" 4:3 screens with the player about 60" away from the screens, which corresponds to a horizontal view angle of 37 degrees. Standard cabs had smaller screens with lower view angles and are still totally playable, but I'm looking for arcade level without having to rent a warehouse to fit a 300" screen in so I can stand 27 feet away.

[greymatr]

It's been a while since I played with it but I used version 04 of the software you linked in your first post in the reddit thread. Which I found by changing the URL to:

http://olli.wtf/vive/

so it is this one:

http://olli.wtf/vive/vive_mouse_v04.zip

So I run the config app I think and then I think I chose the option where you put the tracker in the corners of the screen. Being careful to do it in the order that it wants them being done, top, left, right etc, whatever it says.

Pretty sure that gives the best calibration rather than the aiming one.

I also had to adjust the X axis or Y axis value or something by 90 or -90 degrees so that it wasn't aiming straight down.

I had a python program that could pick up the pogo pin keys, there is a library for it. I had to manually map a key for a game but I was going to make it scriptable.

I had it working on a 86 inch tv and was like 4 feet away or something and it was accurate.

If you want I can dig out the gun and setup the light houses and do it again. Record a video or whatever.

You are right about the hyperkin gun though, the iron sights thing isn't good.

[Toasty833]

So I run the config app I think and then I think I chose the option where you put the tracker in the corners of the screen. Being careful to do it in the order that it wants them being done, top, left, right etc, whatever it says.

Pretty sure that gives the best calibration rather than the aiming one.

I gave it another try with 04 and I've got a better calibration now, but it's still not ideal, out in a few places. I could probably try it a few times to get a good set where the tracking is more accurate (bringing my tracker that close to the screen creates tracking uncertainty due to the reflective nature of the display, so I just have to get lucky with the position being accurate 4 times), but I don't know if it's worth it.

I had a python program that could pick up the pogo pin keys, there is a library for it. I had to manually map a key for a game but I was going to make it scriptable.

This is one of the big issues with vive mouse that I can see, the trigger feels a bit weird like there's a delay for precision or the dragging function, there's no off-screen functionality, and it takes total control of the mouse cursor so you need either a kill script with AHK or something to end the program, or to sacrifice a button on your gun for the menu button that toggles the cursor tracking on and off, and you need a secondary program to do basic stuff like map right click. I think ultimately virtual desktop just feels a little better in the cursor even though you have to set it up each time, because I still need to use input emulator to set my tracker coordinates in line with the gun barrel anyway (I can't seem to find a way to automate this through OVIE, you can save profiles but I don't think you can auto-apply them on boot). Vive mouse offers static calibration, but everything else just feels a little cumbersome. I think I'd have to use something like OpenVR2Key to remap the pogo pins, and that doesn't offer mouse clicks so I'd have to map to keyboard keys which brings a new host of incompatibilities.

[greymatr]

I had a look at what I did and I remember now that I looked into the 04 exe file and I found that it sends the mouse clicks by hooking into USER32.DLL and accessing some windows functions.

So to stop it from doing that I used a C++ program to intercept the vive_mouse.exe process and redirect it to a dummy DLL file for the USER32.DLL calls. (I got AI to give me most of the source code to do it)

It has to be called after starting the vive_mouse exe.

If the calibration can be static (saved) could you put a sheet or something over the TV to stop the reflections just for the calibration phase?

I had a look at the default.config file that the vive_mouse_config.exe creates and it is a text file. The x,y,z co-ords could be adjusted manually or copy pasted from better calibrations.

This is the python library ( triad_openvr ) I used to access the data of the tracker (x,y,z positions, triggers and other pogo pins i.e. buttons of hyperkin):

https://github.com/TriadSemi/triad_openvr

Once I detected the trigger, I made my own calls to user32 dll using python. I don't think I got to doing the other buttons but I could see them being pressed. Right click and keyboard should be available through user32 as well.

I remember I did this for that same reason you were saying, pulling the trigger felt bad and would sometimes be dragging causing issues.

I was trying to write my own calibration and screen co-ordinate system in python but ending up finding that way to make vive_mouse 04 work.

If you want any source code I can send it but it may be a little rough.

By the way, what VR headset are you using Virtual Desktop with?

[Toasty833]

If the calibration can be static (saved) could you put a sheet or something over the TV to stop the reflections just for the calibration phase?

This is a smart idea, I hadn't considered it.

I had a look at the default.config file that the vive_mouse_config.exe creates and it is a text file. The x,y,z co-ords could be adjusted manually or copy pasted from better calibrations.

Yeah, but it's a lot of fiddling. Ultimately, even if I could get a perfect calibration through Vive Mouse it feels like a little more trouble than it's worth. OpenVR2Key works well for remapping additional buttons to keypresses, and I imagine I could just use JoyToKey if I need right click for a game. I've also changed my tracker to be recognised as a controller, which is more useful, so some things that apply to trackers might not work with a spoofed controller.

If you want any source code I can send it but it may be a little rough.

I could take a look at it maybe, but I'm pretty incompetent when it comes to coding. It might be useful to have up anyway in case anyone else is interested in this kind of thing in the future and can make more use of it than me.

By the way, what VR headset are you using Virtual Desktop with?

At this point, an Index. I'm still using 1.0 base stations because they seemed to be better built and I can still use my Vive stuff.

[greymatr]

Yeah, but it's a lot of fiddling. Ultimately, even if I could get a perfect calibration through Vive Mouse it feels like a little more trouble than it's worth. OpenVR2Key works well for remapping additional buttons to keypresses, and I imagine I could just use JoyToKey if I need right click for a game. I've also changed my tracker to be recognised as a controller, which is more useful, so some things that apply to trackers might not work with a spoofed controller.

Actually I think I may have changed my tracker to a controller as well, probably to get it to work with Vive Mouse or something else.

So what do you mean to use OpenVR2Key to work with, is that to use with Virtual Desktop?

I'm trying to think what would give the best setup for you. I'd be happy to help where I can with programming or testing. If you want to let me know what problems you are having at the moment with it?

I can understand the hesitancy with using Vive Mouse, it gave the best results for me with how far I got with the vive but it's a closed sourced program, if it could be modified it would be a lot better.

I was doing something similar with Virtual Desktop using a Meta Quest VR headset and there was a project posted here by NailBuster that did it with a custom program. You posted in the thread:

https://forum.arcadecontrols.com/index.php/topic,166078.msg1760535.html

I tried to do the same with a program on the quest where you had a 16:9 box and I would line it up with the screen and it could see where the controller intersect the screen and give mouse co-ords via wifi to a pc.

But it was a lot of peeking back and forth in the headset. If I went further with it then the controllers would allow the box to be moved around. But I didn't go that far before switching to the vive stuff.

I haven't tried Virtual Desktop with a vive headset though (I got the original one).

[Toasty833]

So what do you mean to use OpenVR2Key to work with, is that to use with Virtual Desktop?

Yeah, VD gives me a left click trigger but for some reason I can't get use it to bind any of the other buttons with a vive controller, with an index one I can bind buttons to right click but I guess the guy developing it broke vive support at some point (or it never had any) and nobody ever noticed. It's not a big issue because OpenVR2Key lets me bind VR buttons to keypresses.

I'm trying to think what would give the best setup for you. I'd be happy to help where I can with programming or testing. If you want to let me know what problems you are having at the moment with it?

I can understand the hesitancy with using Vive Mouse, it gave the best results for me with how far I got with the vive but it's a closed sourced program, if it could be modified it would be a lot better.

Yeah, maybe you're right. Static calibration is fairly appealing, the problem is just everything else about the program is a bit half-baked and harder to get working than VD. I guess I could use OpenVR2Key and then something else to bind to mouse clicks. I'd also need to work out how to block Vive Mouse from receiving/sending keypresses like you did, seems like I need VS for the right dlls. Then I need to try and get AHK to intercept left clicks and turn them into right clicks if I'm clicking on the very edge of the screen for off-screen reloads. I also realised that vive mouse releases control of the mouse if you point it far enough off-screen, so I don't have to worry about wasting a button on it.

[greymatr]

Yeah, maybe you're right. Static calibration is fairly appealing, the problem is just everything else about the program is a bit half-baked and harder to get working than VD. I guess I could use OpenVR2Key and then something else to bind to mouse clicks. I'd also need to work out how to block Vive Mouse from receiving/sending keypresses like you did, seems like I need VS for the right dlls. Then I need to try and get AHK to intercept left clicks and turn them into right clicks if I'm clicking on the very edge of the screen for off-screen reloads. I also realised that vive mouse releases control of the mouse if you point it far enough off-screen, so I don't have to worry about wasting a button on it.

Can I ask what game or emulator you are trying to get working that uses right click for off screen reload? Most games I've seen use the same left click for firing and reload so it seems a bit odd. Maybe I could try it out.

I think using the Inject DLL code I have it would also be able to not just turn off the clicks from Vive Mouse but also intercept them and resend them. It is probably also possible to get the x,y mouse co-ords and then change whether it is left or right click. That should also remove the need for the python code.

If you have any problems compiling the Inject DLL code let me know

[Toasty833]

Can I ask what game or emulator you are trying to get working that uses right click for off screen reload?

Basically every game works with it. The number of games that don't have an off-screen function is less common, and right click is the standard, basically every gun I've seen sends a right click instead of left click if you click outside of screen boundaries. Only titles like Time Crisis that use a pedal instead don't really have off-screen support. The few titles I've tried so far with just a left click gun would just fire at the edge of the screen if I tried to off-screen reload, but maybe there are a few that see "is gun at border? Then reload" rather than relying on the gun to say "I can't see the screen, send right click instead".

I think using the Inject DLL code I have it would also be able to not just turn off the clicks from Vive Mouse but also intercept them and resend them. It is probably also possible to get the x,y mouse co-ords and then change whether it is left or right click. That should also remove the need for the python code.

If something like that is possible it should be pretty useful.

If you have any problems compiling the Inject DLL code let me know

I used the pre-compiled version in the debug folders and it worked fine for disabling the vive buttons, with the exception of the menu button that still disables the cursor but that isn't really a big issue and I'm guessing it's hardcoded in vive mouse rather than something that's passed through and can be intercepted. With trigger, grip and trackpad that's still 3 usable buttons I can remap through OpenVR2Key because SteamVR still sees them fine.

[greymatr]

Basically every game works with it. The number of games that don't have an off-screen function is less common, and right click is the standard

I had no idea this is how they worked, I thought it was all up to the game to know when it was off screen. Well I certainly learned something and the guns are smarter than I thought!

With trigger, grip and trackpad that's still 3 usable buttons I can remap through OpenVR2Key because SteamVR still sees them fine.

So I had a look at OpenVR2Key, I hadn't seen it before and I didn't realise it was open source. I setup my vive controller but thought I was going to have an issue as you said it needed an Index controller to map and he mentions that's what he has.

But I found he did try to support Vive at one point but the releases page is a very old build. So I compiled the code and was surprised to see it was working now.

I dug around a bit in the code and figured I could probably make mouse clicks work but would need to hijack some keyboard keys to test it quickly and not bother with the user interface side of that.

So I've mapped F1 to left mouse, F2 to right mouse and F3 to middle mouse.

I'm attaching my build of the exe and the source code.

[Toasty833]

So I had a look at OpenVR2Key, I hadn't seen it before and I didn't realise it was open source. I setup my vive controller but thought I was going to have an issue as you said it needed an Index controller to map and he mentions that's what he has.

Ah, maybe I didn't make it clear, I was talking about Virtual Desktop when it came to remapping, it has a bindings page. OpenVR2Key works fine with my Vive controller/tracker even on the newest release.

I dug around a bit in the code and figured I could probably make mouse clicks work but would need to hijack some keyboard keys to test it quickly and not bother with the user interface side of that.

So I've mapped F1 to left mouse, F2 to right mouse and F3 to middle mouse.

I'm attaching my build of the exe and the source code.

But this is pretty handy anyway. The only issue is there's no repeating function which I think is a limitation of the program to begin with, if you hold a button down it only sends the input once, which means it's not very useful for games with machine guns. I found that by adjusting the click detection threshold to 1 (might not be necessary) and the dragging threshold to 10 in the Vive Mouse config, the trigger is a lot more responsive, there's no weird delay. You could probably go even lower than 10ms, I think the code just waits for whatever the dragging threshold is before it actually sends a click which is why it's so slow by default.


The other day I had demulshooter running with VD, but now I can't figure out how to get it to respond to clicks with vive mouse or VD anymore. Might have to restart and see if that fixes it, maybe the blocker stuff for vive mouse affected VD somehow, it might use user32 too to send fake mouse inputs to games. In testing I also think VD just responds better than vive mouse. Virtual Desktop is right in line, there's very little delay between moving the gun and the mouse following. With vive mouse it's like there's maybe a 50-100ms latency, if I waggle it back and forth I can see it lagging behind. I have stabilization in the config set to off and the average set to 1 in case that applies anyway, but it doesn't seem to make an impact. If it were possible to get Virtual Desktop to remember the last window position to essentially save the calibration, it'd just be superior.

[greymatr]

Ah, maybe I didn't make it clear, I was talking about Virtual Desktop when it came to remapping, it has a bindings page. OpenVR2Key works fine with my Vive controller/tracker even on the newest release.

I think I just got confused and I saw the author of OpenVR2Key say he didn't have a vive anymore only index so it made sense.

But this is pretty handy anyway. The only issue is there's no repeating function which I think is a limitation of the program to begin with, if you hold a button down it only sends the input once, which means it's not very useful for games with machine guns.

I've made a change to the OpenVR2Key program and now it will hold down the mouse click while the button is being held down on the controller. I'll attach the updated version.

If it were possible to get Virtual Desktop to remember the last window position to essentially save the calibration, it'd just be superior.

I tried out Virtual Desktop, I had used it on the Quest but didn't realise it was a different app on PC. So I can see how you align it to the screen, moving it's rotation and position and then changing size and distance.

So it seems that Virtual Desktop will save the size and distance settings but when you start it again the rotation and position will be out depending on where the headset is in space.

The only thing I can think of at the moment would be to intercept the openvr_api.dll that is in the program folder and send a saved set of rotation/position co-ords to the virtual desktop exe when the program first starts up. And then it will need to resume normally after that.

I think it's possible but will take some time to work out. Also it will need for a position to be captured while setting it up that can be used to replay via dll as it will be specific to your lighthouse setup / monitor position.

By the way, I also found while searching there is another virtual desktop type app called Desktop+ , it's free on Steam and is also on github with source code.

But I think maybe it's not going to be as good as VD.

[Toasty833]

I've made a change to the OpenVR2Key program and now it will hold down the mouse click while the button is being held down on the controller. I'll attach the updated version.

Works great now. Had no problems using it in Blue Estate (which needs a continuous reload for the shotgun, probably one of the only games that do). It occasionally seemed to send a left click instead of a right click in M2 emulator, but I'm pretty sure that's an issue with the emulator or the game rather than OpenVR2Key, I've had similar issues before when trying to spam, and I can get similar behavior with a mouse by going too fast between left and right click. Still, got me through a full run of VC2 with very little issue, which is something I haven't been able to do in a while.

So it seems that Virtual Desktop will save the size and distance settings but when you start it again the rotation and position will be out depending on where the headset is in space.

The only thing I can think of at the moment would be to intercept the openvr_api.dll that is in the program folder and send a saved set of rotation/position co-ords to the virtual desktop exe when the program first starts up. And then it will need to resume normally after that.

I think it's possible but will take some time to work out. Also it will need for a position to be captured while setting it up that can be used to replay via dll as it will be specific to your lighthouse setup / monitor position.

Yeah, seems like a pain. I find recalibrating VD to not take that long, 2 or 3 attempts get it close enough across most of the screen. Setting up the screen dimensions is the fiddly part, and at least that saves. In my mind the way to do it would be to have something like OpenVR Input Emulator fix the headset position in space to a preset location (with maybe fine adjustment to dial in calibration), click the recenter button on the program to set the screen angle and spacing, and then cancel the headset position so it goes back to the actual one. I imagine if you did it while wearing the headset you'd want to throw up, though.

By the way, I also found while searching there is another virtual desktop type app called Desktop+ , it's free on Steam and is also on github with source code.

But I think maybe it's not going to be as good as VD.

I gave it a try and the tracking latency seems halfway between VM and VD, but the problem with it is that when the headset goes to sleep after 20 seconds it stops letting a controller act as a mouse, so it's no good. I think it also interferes with OpenVR2Key because it's part of the dashboard and OVR2K is blocked there (although I think older builds didn't have this functionality).


I've also noticed that when a tracker is set up as a controller, the pin for the touchpad button doesn't seem to do anything. Which means I've got to use the menu button instead, which is another point against vive mouse because it uses that input to disable the cursor. Not a huge number of games need more than 2 buttons, but it's a point of consideration.

[greymatr]

In my mind the way to do it would be to have something like OpenVR Input Emulator fix the headset position in space to a preset location (with maybe fine adjustment to dial in calibration), click the recenter button on the program to set the screen angle and spacing, and then cancel the headset position so it goes back to the actual one.

This is a good idea. I did have a look at OpenVR Input Emulator and I had heard about it before with regards to changing the angle from the controller to the screen. It's open source which is good, but I read on there that people said it was out of date and that it wouldn't work. Did you have to downgrade SteamVR/OpenVR or anything like that to be able to use it?

I'd have to look into how it actually changes the pose etc and if it could do it for the whole position with virtual desktop picking that up

[Toasty833]

This is a good idea. I did have a look at OpenVR Input Emulator and I had heard about it before with regards to changing the angle from the controller to the screen. It's open source which is good, but I read on there that people said it was out of date and that it wouldn't work. Did you have to downgrade SteamVR/OpenVR or anything like that to be able to use it?

I'd have to look into how it actually changes the pose etc and if it could do it for the whole position with virtual desktop picking that up

When I went looking earlier I found https://github.com/Louka3000/OpenVR-InputEmulator-Fixed, which works for the pose based stuff, I use it to realign my tracker to the gun barrel on the latest SteamVR. I couldn't get the input remapping side of things to work though, I think that was broken at some point. OVRIE seems to be offset based in that it takes the current position of the device and just moves and rotates it, I don't know if it can be used to send hard coded values, although you could maybe have a virtual tracked point as a spoofed controller/tracker (via the client_commandline protocols), and then switch the tracking between the headset and that fake tracker to position it precisely for VD recentering. One other thing of note is that either redirect or swapping 2 devices in the interface causes it to crash, I think swapping is the one that works while redirect causes crashes.

[greymatr]

I found this youtube before that gives a link in the description to a dll file linked on the github issues page:



It enabled me to use the offset menu and I think it may also fix the remapping but I didn't test that. I can see now how the offsets work as it is.

Unfortunately they didn't seem to provide any code and it's already compiled so it still leaves me to get it working.

But that github you sent could be very useful because I'll need to compile the program and use it to figure out exactly how the offsets work and if I can send a fixed position.

I had tried something with the client commandline program but it didn't work. But I didn't know about it faking a tracker and maybe that's why

[Toasty833]

I had tried something with the client commandline program but it didn't work. But I didn't know about it faking a tracker and maybe that's why

I tried messing around a bit with modifying the create_virtual_controller example file to generate a fake controller and move it roughly to the middle of my play field, it partially worked after swapping it with my headset but messing with both the offsets in OVRIE and directly writing them with the commandline didn't really generate the desired effect, the controller visually moves around the playspace and the calibration seems static to the fake controller, but adjustments don't make any changes. I think that what might actually be happening during the swap is that the headset gets confused and becomes stationary at its last position rather than swapping the tracking to the fake controller, so maybe it's not usable. Swapping a real tracker with the headset seems to work, maybe it's a limitation of the virtual device, and I'm unsure if the client can modify the position of real devices.

[Toasty833]

So I've been doing a bit of testing with Virtual Desktop + modified OpenVR2Key and so far things are working pretty well. I've got an AHK script to make clicks on the edge of the screen convert to right click, which works with all the games I've tried so far, but it still has a few kinks to iron out.

M2 has 16:9/4:3 cursor mismatch, but calibration seems to fix it. Mame has the same issue (after setting mouse/lightgun handling to win32, none of the other options worked), but I haven't worked on it yet. Teknoparrot picks up the clicks as "unknown device", and works on the couple of games I've tested so far.

Demul (with nomousy), Model 3, and PS3 emulation all seem to work with no issues so far.

[greymatr]

I've had a look at the OVIE source code and I got the original source code to compile and I think the important part of the fixed version.

I looked through the code and I think I know what I need to do to be able to lock the pose and position of the headset but I need to get the overlay program to be able to read the current data and then send it through to the driver dll to be able to make it stay at that saved position.

I will also have to work out how I can do all the user interface side of that in the overlay.

I think it is all possible but it'll take me some time to work through.

[greymatr]

I've got something to work now. I had been trying to get it to work with the OVIE overlay/desktop program but I couldn't work out how to change the UI. They use Qt which I'm unfamiliar with and it sent me in circles.

So I ended up modifying the client_commandline program and the driver instead.

You will need to copy the driver_00vrinputemulator.dll into the steam folder:

C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers\00vrinputemulator\bin\win64

I would rename the one that will be in there.

Then with the program I have made two batch files, save.bat and replay.bat

The commands they run are:

client_commandline savepose 0 vrdeskHMD.txt

and

client_commandline replaypose 0 vrdeskHMD.txt

0 is always the device ID of the HMD and the filename can be changed.

Use save.bat to get the current position and rotation of the HMD and put it in the txt file. Then replay.bat will use that and start the driver replaying the saved pos/rot. It will do so until a key is pressed

OVIE can be used to enable offsets after the replay.bat is replaying. But only the last set of values, i.e. "Driver Offsets" should be changed. This will allow for some adjustments. Keep in mind they won't do anything to Virtual Desktop unless you hit the orientation button (or F4 key).

You can save the position again with the offsets being enabled by running save.bat but make sure replay.bat is still running when you do. After that you can disable the offsets, stop replay.bat and then run it again when you want

[Toasty833]

Great stuff. As of right now, the program kind of works for me. When I save coords, they replay as upside down and in the floor. But, I can mess with the values to potentially align the false values with the true world values through trial and error, and then it might work for calibration. Are you using a vive still? Maybe the offsets are different for Index and other lighthouse HMDs.

Edit: Well, I was getting close, and then all the pose information just randomly changed on me, it was consistent for about half an hour. Where I was seeing the screen straight ahead became about 90 degrees to the right. Maybe it's some kind of issue with my setup. I'll mess with it more tomorrow.

Edit 2: Tried restarting SteamVR, and it seems like the pose data is back to the initial set so I can keep working on my calibration. It might've changed slightly, I'll see if it stays consistent through resets and over play sessions. As long as it's the same on boot every time it should work fine though.

[greymatr]

Are you using a vive still? Maybe the offsets are different for Index and other lighthouse HMDs.

Actually I remembered that my vive had a problem, the hdmi part of the 3 in 1 long cable isn't working. I can get the headset to work if I snake a hdmi cable in there so I know it's just the cable. But I remembered that I had a Pimax HMD and that also works on lighthouse so I've been using it.

I don't really think the headset should matter that much as the values should still come out the same but I could be wrong. But the room setup may affect it, are you using a room setup or like a sitting mode or something else?

There is a variable I'm using that is for standing and the other options are for sitting or raw and uncalibrated. I think I thought standing would work for a normal room setup so chose that.

When I save coords, they replay as upside down and in the floor.

If you save the coords, then replay them and then save them again while replaying. And then replay that save, is it still printing out the same coords on is it changing around?

Probably best to open two cmd prompts and run the commands so you can see all the outputs.

Also you can use OVIE offsets to try to do the correction, I found you can change the numbers by typing them in so that saves pressing the + / - buttons

[Toasty833]

There is a variable I'm using that is for standing and the other options are for sitting or raw and uncalibrated. I think I thought standing would work for a normal room setup so chose that.

I think I'm using standing too. OVIE works fine for remapping my tracker position, so who knows.

If you save the coords, then replay them and then save them again while replaying. And then replay that save, is it still printing out the same coords on is it changing around?

Yeah, they change. If I manually adjust the coords they change too, by saving with my headset inverted I got kind of close and have been manually dialing it in by adjusting and comparing.

Also you can use OVIE offsets to try to do the correction, I found you can change the numbers by typing them in so that saves pressing the + / - buttons

I worry that this will have the same offset issue so I haven't tried it yet, maybe I will for fine tuning when I get a good alignment. It's also hard to get a good view of when my headset is so far away from the actual playspace, can't get the controllers to hit the buttons a mile in the air.

Edit: Tried it out by adjusting yaw, but it sent the new saved coords off into the stratosphere again. Manually adjusting the vrdeskHMD pose is what works for me, so I'm going to stick with it. Didn't know about the desktop mode though, I might try that out in the future.

[greymatr]

It's also hard to get a good view of when my headset is so far away from the actual playspace, can't get the controllers to hit the buttons a mile in the air.

Do you know about the desktop mode of OVIE?

If you go into:

C:\Program Files\OpenVR-InputEmulator

there is a batch file called startdesktopmode.bat

That will bring OVIE up on the desktop. That is how I've been using it

Edit 2: Tried restarting SteamVR, and it seems like the pose data is back to the initial set so I can keep working on my calibration. It might've changed slightly, I'll see if it stays consistent through resets and over play sessions. As long as it's the same on boot every time it should work fine though.

I saw this edit too, so I hope that it will work. Let me know how you go

[Toasty833]

So I did some more testing today. First run, all the offsets were off again, second they were as before. It seems that when my headset loses tracking it can switch to a different offset pattern. The first offset is roughly upside down and 3-4 feet higher. The second one is a 2-3 feet higher turned 90 degrees right, with the headset also being turned about 90 degrees right. These both affect the replays, but cycling the HMD tends to put it back on the first offset.

I tried updating to the latest steamVR beta, no change. I tried roomscale rather than standing room only, no change. I tried saving poses from my tracker instead by trying to save a different device number, but this always returned the same value. I tried uninstalling other VR overlays and such, didn't help.

I went back to manual adjustment, this time with the help of OVIE on the desktop. I can't save the offset over the replay without it just compounding the existing offset bug, but I can just save it as a profile in OVIE and then load up the replay and apply the profile to fine tune it. Which also lets me adjust on the fly using VR view, which is useful. As of right now I've got it in the general ballpark, just needs a lot of fine tuning, which OVIE isn't great at.

[greymatr]

I will try to get my Vive headset out later today and see if I can get it working and then see if I can replicate the issues.

The reason it doesn't work with a different device number is because it uses a OpenVR call that is specific to the HMD but this could also be part of the problem if the Vive works differently to the Pimax. I'll have to test that and may have to rewrite it.

Either way though, the replay function is only working to replay on the HMD at the moment. That could possibly be changed but seemed like we only needed to replay the HMD.

[greymatr]

I've got a new version. This one should work now.

I got my vive headset out and I couldn't get anything to display in the lenses. But I was able to see that the headset was tracking by right clicking on the HMD icon in Steam VR and selecting Display VR View. So using that view I could see it was causing the same issue when I run the program I sent. It flipped around and jumped up or down etc.

So I tried a few things but didn't get anywhere so I re-wrote how I did the pose and now get it from the OVIE driver instead of through Open VR. It gets the pose after offsets are applied, so the way that I described with the saving while replaying works. Just remember to unapply the offsets when replaying after saving or it'll double the offsets.

The values that are replayed come straight back out again when saving so it's rock solid now. I also got it to work with any device ID and I tested a controller with Virtual Desktop and the cursor replays in the exact spot. It should work if you wanted to save controller pose and then replay to HMD as well.

I tested the OVIE offsets after getting the replay to work but the desktop mode was failing to run. I figured out I must've changed the driver dll too much for it to work so I had to compile the overlay exe again and that worked. But I had to make a change to the overlay code to get it to compile for me and I don't know if that will have broken anything else in the program. It had to do with keyboard codes so maybe only the remapping section which I'm not sure was working anyway.

Hopefully this will all work now and be much easier to dial in the calibration

Edit 1: I should've mentioned too that you can use all the offsets in OVIE now. Interestingly though, when I couldn't get the desktop mode of OVIE to work, I went back to the pimax and noticed the pose data it gets is always only the driver values which probably also has something to do with the problem vive was having

[Toasty833]

New version seems like it works perfectly on Index, thanks a lot. Got it pretty close to dead on with a couple of tries, fine tuning should be easy enough with OVIE.

Before I forget I thought I should attach my right click border script (courtesy of a guy over the AHK forum) for anyone interested, which has worked so far in every emulator and game I've tested it in. It just takes the 5 pixel outer border of your monitor and converts left clicks in that area to right clicks, which works well with VD. You might have to adjust the timer delay from 20 to something higher on slower systems, and the program is set up for 4k so you might need to redefine the posX and Y to match something smaller.

Right now I think I'm pretty happy with my setup. I've got a few more emulators to try out, but I can play the vast majority of stuff I've tested. CXBX seems to have some calibration issues I've got to work out because most people are just using DemulShooter instead but I can't, and I can only play MAME stretched fullscreen right now due to it not properly offsetting for 4:3 and expecting that to be handled by the gun.

[greymatr]

New version seems like it works perfectly on Index, thanks a lot.

I'm glad to hear that it is working now.

Before I forget I thought I should attach my right click border script (courtesy of a guy over the AHK forum) for anyone interested

Thanks for that AHK script, could be very useful in future. I might have to get out my tracker gun and give it another go.

I can only play MAME stretched fullscreen right now due to it not properly offsetting for 4:3 and expecting that to be handled by the gun.

People seem to have frequent issues with MAME and aspect ratio. There has been threads on this board about it. I don't know if anyone has found an easy solution though but I might give it a go too.

CXBX seems to have some calibration issues I've got to work out because most people are just using DemulShooter instead but I can't

Do you know what the problem with using DemulShooter is? I'm guessing it's some kind of mouse input type related problem?

[Toasty833]

People seem to have frequent issues with MAME and aspect ratio. There has been threads on this board about it. I don't know if anyone has found an easy solution though but I might give it a go too.

Yeah, I thought it'd be an easier fix but I'm still at a bit of a loss. From my research it seems like MAME just doesn't like aspect ratio mismatches, so if you try and play 4:3 games on a 16:9 screen, only vertical lines up with horizontal losing alignment the further away from center you go. It seems most other guns offer a 4:3 offset to fix this, like Gun4IR. I haven't yet tested alternative versions of MAME though, they may have different input handling that gets around this issue, retroarch may fix it and I might wind up setting it up anyway for PS1 games and other console titles.

Do you know what the problem with using DemulShooter is? I'm guessing it's some kind of mouse input type related problem?

I believe the issue is that whatever output VD is sending out is just a simple windows low level input hack, while DS is expecting an actual hardware device. You can't select the input from VD inside DS. I can partially get it working by using it with a hardware mouse and the usesinglemouse console command, the gun doesn't do anything but if I move the mouse it updates to the absolute position of the gun, and if I click the hardware mouse shots are fired at the position of the gun. I imagine you could fashion a clunky workaround with a fake hardware ID, but from my testing so far the only thing that DS seems useful for on a 1 gun setup like this is for CXBX, which is still fairly buggy and only useful for 2 games. Everything else has worked without it, and there's a good chance that CXBX will eventually get some better native mouse input support, or I can fix the issues I'm having with it.

[ThatOneSeong]

Since ^^^ up there was asking in private earlier, thought I'd just paste the same reply I gave them for everyone else reading:

MAME's mouse position handling is very dumb, and just uses the entire output screen to map to the game screen - meaning no matter how you stretch or squash the image, the edge of the TV screen correlates to the edge of the game screen.

Libretro, on the other hand, treats the game screen 1:1 with the display - so if you click in the game play area, it'll click in that position, and clicking on the black border area will be a screen edge/off screen shot. This will actually respect the game screen output; whether that be integer scaling or aspect ratio difference, so the gun can continue to operate in its normal mode.
It's not just Libretro that does this, FWIW: PCSX2-nightly does the same thing, both for mouse inputs and a gamepad stick acting as an absolute position surrogate mouse. Others like Model 2 emu, or Flycast (both standalone & libretro core) respect this too, though the former has no real off-screen detection and uses a hack to get around it lol. Most others that use a fullscreen window in native desktop resolution should respect this. That is why I tend to say that MAME is doing it wrong.

Of course, that doesn't mean the 4:3 restrictor in gun firmware is useless - at least on Windows, it is necessary due to how fullscreen apps often take over the screen resolution, and idiosyncrasies with absolute mouse positioning if the game doesn't use the same aspect ratio and... it's a lot. That's why 4:3 mode would be necessary. On Linux w/ Proton, or a version of Wine with the fshack, none of this is necessary since everything scales up to desktop resolution so the accuracy of the mouse cursor doesn't change, much like Libretro et al. 

That said, this is mostly talking wrt lightgun systems like GUN4ALL and GUN4IR. No idea how that applies to VR-tracking-on-pancake screens. This is mostly why I lean towards using RetroArch whenever possible.

[Toasty833]

Thanks for the response. I gave Retroarch a try, and sure enough, the gun mapping is fine for the standard MAME core, and probably the offshoots too for games that run better through them. The other console cores (NES, PS1) I've tried have all been fairly easy to setup through RA, so I'll probably stick to those going forwards.

I think all I've got left is Dolphin, which I haven't tried setting up yet, and a few native PC versions of games. Aside from CXBX everything else looks like it's working fine.

[greymatr]

I've had a look at CXBX and was able to replicate everything Toasty was saying. So after a lot of testing and coding things, I found I was able to make a small one line change to the DemulShooter code and it allows the Vive controller via Virtual Desktop to work.

I've attached the compiled exe's (standard and x64) and a txt file with the code change.

When running the exe it needs to be done from the command line like this:

demulshooter -target=chihiro -rom=vcop3 -usesinglemouse

i.e. must have -usesinglemouse. I only tested vcop3 but it should work for all games though and other platforms.

I tested it on a newly unzipped DemulShooter (latest version) and I found out that you must run the DemulShooter_GUI.exe first and choose a device for P1. I also don't know for sure that added P2,P3,P4 etc will have any effect.

It seems you need to choose a hardware mouse for P1 or the mouse clicks don't work, the crosshair still moves around though. I think this should also work for other programs that fake the mouse movement and clicks. For instance I think this may work for certain Wiimote guns programs etc.

By the way, thanks for that reply ThatOneSeong. It was enlightening. I think choices like this get made earlier on in making a program and then get harder and harder to fix later on. So it just becomes the way it is.

Edit 1: I should mention specifically that the DemulShooter version that it needs to overwrite the exe's is version v12.2 https://github.com/argonlefou/DemulShooter/releases/tag/v12.2

[Toasty833]

Hey, great stuff. Works perfectly with my gun. Well, there was one issue, and it's that CXBX seems to have the same 16:9 offset thing going on. I found mention of a command line option that would fix this for DS, but it didn't work anymore. Checking the code, it turns out that -ForceXratio has been changed to -ForceScalingX at some point in the last year, so running DS with -ForceScalingX=4/3, everything lines up.

Game works good, although I think my AHK script conflicts slightly with DS, I had some occasional right click failures and using my off-screen reload caused a left click to be sent first. I also started using launchbox and found that it didn't support AHK 2.0, so I changed the off-screen script to match v1 instead so launchbox can auto run and close AHK on emulator exit. And finally, I tried out Dolphin, which works fine.

[greymatr]

Edit: This may be irrelevant now, see next post

Does the AHK script always send a left click first with DS and does it still do it after converting to v1? I'm not sure about the right click failures. It may be possible to write the same functionality into DS but I'll have to investigate how the clicks work. As I only figured out the position and the line I changed made both work

[greymatr]

I've got a new DemulShooter mod to test. This adds offscreen reload along the borders like the AHK does.

So to use it, again paste the exe files into DemulShooter v12.2 ( https://github.com/argonlefou/DemulShooter/releases/tag/v12.2 ) use DemulShooter_GUI.exe etc.
and run the command line like e.g.:

demulshooter -target=chihiro -rom=vcop3 -usesinglemouse -offx=20 -offy=30

This means that the first left 20 pixels and the last right 20 pixels will change the left click (shoot) to right click (reload).
And the top 30 pixels and bottom 30 pixels will do the same. A value a 0 will have no reload effect or leaving either -offx or -offy from the commandline.

I've put the changes again in a txt file in a directory should it need to be changed with a new version of DemulShooter.

I don't think the -ForceScalingX=4/3 option would affect anything for the reload. I tried to use it but seemed like it was more inaccurate for me when I did. But then I don't have a calibrated Virtual Desktop going. I'm using it more like a mouse atm.

I also didn't realise I could run the game in 4:3 res, I was using 16:9. So I found that there is the two options, change the video mode res or turn on/off Maintain Aspect Ratio but any combo I tried didn't seem to need the ForceScalingX option.

If it's a problem then let me know. Hopefully this means you won't need to run the AHK script with DS