it could be a few things.
1. are there any spaces in the path to your front end? (ex: c:\program files\mala\mala.exe) if so, when you specify the path in that registry key, put it in quotes..ex: "c:\program files\mala\mala.exe"
usually if you are getting a My Documents window popping up, there is something calling explorer.exe to launch, which is wierd. i had a similar problem on my cabinet where i could not get the front end to load as shell no matter what i did. as it turned out, it was due to a software conflict with a dell wireless usb adapter driver. i had a dell usb wireless adapter installed for network connectivity, and when i installed the driver, it installed a wireless utility. i had to remove the wireless utility and just manually install the driver for the adapter..as soon as i removed that utility, everything worked well. so it could be a conflict with something you have installed, but figuring that out could be very difficult