My gripe about most other front-ends is that they look too much like meda players.
...and now you want to turn Mala into a media player?!?!

Autohotkey all the way. A lot will depend on if the bluray autoplays when inserted. I'm guessing it will.
If it does, just have ahk kill VLC (process, close, vlc.exe) when the exit button is pressed.
Throw in a
mouseclick, left afterward to return focus to Mala.
There is probably also a way for ahk to send a specific command line when a disc is inserted.
A quick search turns up this interesting script:
http://www.autohotkey.com/board/topic/44433-a-simple-script-to-add-psuedo-hd-dvd-autoplay-in-win7/IfExist, %Drv%:\HVDVD_TS
Sounds like it detects if a disc is present and will do what you want if it is.

.....of course you don't want it to relaunch after the movie has exited and the disc is still present.
The easy fix would be to have the ahk script also eject the disc when the move exits, but there is probably a better way.
I'm no expert. I just cobble together sloppy scripts from examples I find online.
