I struggled to get videos working in Mala and Visual Pinball vids. They would play but come up all distorted.
After a tip from a friend, he suggested that I use the free open source screen recorder
http://camstudio.org. Recording the videos couldn't be easier and Mala has no problem playing them.
One thing to keep in mind is that you need to record a fixed area in the standard video resolutions (640x480, 800x600, etc.). Give the avi the same name as your vpt table, tell mala where the video directory is and you're done!
Been a while since I posted here but the great Mala front end has my re-energized me in getting the software on my cabinet up to snuff.
Hope this helps any of you wanting to create custom vids for your Tables ( and I am sure this works with MAME as well but I have not tested that).