Yeah seriously man, you are running mame plus another app and you are in xp... that's a lot of your issue right there. With that being said, you shouldn't get memory errors on a modern os if your page files are setup correctly. I'd suggest looking at your xp setup before anything.
Also some things of note....
Don't use mame32, j5 is designed for mame, mame32 support is not guaranteed. Yes there are differences, most noteably that mame32 doesn't always support all the command line flags of mame, which are used extensively by j5.
If you have a badly corrupted ctrlr or cfg file, j5 could be getting sent into some sort of infinate loop trying to read it, which could eat up all your memory. This is highly unlikely though.