You need to create a batch file that will be the executable. Mala will pass the rom name to the batch file. If the batch file has an ISO extension (or whatever extenstion your images are), then it will execute a command such as:
"C:\Program Files\DAEMON Tools\daemon.exe" -mount 0, %1
%1 is a variable that will be equal to the rom name and rom path that Mala passes to the batch file
The batch file needs to have an IF statement that will bypass the daemon loading of the image if the rom has a PCE extension.
After this of course the batch file will need to execute the emulator with something like this if its a PCE rom:
C:
cd \emulators\MagicEngine
pce.exe %1
The command will be slightly different if the rom is an ISO. Also I think you also have to edit the PCE.ini file to point to the virtual drive that D-tools created.
I've done all this before with PC type games that require ISOs, but its been a while. If you can't get it, I should be able to put something together in the next day or so.