I'm using NullDC (the other didn't work very well).
What you need is a program called CloseMul which lets you send keypresses and hijack the ESCape key and to wrap it around a batch file:
You need to get yourself a copy of CloseMul (
http://vertrixgame.com/downloads) and put it somewhere nice. Then in the nulldc folder create a file called Closemul.ini and in there have the following:
[Parameters]
Exit Key 1=27
Extracted Ext 1=cdi
Automation="nullDC v1.0.4 (built Jun 20 2010@01:32:06"
Automation Steps="WAIT 700,ALT+F,ENTER,WAIT 700,TAB,WAIT 700,SPACE,WAIT 700,FILE,WAIT 700,ENTER"
This file sets ESCape (ascii 27) to be the exit key. It also sets the automation to wait, go to the file menu, wait a bit , tab, wait a bit, press space, wait a bit, type the file name sent in, wait a bit then press enter

Then create a file called start.cmd and in there put the following:
cd C:\arcade\Emulators\Programs\nullDC
cmd /c c:\arcade\tools\closemul\closemul.exe nullDC_Win32_Release-NoTrace.exe file="c:\arcade\emulators\RomFiles\SegaDreamCast\%1"
This calls the closemul program, which starts up nulldc, reads the closemul.ini file and passes in the rom sent via mala (i.e. in closemul.ini FILE is represented by 'file'.
Change the paths to match your folders for the emulator, the location of closemul and the roms.
Inside MaLa, create an emulator in the 'other emu config' and for the settings:
BASICS
executable: C:\arcade\Emulators\Programs\nullDC\start.cmd
i.e. change this to the path where your nulldc is, which is also where the start.cmd is that you created a few steps before.
In the rom path enter the path to the roms, e.g. C:\arcade\Emulators\RomFiles\SegaDreamcast
In the rom extensions enter 'cdi'
EXECUTION
%rom%.%ext%
What this does is start the emulator (nulldc) with a batch file (start.cmd) which will run the emulator via closemul.
What closemul does is allow you to hijack and inject keypresses, in the case above when you press escape it will terminate nulldc and when you start it, it will run file open, etc.
Note this will only work with filenames and paths with no spaces, so get rid of them.
Note, you will also need to set up nullDC how you prefer (you can do it via closemul such as pressing fullscreen button, etc but I prefer to do all the config in the emulator).
In nullDC you need to create some memory files and configure the joystick/keyboards. If you need further help I'll post a document I'm working on.
If things don't work the best way to do things is to not use mala and run things from the command line to see where they are going wrong.