While the game is playing, press te TAB button. Scroll down to "video" and then "mode". From here choose a different video mode. Alternatively, under the video menu use the scale methods to scale the current video selection to your screen.
Advmame is designed for drawing games at exact video modes to monitors. If you desire to scale your games up to high screen resolutions, you're much better off using either the standard Win32 MAME with DirectDraw or Direct3D blitters in windows or xmame with SDL or OpenGL blitters in linux. Advmame was not really designed for game scaling, and as such won't look good on high-res monitors.