OpenGL? Which version of mame are you using? SDLMAME? (The official mame uses directX.)
As for the number of scanlines, sounds like the png you're using is 4 pixels tall. I'm not sure if you'd like the results if mame scaled the screen effects; from 4 to 5.555555 at 1600x1200(p) and from 4 to 4.444444 at 1280x1024(p) for pacman. This means a very fuzzy scanline, IMO
nothing like what it looks like now.
One way to get your 288 scanlines
today is make your own png file with 288 scanlines and use it as a overlay. Mame does scale overlays, so you'll get to see how mame auto-scaling the scanline effects would look. (The problem, of course, is you'd need a different overlay for each of the different resolutions.) The other way is to run mame at a resolution that's an integer multiple of 288 (I know, not possible for most cards, but possible for other games with different resolutions).
Please note I don't use the effects feature, so I could be way off.

IMO we'll have to wait until monitors get in the 250 to 300 dpi range before worrying about scaling the scanlines (current monitors run around 85 - 100 dpi, while laptops usually run higher about 100-150 dpi). Far too ugly until then. (I've wanted 300 dpi monitors even since I tried using "WYSIWYG" editors, and not seeing on the monitor what I was getting off the college 300 dpi laser printer, back in the early 1990's; I think I still have a looong time to wait though.

)