I've made some progress here. There seems to be some kind of internal data corruption on the shadow_mask_texture variable. I've been looking through the source but I haven't located the problem yet.
My original build was MAME143, 64bit on Win7, so I first grabbed the latest diff (143u1) and recompiled. No help there (I expected as much, there was no mention in the changelog). Then I figured it might be some kind of 64bit/32bit issue in the mingw libraries, so I recompiled as 32bit without the PTR64 flag. Again no help. Next I thought it might be some kind of overflow in the data structure, so I reordered the elements (and their initializers) to see if I could alter the behavior. No change.
I noticed in my testing that the scanlines would show up, but the shadow mask was not there. So it was apparently corrupted before it even got used. I made the following edit in the source (in blue) to hardcode the path, then recompiled:
d3dhlsl.c - lines 892-896
// experimental: load a PNG to use for vector rendering; it is treated
// as a brightness map
emu_file file(window->machine().options().art_path(), OPEN_FLAG_READ);
//shadow_bitmap = render_load_png(file, NULL, options->shadow_mask_texture, NULL, NULL);
shadow_bitmap = render_load_png(file, NULL, "aperture.png", NULL, NULL);
Bingo - shadow mask reappeared. It still corrupts the shadow_mask_texture variable in the hlsl.ini file every time it exits, but now it doesn't matter since it is hardcoded - this fixed the "black screen hanging on startup" problem I was having.
As far as the overall darkness, I had to crank the gamma way up as well as tweaking contrast. The two shots below illustrate. The first one is my reference - HLSL off, brightness/contrast/gamma all set to 1.0. Second one has the following settings:
brightness 1.0
contrast 1.15
gamma 1.8
shadow_mask_alpha 0.10
shadow_mask_texture aperture.png
shadow_mask_x_count 320
shadow_mask_y_count 240
shadow_mask_usize 0.093750
shadow_mask_vsize 0.093750
curvature 0.05
pincushion 0.05
scanline_alpha 0.25
scanline_size 1.0
scanline_height 0.5
scanline_bright_scale 1.30
scanline_bright_offset 0.6
scanline_jitter 0.0
defocus 1.5,2.0
I'm still tweaking these. If I get some time I may try to track down where that corruption is happening.