Well, the best system out there for custom showers is the schluter kit. It comes with waterproof membrane, proper drain fittings, and base and threshold forms (a fancy word for shaped styrofoam). All the custom showers I did started with this kit. It was pricey, but when you factor in the time spent forming the base (and proper drain slope) and the membrane...it was worth it.
Greenboard is also crap for a shower, don't use it. I have used both concrete board, and a newer product that was a water and mildew resistant tilebacker board, which was much denser than drywall, but easier to cut than concrete board. Either is a good choice though. My method was to do the rough plumbing, then build the basic frame and sheathe that in your board. Caulk the seams, then apply a modified thinset, and cover all of that with overlapping layers of the schluter membrane (the kit even has inside and outside corner pieces. The tricky part was squeezing excess thinset from edges I was going to have to tile. When this dried, I would use an ultralite thinset scratchcoat, and then use the ultralite to mount my tiles. Once all that set for a day or two, I would grout, seal, then caulk all the meeting edges with silicone (wherever two surfaces came together like wall to floor, wall to wall, seat to wall, etc.
Then when everything is done, call in a glass specialist to measure, manufacture and install custom glass walls and doors, this was something I always left to a pro...it's REALLY tricky work.
I had a preference for travertine or any kind of natural stone, because edges can be made with a diamond file, and with tile, you are much more limited....
If everything goes right, you should have a leak free shower, but really try and make sure that after all that work, the plumbing for the shower can be accessed by a back wall, so that if there is trouble, you don't have to smash through all that. As well, check your structure, if you are adding things like natural stone and thinset to a spot that used to just have a steel tub and plastic surround, you might find that doors downstairs stop closing correctly if there isn't enough support.
There's a lot to it, and sometimes, honestly, it's just better to hire a pro... but I always admire someone who will at least give it a shot (even if you have to bite the bullet and hire a pro later!).
