CF cards are far more common for this use than SD since a simple passive adapter (a bunch of wires and connectors) makes for really cheap conversions. They show up to the PC as a normal IDE hard drive, and you can use it as such. Trying to use USB has some issues, especially with windows, since USB mass storage is emulated as a hard drive by the BIOS, but once the OS loads has to be natively accessed, and most OSes don't support having their main filesystem on USB (Linux can be done with some work, dunno about the BSDs - I've been told this is impossible with Windows). You could pull DOS off, but beware that it will be ungodly slow on many (though not all) motherboards.
As far as the reliability of flash, it's a helluva lot more reliable than a mechanical hard drive. Most flash devices are good to at least 10,000 writes per sector. Many are good to 100,000 or even 1 million. All good CF cards employ wear leveling to prevent a single sector from being used repeatedly, and they also support sector remapping to map around bad sectors. Unless you're using it as a swap device (don't, they're far too slow when writing), you will pretty much never wear out a large CF card.
Somebody calculated once that if you were to continuously erase and re-write (the erase is what's limited, not the writes themselves) the flash in the original 16MB iPaq handheld as fast as it would go, it would take 12 years to wear it out. That's a long time, and not much flash. Modern flash is faster, but there's so much more of it that you won't have problems. The erase limitations of flash are VASTLY overstated. Use a high quality card (like the pro series from Sandisk), and you'll be fine. Just be aware that writing is rather slow, though reading is fairly quick, at least in comparison.
You can fit a lot onto a 4GB CF card, and these are now reasonably priced. 8 and even 16GB are on the market, though at a cost premium. Just be aware of how big your OS is: Linux can be tiny (or huge), Win98 is reasonably sized, and XP is usually at least a gig. Vista is...well, huge: it might fit on the card... My friend says his install is 7GB, though there's some OEM crap in there.