Anyone here use NAS (Network Accessible Storage) devices?

[drventure]

kahlid74 makes some good points, but for a home environment, I'm not sure it's worth the extra cost for full on raid certified drives. But YMMV.

One thing that IS important to point out is with raid, you have basically 2 choices (3 if you're willing to spend a good bit more $$$)

1) Take two drives and mirror them. You get faster read access, the same write access, and if one drive fails, you still have the other drive till you replace the failure.

or

2) take two drives and stripe them, you get faster read and write access, BUT if either drive fails, you're toast. So +absolutely+ make sure you have good backups. The reason this option is considerably more dangerous is that you now have 2 drives, but a single "system", so if either drive fails, the system fails. Since both drives have the same MTBF, this effectively means you're +decreasing+ the MTBF to half what it would be with only a single drive. In other words, the system is far more likely to fail earlier than using just a single drive.


The other option is striped with parity, but that usually requires 5 drives and is a bit more difficult to setup.

I've used a mirrored setup for years, with reasonably regular backups. The mirrors have saved my bacon more than once.

[kahlid74]

kahlid74 makes some good points, but for a home environment, I'm not sure it's worth the extra cost for full on raid certified drives. But YMMV.

One thing that IS important to point out is with raid, you have basically 2 choices (3 if you're willing to spend a good bit more $$$)

1) Take two drives and mirror them. You get faster read access, the same write access, and if one drive fails, you still have the other drive till you replace the failure.

or

2) take two drives and stripe them, you get faster read and write access, BUT if either drive fails, you're toast. So +absolutely+ make sure you have good backups. The reason this option is considerably more dangerous is that you now have 2 drives, but a single "system", so if either drive fails, the system fails. Since both drives have the same MTBF, this effectively means you're +decreasing+ the MTBF to half what it would be with only a single drive. In other words, the system is far more likely to fail earlier than using just a single drive.


The other option is striped with parity, but that usually requires 5 drives and is a bit more difficult to setup.

I've used a mirrored setup for years, with reasonably regular backups. The mirrors have saved my bacon more than once.

My apologies, my intention was never to say you have to buy raid certified drive, only to warn you that in most cases, those regular drives you buy will die faster in your NAS then in your PC.  So make sure your data is backed up for the worst case scenario.

[drventure]

My apologies, my intention was never to say you have to buy raid certified drive, only to warn you that in most cases, those regular drives you buy will die faster in your NAS then in your PC.  So make sure your data is backed up for the worst case scenario.

No worries. That's generally true. You +can+ setup nas drives to sleep as well (in my case, I'm running win7 on my "server" and just sharing the drives, so I can sleep them as you would a desktop PC). I've got some backup processes and other things that run on that machine, though, so I'm not sure how much they actually sleep.

NowI'm curious, I'll have to check that out...

[shponglefan]

I'm curious to know how much NAS drives really do run active vs desktop PC.  In my case, I mainly use my NAS for storage, so I'm not accessing it all that much (certainly not as much as my regular PC), so I can't imagine they would see more 'mileage' on the drives.  And I've got hibernation active again on the NAS, so...

I agree though that drives like WD Green which are designed to spin down to save power would be wholly non-ideal in a NAS, since they would be constantly spinning up/down.  I use a WD Green drive as a backup on my main PC, and notice that behavior a lot.  But now that I've offloaded my files to the NAS, I don't really need the WD as much...

[MonMotha]

I've not noticed particularly higher instances of failure, but my storage environments are more like normal PCs that just happen to run 24/7, so if you have one of those little SAN enclosures or something, things may get a bit hot or whatever.

What IS somewhat problematic is the handling of error reporting by the disk.  Mostly for market segmentation, but I'm sure it makes them a hair cheaper, too, the "consumer" drives don't report errors very well.  In many cases, they'll just return invalid data as though it were correct.  This can be a real bummer when operating RAID arrays.  If you have an optimistic read strategy on a RAID 1 (where you only read from one device rather than always read from them all and check the integrity), you could serve up bogus data.  During rebuilds, you'll silently rebuild the array with bogus data.  The "NAS grade" drives actually check for and return a read error when one occurs.  It still won't save your data, but at least you'll know that you need to reach for the backups rather than having to endure silent data corruption that many go unnoticed for weeks, months, etc.

Moral of the story: RAID is not a backup in any case, and especially not when using cheap drives.

[lilshawn]

Wd red drives which are made for nas use are only about $20 more than equivalent blue drives.

[MonMotha]

Yes, the WD Red drives are one of your best value options in this particular case.  They seem to work well, have the features you want in this application, and are of reasonable cost.

[lilshawn]

freenas has some power options built in to power down drives that aren't being used after a period of inactivity. every minute an HD spends not turning, is a minute longer it will last at the end of it's life.

if you can stand the couple of second delay of spinning up the disks, that's an option too.

[lilshawn]

well, i've ran into my first hitch.

the NAS dropped off the network last night. Was reading videos off it from 2 XBMC xboxes at the same time. they both came up and said the host was unreachable after about 3 hours or so. I just reloaded the videos of my desktop share to diagnose the issue no problems there... For the time being i've left it (crashed or otherwise).

i tried to ping the ip address from my desktop of it and wasn't able to reach it. I searched the network logs on my firewall to see if it had picked up a different IP address or something...didn't see anything new. did a few quick searches about the network drop issue and found a few people with multi port NICs that had issues, but one thing i did find was I think i might be a bit skinny on the RAM.

4gb seems to be the consensus for the minimum needed when you use the ZFS file system. (which i used to stripe the drives together.) having only 2gb of ram might be getting full with checksumming data, data, logging, read cache etc etc etc... so it seems this is the most plausible answer. info Ive found says ZFS typically requires a minimum of 8GB of RAM in order to provide good performance. The more RAM, the better the performance...a general rule of thumb is 1GB of RAM for every 1TB of storage.

I grabbed another 2gb to throw in there to see. (i left it as it was until i figure out the issue.) if it's just crashed then I'll reboot it and test it again with the 2gb and see if it happens again. if it crashes again i'll try with the extra RAM. if still...maybe an installation or configuration issue. Though i didn't stray too far from the standard installation. Hopefully 4gb of ram is good enough for my 1tb of storage. if not, i may have to rethink my file system used or the hardware i've chosen.

[kahlid74]

well, i've ran into my first hitch.

the NAS dropped off the network last night. Was reading videos off it from 2 XBMC xboxes at the same time. they both came up and said the host was unreachable after about 3 hours or so. I just reloaded the videos of my desktop share to diagnose the issue no problems there... For the time being i've left it (crashed or otherwise).

i tried to ping the ip address from my desktop of it and wasn't able to reach it. I searched the network logs on my firewall to see if it had picked up a different IP address or something...didn't see anything new. did a few quick searches about the network drop issue and found a few people with multi port NICs that had issues, but one thing i did find was I think i might be a bit skinny on the RAM.

4gb seems to be the consensus for the minimum needed when you use the ZFS file system. (which i used to stripe the drives together.) having only 2gb of ram might be getting full with checksumming data, data, logging, read cache etc etc etc... so it seems this is the most plausible answer. info Ive found says ZFS typically requires a minimum of 8GB of RAM in order to provide good performance. The more RAM, the better the performance...a general rule of thumb is 1GB of RAM for every 1TB of storage.

I grabbed another 2gb to throw in there to see. (i left it as it was until i figure out the issue.) if it's just crashed then I'll reboot it and test it again with the 2gb and see if it happens again. if it crashes again i'll try with the extra RAM. if still...maybe an installation or configuration issue. Though i didn't stray too far from the standard installation. Hopefully 4gb of ram is good enough for my 1tb of storage. if not, i may have to rethink my file system used or the hardware i've chosen.

The way Freenas works, you want 4 or more memory wise.  It's highly inefficient in it's processes compared to streamlined systems.  Freenas is fun for a while, but quickly you'll come to the consensus most of us have about it, which is it's not really worth the hassle and poor performance.

[lilshawn]

from what i understand ZFS is a real memory hog. if I stuck with straight UFS I wouldn't have an issue.

apparently it's something like this...

right off the bat ZFS leaves 1GB of RAM available to the OS automatically. That means the ZFS is assuming control of the remaining 1GB of my RAM right now. ZFS will use all the RAM you give it to try and improve performance...what happens if it needs more is unknown at this point, i would assume performance degradation, but i'm guessing I get what I had last night and the thing crashes.

copy paste from some other info on ZFS memory requirements... which sounds like ~2GB per TB of disk space.

Code: [Select]

1) Read cache. This function is known as ARC. Estimate this at about 1 GB per TB of zpool.
2) Temporary Write cache. Incoming writes queue up here before being added to the ZFS Intent Log (ZIL), and then ultimately to the live filesystem. This is generally only 200-500 MB, but estimate at least 1 GB.
3) Checksums: ZFS performs checksumming on all blocks of data stored within the zpool, and those checksums have to be calculated and verified for each read and write. These operations use considerable CPU and RAM. Difficult to estimate.
4) Parity-Data: If your drives are in a RAID-Z configuration, then ZFS uses RAM during the calculation of Parity Data which allows the fault-tolerance offered by RAID-Z. Calculating parity data uses considerable CPU and RAM, and must be done in addition to the standard checksum operations. Difficult to estimate.


[ChadTower]

Same idea as the discs in a DVR but for a different usage reason.  They fail faster because they get worked much harder.

[drventure]

Same idea as the discs in a DVR but for a different usage reason.  They fail faster because they get worked much harder.

Definitely. If you're picking a drive for a DVR, you generally want one that's going to run cool. That usually means a 5000rpm drive vs the faster (but hotter) 7200rpm drives.

I know WD makes some large, slower speed drives specifically for DVR uses because they run quieter and cooler. I used one when I upgraded my Tivo HD. Haven't has a single issue in several years, and it's reading or writing almost continuously.

[kahlid74]

Don't forget that real raid drives often times used different epoxies that have a longer shelf life.  This translates to better life overall as they sustain hotter/faster speeds for prolonged periods of time without negative results.

Again, get whatever disks you like, BUT SPEND THE MONEY to have a backup solution.  If you can't lose the data, B A C K IT UP!

[lilshawn]

a quick update...just for prosperity's sake, and as a note to anyone else who decides to attempt this as well...

i went ahead and popped in the extra 2gb ram (for 4gb total) and have had no problems at all.

this file system is pretty memory hungry. It's all in the name of reliability and performance, which is okay, but neither of which i really have interest in at this moment. It's basically that would be geared towards the corporate users...but it's a good learning exercise anyways. It's really only because I striped the 2-500gb drives together using ZFS to get a terabyte for testing. in actual practice, when i get a couple of 2tb drives, i think i'll go straight ufs and just do a simple dupe backup on it. that should reduce the memory footprint quite a bit that I could probably drop back to 2gb again. It's in there so i won't bother. I don't really need the performance and safety and checking of the ZFS since i'm just serving video off it. It's not imperative that data be available instantly.

but, so far so good.

 

[wesbrown18]

I admit that I'm a bit of an outlier in this regards, but here's my home office NAS:

• OpenIndiana
• AMD Opteron 4334 (6 cores, 3.1ghz)
• SuperMicro MBD-H8SCM Socket C32 motherboard
• 32GB of ECC RAM
• LSI 9201-16i HBA
• 1x 64GB Crucial M4 as system disk (root)
• 2x 20GB Intel 313's as a write log disk (ZIL)
• 1x 256GB Samsung 840 Pro as a cache disk (L2ARC)
• 12x 2TB SATA WD Greens (will replace one by one with Seagate Constellations)
• 2x 10gbps Infiniband network connections
• 2x 1gige network connections

This gives me an effective 24TB of space that I can push at 10gbps across my network.  Ironically, my cabinet doesn't use any of that -- I don't believe in having every game made to man available right on the cabinet, so it has a 64GB SSD.  I do have a collection of games stored on there.

[drventure]

Damnation. That's a hell of a home server!

[wesbrown18]

Damnation. That's a hell of a home server!

Heh.  Yeah. I do it this way so that the rest of my test and development machines don't need any local disks.  I have a pair of 12 core AMD 1u servers with Infiniband.  They are disk less and bootstrap over TFTP and then access the disks via DMA over Infiniband.  Nice thing about this is that with ZFS, I can snapshot images before making changes.  Kind of like VMware's snapshot of VMs but with real hardware.

[kahlid74]

I admit that I'm a bit of an outlier in this regards, but here's my home office NAS:

• OpenIndiana
• AMD Opteron 4334 (6 cores, 3.1ghz)
• SuperMicro MBD-H8SCM Socket C32 motherboard
• 32GB of ECC RAM
• LSI 9201-16i HBA
• 1x 64GB Crucial M4 as system disk (root)
• 2x 20GB Intel 313's as a write log disk (ZIL)
• 1x 256GB Samsung 840 Pro as a cache disk (L2ARC)
• 12x 2TB SATA WD Greens (will replace one by one with Seagate Constellations)
• 2x 10gbps Infiniband network connections
• 2x 1gige network connections

This gives me an effective 24TB of space that I can push at 10gbps across my network.  Ironically, my cabinet doesn't use any of that -- I don't believe in having every game made to man available right on the cabinet, so it has a 64GB SSD.  I do have a collection of games stored on there.

I'm glad you're replacing those green drives, that LSI card doesn't allow spin down, so you've effectively got a time bomb in that NAS/SAN.  I'm assuming stripped disks as you say you've got 24TB of space?  Kind of dangerous for that large of a chunk of data but as long as you've got it backed up/are okay with a complete loss of data no worries.

Good ole infiniband.

[wesbrown18]

I'm glad you're replacing those green drives, that LSI card doesn't allow spin down, so you've effectively got a time bomb in that NAS/SAN.  I'm assuming stripped disks as you say you've got 24TB of space?  Kind of dangerous for that large of a chunk of data but as long as you've got it backed up/are okay with a complete loss of data no worries.

Good ole infiniband.

Real capacity is in the 14TB range -- I'm not crazy enough to run it in a JBOD striped configuration:

Code: [Select]

NAME                               USED  AVAIL  REFER  MOUNTPOINT
internal                          9.38T  4.88T   258G  /internal

Actually, the WD Green drives have spin down/timeout disabled.  They were giving me performance issues with the constant spinning down -- with that many disks, it becomes a cascading spin down/spin up issues.  Once spin down was disabled for each individual drive, which was a ---smurfing--- pain in the ass, as I had to hook it up to a motherboard where the utility could see it from DOS.

You're absolutely right about it being a time bomb still.  Striped?  Not exactly.  They're configured as a pair of RAIDZ2's.  I can sustain up to two drive failures per RAIDZ2.  Regardless, I am replacing them with non-Green drives as personal funds are available.   That's the nice thing about ZFS, that I can incrementally replace the drives.  I am probably replacing them with 3TB or 4TB Seagate Constellations -- when all the drives in a RAIDZ2 are replaced, I get upgraded capacity.

Topology:

Code: [Select]

ZPOOL
   2 drive ZIL (20GB SSD)
   1 drive L2ARC (256GB SSD)
   6 drive RAIDZ2 - 4 data, 2 parity
   6 drive RAIDZ2 - 4 data, 2 parity

The funny thing is?  I used to have a motherboard with a regular AMD in there.  The lack of ECC RAM was causing ZFS parity errors on the drives, which I blamed on the WD.  Which disappeared when two DIMMs in a row died and I replaced them.  But I will never build a ZFS server on this scale ever again without ECC RAM.

That's why I replaced it with an Opteron-class motherboard and CPU with ECC RAM.

Infiniband is kind of like a high end sports car in that it needs constant tuning.  I built a 40gbps Infiniband fabric for the HPC cluster at my $DAYJOB, and there is so much lore involved.  But we push 8 gigabytes a second of data, continuous, across the entire fabric.

[lilshawn]

I built a 40gbps Infiniband fabric for the HPC cluster at my $DAYJOB, and there is so much lore involved.  But we push 8 gigabytes a second of data, continuous, across the entire fabric.

i'm too lazy/cheap to buy a gigabit switch for my network. still runnin' 100 

[MonMotha]

I honestly don't know how people put up with 100Mbps LANs, anymore.  I'm seriously considering upgrading to 10GbE. The only reason I haven't is that most of my storage at the moment is cheap and can't really go much faster than 1Gbps, anyway.

[drventure]

Gigabit's cheap. I don't know much about 10gb so I can't comment there.

[shponglefan]

Gigabit is definitely worth it over 10/100.  I recently upgraded my network to gigabit; turns out I just needed to replace one 10/100 switch and it cost all of $30 to do so.

However, I can't see going faster.  I find I top out my network speeds at about 40-45 MB/s (uploading to my NAS).  Looks like HD speed is the limiting factor.  I suppose if I had striped RAID on both ends, it would be faster, but honestly doesn't seem worth it at this point.

[wesbrown18]

Gigabit is definitely worth it over 10/100.  I recently upgraded my network to gigabit; turns out I just needed to replace one 10/100 switch and it cost all of $30 to do so.

However, I can't see going faster.  I find I top out my network speeds at about 40-45 MB/s (uploading to my NAS).  Looks like HD speed is the limiting factor.  I suppose if I had striped RAID on both ends, it would be faster, but honestly doesn't seem worth it at this point.

If you just replaced your switch, you should consider replacing the ethernet cabling with good quality Cat-5e.  I've done extensive testing with Gigabit, and find that usually when we hit 50MB/s, it's an issue with the cabling.  What's adequate for 100mbps isn't adequate for Gigabit.

You can get about 100MB/s to and from a consumer-grade NAS if you have about four drives.  If you have it in a RAID-5, it's going to really slow things down.

My own setup pushes 400MB/s, but that's with aggressive caching, look-ahead reads, a caching SSD, and a pair of log SSDs for writes, and 32GB of RAM pretty much devoted to the NAS.

[shponglefan]

If you just replaced your switch, you should consider replacing the ethernet cabling with good quality Cat-5e.  I've done extensive testing with Gigabit, and find that usually when we hit 50MB/s, it's an issue with the cabling.  What's adequate for 100mbps isn't adequate for Gigabit.

I can probably give that shot.  I'd only have to replace two cables between my main PC and my NAS anyway.

[MonMotha]

I find that, with an SSD on one end but just a single, 7200RPM consumer grade revolving metal drive on the other end, the best I can do is about 70-80MB/sec, and that is indeed limited by the spinning metal.  Using scp or sftp sometimes imposes other limits, depending on the CPU in the system on both ends, due to it not being able to run the crypto any faster.  It should be noted that transferring something TO the machine with the spinning metal isn't necessarily limited by the speed of the drive since the OS is free to dump the data into the filesystem cache for later flushing, assuming write-back cache policy which is the default on most UNIXy OSes as well as modern Windows for fixed (non-removable) disks.

Obviously an SSD can go quite a bit faster.  The Intel 520 series I have in my laptop is far and away limited by the speed of the SATA interface, which is only 3Gbps (the laptop is just a hair too old for 6Gbps SATA) less overhead of course.  I typically see ~250MB/sec to/from it.

Gigabit ethernet is cheap as hell these days.  Almost all PCs have NICs that'll do it, and reasonable switches can be had for about $10/port.  Heck, you can even get a "mostly managed" switch for just a hair more (Netgear GS108Tv2 is going for ~$95 these days).  10GbE is quite a bit more expensive.  NICs are a couple hundred used for SFP (+ add optics) or CAT6a PHY, pushing $500-700 new.  You can get some old 10GBASE-CX4 stuff for <$100 on the secondary market.  The switches will really nail you, though.  Figure minimum $200/port.  The cost of 10Gb combined with the fact that most of my systems lack the disk throughput to back it up has somewhat made my shy away from 10Gb.

Regardless, don't expect to hit 10Gbps right now even if you've got the network for it without relatively large disk arrays, striped SSDs, or transfers of predominately in-memory data.  1Gb link aggregates are still somewhat attractive for that reason, too.  If all you need is 2-4Gbps and can afford to burn the extra ports and deal with the administrative overhead of the LAG, it's usually cheaper.

There are circumstances where disk throughput doesn't matter.  Distributed file systems where the server does in-memory caching are a great example.  You can have multiple systems with a coherent view of some files, and have it all run at very usable, essentially local speeds using commodity 10GbE hardware.  It generally (but not always, if you consider used gear) works out cheaper than Infiniband.  These applications are somewhat esoteric, though, and I doubt you'd ever encounter them in a home-use scenario.