Nightmare - External hard drive crashes to floor

[ark_ader]

All these specs assume you have a quality burner, which most people don't.  For your cheapo $35 CD/DVD burner, quarter the lifetimes.

Heh.  Thats Scammer Info  IMOP.  Prove it with any form of Logic.

It's not really a fair challenge cos you are limiting him to logic and not evidence, when the obvious method of proof here would be citation rather than argument.   

+1 

[MonMotha]

You asked for an explanation, so here's one, albeit a bit technical at times.

As a data point on flash memory retention, Google a datasheet for the MT29F1G08.  This is a 1Gb (256MB) SLC NAND device from Micron that is reasonably mature (it's been out for several years).  Micron recently placed all their NAND flash datasheets under NDA for some reason I can't comprehend, but this one's old enough that copies not covered by NDA are still floating around.  The datasheet says 10 years over full temp range, which goes to 85C on the extended temp parts or 70C on the commercial temp parts.  I've not known Micron to lie about things like this.  They've been making memory devices (volatile and non-volatile) pretty much as long as such devices have existed.  Can they 100% guarantee those specs?  No; the parts haven't existed long enough.  However, they do accelerated aging tests to arrive at those numbers, and I'd say they have the data to back those aging tests up.  They spec these things out for military/aerospace use too, and those guys run their own independent tests to validate.

Do be aware that the higher density devices, even in SLC, sometimes have slightly lower reliability due to the smaller geometries.  Generally, they make up for it by applying lots of ECC.  MLC devices also can have lower reliability since they have to discriminate between 4 or 8 levels per cell rather than just 2.  Again, these larger devices tend to have lots of ECC to mitigate this.

As another data point, the microcontroller I'm currently using from Atmel specifies 15 years flash data retention across operating temp range (-40 to +85C).  Atmel's also been at this a while.  Reliability of microcontrollers is generally taken to be a pretty freaking big deal, much more so than in the consumer market segment.  These things get used in applications where failure could result in injury or death.  Obviously, the designer of the device should (and generally does) take precautions to make the device "fail safe", but even then, failure is often costly.

In talking with my reps from various companies that do non-volatile memories, there is apparently something of an exponential relationship between temperature and retention.  A device that retains 10 years at 85C may only last a few hours at 150-200C.  There are often characterization charts that exceed the specified operating ranges kicking around inside semiconductor companies for many devices (yes, I've worked for one, and yes, I've seen such charts for various devices, though not flash memories).  If you ask nicely (and often sign an NDA), you can often get access to those charts with the caveat that the data is not "guaranteed".  Semiconductor companies take electrical performance characteristic guarantees very seriously.  Guaranteed characteristics are generally valid all at once, so worst case across the board.  I've never had a semiconductor device fail to meet basic electrical performance characteristics in a manner that wasn't disclosed as an errata.  In general, devices VASTLY exceed their specified performance.  I once ran a micro specified for 6MHz@5V at 20MHz@3V, though it couldn't go any higher, and I was operating it at room temp only.  Generally, reliability problems are design related, but I have also found functional errata (the device does not behave entirely as described) before.

Most of the industry reports and technical literature I read seems to agree with these numbers.  There are also flash device integrators out there who make SD and CF cards that will sell you devices with a warranty.  These guys will say, for example, "5 years minimum retention at 85C or your money back plus losses incurred up to $X".  Expect to pay a substantial premium for such devices just because of the insurance they have to carry, let alone to more rigorous design review and testing that goes into such devices, but such integrators do exist.  The fact that people will put their money where their mouth is again says to me that the numbers are probably at least reasonable.  Flash memory first started showing up in the mid to late 90s, and most of it is still happily chugging away, so a number of "10 years" is at least pretty well validated at this point without any accelerated testing.

As to optical media, my experiences are much more anecdotal.  I do have media burned in the late 90s that still seems to read fine on various readers.  You can get special "archival" media.  Memorex has a "gold" line that they claim 300 years in highly controlled storage.  I'd expect (though without any data to back it up) that a typical controlled home environment could get at least within an order of magnitude of that, so 30 years.  I've got friends who have pulled out CDs from the early days of CD burners who can also still read them just fine.  That's generally 10+ years.  Your media lifetime will also be affected by data mode: CD-ROM or CD-DA.  CD-DA has far less error correction on it due to the assumption that an incorrect bit here or there does not render the disc totally unusable (it just creates an audio artifact), while a single bad bit CAN render a CD-ROM useless.

Now, as to writer quality, that really is just based upon my experiences, I'll admit.  The "quarter the lifetime" suggestion was an attempt to be conservative in the resulting media lifetimes.  After all, if you expect it to last 10 years but it lasts 40, you probably won't complain.  If you expect 40 but only get 10, you may be in trouble.  However, my reasoning on doing this is based upon actual experience.  My discs burned on quality burners all still seem to work, but I do have discs burned with cheap writers that are unreadable after 3-6 years of reasonable storage (though I'll admit that they are in general also on crappy media).  I've also noticed that newer, cheaper writers are more likely to create discs which have trouble being read by other devices (especially standalone CD players) even when writing at slower speeds.  My old Plexwriter 16/10/40A seems to be able to write the cheapest media I throw at it at 16x and produce something that reads on pretty much everything I chuck that disc into (from standalone players to other burners to CD-ROM drives that have seen 10 years of arcade duty) with no issues.  Again, anecdotal I know, but it at least suggests that there is a difference.   I've read some articles to this effect as well, but nothing scholarly (though I'm guessing something is available).

[Xiaou2]

I just don't think you're the god that you claim to be. 

 Heh.  I dont think Ive ever claimed myself as any form of Deity lol.  But man, Im somewhat
flattered that you credit me so high, to be that jealous.  Im just some guy whos average at some
things, and have been dedicated enough to get very exceptional in other areas.. much the same as
many others out there who you may be jealous of.

 I do have gifts in some areas, such as design/art, spatial awareness, heightened awareness
thru ability to make associations, and mechanical ability... but for all my gifts, I have a slew of
problems which darken my days.  Poor memory retention, a hair of OCD (which causes mild depression at times), poor level of ability to fully understand complex information... so will never be
a programmer, do complex maths, etc. 

 Health isnt the greatest.  Tired too much. (poor ability to process food to energy?)  Horribly crooked teeth I cant afford to fix. Larger pores which cause me to often get 2 whiskers growing
out of one pore... ugg

 Basically, Im Human, like everyone else.

It's not really a fair challenge cos you are limiting him to logic and not evidence, when the obvious method of proof here would be citation rather than argument.

 Heh.  I guess so    But...

 I seriously doubt anyone will be able to dig up actual proof that one burners 'Burned Cd' will last
less years than a higher priced burner, which is exactly why I asked for some sort of Logical
reason for that.

 There IS physical proof that the MEDIA you burn on your plays the biggest role in data lastability.
If you buy a DVD movie at the store, that DVD will have been burned with a different type of
media than what they sell at the stores for your PC burner.  Of course, the burners they used to
burn those discs are different too..  But if you put the PC media discs into them, them will
fail just as fast as a cheap home burner.

 A burner is simply a laser that burns pits into a surface.  The pits are no different from
a pro burner, than a home burner,  ...else they would not be able to be read by the same types
of players.

 A pit does not change over time based on the quality of Laser that burned it. The only thing
that can change, is if the surface the data is on, is scratched/flaked off. (which happens on
cheaper media, or simply media meant for home burners)

 So, If Im missing something here... then correct me... but logically, that theory that a
$200 burners burned cds will last longer than a $50  burners cds... is IMOP, seriously flawed.

[MonMotha]

The issue is that some burners may create a better transition in the media than others (the media quality obviously has a lot to do with this, as well).  The higher quality transition has more margin for error than does the lower quality one meaning that it is more likely to be able to be recovered properly given the same amount of media degradation.  This is generally true of any storage medium (optical, magnetic, electrical e.g. flash, ink on paper, chiseled wood, etc.).  The better your SNR to start, the more margin you have for recovery.  Now, whether or not cheap burners have less margin to begin with than more expensive/higher quality models may be questionable, but my experience has been that there is, indeed, some sort of phenomenon along these lines going on.

Imagine my goal is to burn a hole in a piece of quad-rule graph paper using a laser.  If a square has a hole in it, that's a 1.  If it has no hole, that's a zero.  I need a laser powerful enough to actually burn a hole in it, but not so powerful that it just sets the whole piece of paper on fire.  Furthermore, I need to control the size of the hole.  Over time, the holes will either get bigger or they may get plugged with debris.  I need a hole that's big enough to be reliably seen but no so big that it causes adjacent squares to also look like they have a hole burned in them even if they were not intended to.  You can see how my laser power would have an effect here.  Same goes for CD media, but the squares are way smaller, and you're just causing a change in reflective/diffractive properties by deforming or ablating a dye layer rather than burning a hole straight through the thing.

[shmokes]

I don't know much about this, but I can at least hypothesize about scenarios in which you would be wrong.

First, it's important to understand that commercial DVDs and CDs (i.e. movies and music) are NOT burned.  They are pressed.  An image of the data is tooled and physically stamped into the polycarbonate to create the permanent pits that make up the little 1s and 0s.  Since pressing DVDs at home is not feasible they needed another method, hence the burners.  But the burners don't burn pits right into the plastic to mimic what's going on with the commercial DVDs and CDs.  The laser inside a burner isn't nearly that powerful.  Instead, DVD-Rs and CD-Rs have a layer of dye in them that reacts to the laser; the laser turns the dye opaque if I'm not mistaken.  This dye isn't nearly so robust as a physical pit pressed into the plastic, which is what makes a commercial disc so much sturdier and more reliable than one burned at home.

So, with that in mind I would suggest (mind you without any basis) that perhaps the dye layer reacts differently to different intensities of laser.  For example, maybe the spec calls for the dye to be heated to 200 degrees in order to trigger the effect that turns the dye opaque.  This is a spec that produces highly consistent and relatively permanent results.  But maybe using a cheaper to manufacture laser that only heats the dye to 180 degrees turns out discs that work nearly as often, but the dye is more likely to revert back to its previous state, or simply fade into something unreadable in a shorter period of time.

I have no idea if this is the case, but it or something like it seems at least plausible.  In other words, without more I see little reason to dismiss the possibility out of hand.



edit: heh . . . I hadn't read MonMotha's post when I posted, but it's pretty close to what I was guessing at

[CheffoJeffo]

Basically, Im Human, just not like everyone else.

Fixt.

[CheffoJeffo]

Xiaou, the last time you asked for citations in a storage-related discussion and somebody gave you references, you dismissed them, insisting that there was a conspiracy between Google and hard drive manufacturers ... or something equally entertaining.

And  to MonMotha for good explanations of his position.

[protokatie]

...And to actually add NEW ideas to this thread;

If the data you want to protect is of a photographic nature and is important enough, there is a new technology out that uses processed wood scrap to make a medium, then it is impregnated with salts that specifically absorb or re-emit certain frequencies of light. You can only store one image per medium, but it has been proven to last atleast 130 years.

Just a thought. Also, it is highly portable, no OS is required and electricity is only required if it is dark out and there is no fire nearby.

[saint]

Just don't get it wet...

...And to actually add NEW ideas to this thread;

If the data you want to protect is of a photographic nature and is important enough, there is a new technology out that uses processed wood scrap to make a medium, then it is impregnated with salts that specifically absorb or re-emit certain frequencies of light. You can only store one image per medium, but it has been proven to last atleast 130 years.

Just a thought. Also, it is highly portable, no OS is required and electricity is only required if it is dark out and there is no fire nearby.

[protokatie]

Just don't get it wet...

Nor get it too close to the fire, but how is this any different from modern media? Submerse an Hdd, and it is likely toast. A CD-r will fair better with water, but heat will kill it. This just goes back to the idea of storing data on a multitude of media (recommended) as opposed to a view of only one medium (not recommended). Besides, paper is really useful, and for many applications, it is durable and long lived. I remember them(c) saying we would be a paper-less society by 2015. Doesn't look like that will happen. The stuff is too useful.

[saint]

You put entirely too much into my reply

[SavannahLion]

MonMotha, I'm a little confused at your post there. At first you write about individual components, then it seems like you're talking about integrated devices.

From an archival standpoint, the fact an SLC or any other component lasts 10 years across a temperature range matters jack when another component on the device lasts only say.... three to five. You're going to end up going with the smaller value and paying out the ass to recover if that ever happens.

(I would kill to find out where my 16/10/40A is. I'm stuck using this Pioneer DVR-116D in the meantime)

[MonMotha]

MonMotha, I'm a little confused at your post there. At first you write about individual components, then it seems like you're talking about integrated devices.

From an archival standpoint, the fact an SLC or any other component lasts 10 years across a temperature range matters jack when another component on the device lasts only say.... three to five. You're going to end up going with the smaller value and paying out the ass to recover if that ever happens.

(I would kill to find out where my 16/10/40A is. I'm stuck using this Pioneer DVR-116D in the meantime)

Data on discrete components is much, much easier to obtain since most device manufacturers just plain don't give it out (or sometimes even know it!) or will only give a MTTF/MTBF which isn't a particularly useful number.  Device integrators don't seem to take "product specifications" as seriously as semiconductor makers.  You have to pay a LOT more than normal consumer products to get warranties that cover data loss due to product failure.  That's just the way the industry has gone.  However, there are some integrators that do publish specs and who are known for consistently meeting their specs (and often exceeding them) while others can frequently fail to deliver.  Caveat emptor.  You do often get what you pay for.

One other thing to consider is that there are a LOT of flash media integrators but not many flash semiconductor makers (at this point, there's only 2 or maybe 3).  Therefore it's easier to speak generally about the bare semiconductors than the integrated devices.

Integrated device lifetimes DO vary, and that's what you care about in the end.  The cheap crap sometimes barely makes it a year.  Mostly, this is due to terrible firmware (bad block replacement or lack thereof, bad wear leveling or lack thereof, etc.) but sometimes also cheap components or improper design (bad power is a common issue).  If you intend on archiving on flash media, make sure you get something that doesn't suck.  Sandisk has some products that they give real specs on (5 years minimum lifetime non-operating from -55C to +95C or 43,800 power-on hours is a spec I just pulled for one of their SSD products, but at the prices they sell them normally, there is no warranty to that effect), but generally only on their OEM products.  Sandisk does have a reputation for meeting their specs, at least.  I also note that their standard SD cards include a 5 year warranty.  While they may not be on the hook for data loss, they would have to deal with the cost of replacement and all the logistics of failures, so I'd guess they're pretty confident that the devices will last at least that long.

When people write up warranties and such, they tend to be rather conservative.  It's expensive to pay out on a warranty (it generally costs a lot more than the cost of the product).

As another anecdotal data point, I do have a few CF cards from ~2003-2004 that still seem to read fine, but it's been a couple years since I tried them.  I did have one really cheap card that died about a year after I got it, but it wasn't due to flash retention issues.  It just completely stopped working.  A failure nonetheless, but again, it was a real cheapie.  I think all my Sandisk and Kingston branded media still works.

As an unusually topical example, there are lots of arcade games floating around that store their data on flash memory.  In many cases, a standard PCMCIA or CF card is used.  Compared to everything else that goes wrong in these things, the flash devices seem to be very, very reliable.  These things sit around for years powered up and operating every day.  Flash memory is EVERYWHERE.  It's been used to store firmware on many production devices for the past 10-15 years.  On the whole, it sure seems pretty reliable.

My experience is mostly with discrete devices as that's what I'm normally specifying when doing designs.  Retention and temp range is definitely something I look at.  I have been known to request additional characterization data from vendors, and they're generally happy to provide.  If you do plan on archiving using flash media (and I can't say I wholly recommend it), you should get a device from a reputable integrator, and, of course, you should always have more than one copy.  Overall device failures tend to be sudden and catastrophic due to some outside influence or design flaw, while bitrot due to floating gate charge degradation on flash happens over time at a fairly steady pace (you just don't notice it because, as long as the sense amp can still figure things out, a nice, clean 1 or 0 comes out).  Multiple copies guards especially well against the former, while it doesn't really help much with the latter.  Of course, multiple copies on different media types is even better.

Nothing is immune to failure.  Unexpected stuff happens.  That's why you keep multiple copies.  Heck, that's why you're keeping backups in the first place.  If you've got reasonable assurance that something will last 5-10 years with the potential for it to last several dozen more, I'd say that's a pretty good medium term storage solution.

[Grasshopper]

One thing I intend to do relatively soon, ie when I have a real job and can afford it, is start backing up to an online location for example with Amazon's S3 cloud storage offering.  I've never used it but Microsoft offers 25 GB of free cloud storage with Windows Live SkyDrive.  It requires a LiveID, but I'm pretty sure that by now everyone already has one.

As for photos, there's really no excuse to lose digital photos.  How many unlimited free services do you need before you get your photos backed up online?  Flickr, Picasa, Shutterfly, Photobucket . . . even Facebook.  Apps exist for all of these to automate the process so you can just mass upload your photos.  Get.  It.  Done.

Personally, I think the whole cloud computing idea has been a bit overhyped. It requires a degree of trust in large faceless corporations that I simply don't have.

Online backup is fine as long as it's being done in addition to local backup, and the data being backed up is either of a non-confidential nature, or strongly encrypted. But if those conditions are not met then I wouldn't touch it with a bargepole.

The free services are the worst. They can (and sometimes do) pull the plug at any time without warning and you have absolutely no redress. Just look at Geocities for example. Even the paid for services give a false sense of security. Sure, you can sue the company if they lose your data but if they go bust you're still screwed. And even if you can sue them that doesn't actually get your data back. This might sound paranoid but events of the past year have shown us that even large household names can go bust overnight without warning.

There's also the issue of data security. Even if the company is well known, there's so much outsourcing and outshoring going on these days that you can never be really sure who you're dealing with.

[shmokes]

But as far as free services are concerned I'm mainly talking about backing up photos.  And by backing up, I mean you have a local copy, and your backup is uploaded to your Flickr or Picassa account.  Sure those companies could go belly up (Picassa is Google ), but the chances that the company happens to go belly up at exactly the same time that your house burns down or you accidentally drop your computer in a tub of water are not very high.  But even for other data, for example backing up to Microsoft's SkyDrive . . . there are definitely concerns and caveats, but no matter what you have to work within your means.  There is ALWAYS a better backup option.  ALWAYS.  ALWAYS.  ALWAYS.  But the better solution may not be feasible, whether because of finances, technical capabilities or motivation/discipline.  You have to ask yourself, what can I do, and what am I willing to do.  Then see what options correspond with the answers to those questions.  For some people the free services are very likely the best/only realistic option.