backtop

Print E-mail del.icio.us 43 comment(s) - last by root mean sq.. on Apr 10 at 1:22 PM
Digital media poses significant challenges for archiving data

Today we really don’t put much thought into the way we store our data. Yet people all over the country and the world save data that is important for many reasons each day onto media that may not last the decade, much less the century.

For a lot of people, the most important bits and bytes of data are more personal in nature. Things like digital images for our children, family and friends -- the sort of data that only 20 years ago would have been on film or a photo produced via traditional processing methods. Today we archive our photos on external hard drives, CDs and DVDs.

The problem with the archival formats we use today is twofold. The first issue concerns whether the media itself will last long enough for data integrity and secondly, even if the media will last for the next decade or more, will the needed hardware and software to read the files be available.

Vita Paladino, director of Boston University’s Howard Gotlieb Archival Research Center told The Boston Globe, “Who knows how long they're [digital media] going to last - how much time before the information on a zip disk just goes into heaven, cyberspace heaven."

A perfect example is the 5.25-inch floppy disk. Around the country there are untold numbers of these discs with potentially valuable information on them that may well still be viable. The problem is reading the discs; 5.25-inch floppy readers simply aren’t widely available.

This exact problem poses a significant obstacle for data archivists. Francine Berman, director of the University of California San Diego Supercomputer Center and head of a digital protection taskforce told The Boston Globe, “You can file and forget a book, but our storage media will see the next generation every two to five years, and if we want to keep that material we have to carefully migrate it from one generation to the next."

Archiving digital data and migrating it to newer forms of storage as the need presents will present a daunting task over the next several years to those tasked with archiving it. The world’s data is estimated to reach 1.8 zettabytes by 2011.

Comments     Threshold


This article is over a month old, voting and posting comments is disabled
Not such a problem
By AlphaVirus on 4/9/2008 11:13:52 AM , Rating: 3
quote:
Archiving digital data and migrating it to newer forms of storage as the need presents will present a daunting task over the next several years to those tasked with archiving it.

Maybe for enterprise and company users but not so much consumers.
A consumer can purchase a PC with both a floppy drive (old age) and a cd/dvd burner (new age). They can also visit Frys and pickup a Zip drive (old age).
For your standalone players, you can purchase a combo VCR+DVD player, for a little extra VCR+DVD burner.
CD/DVD burners (old age in a few decades) can copy to SSD (new age).

I think we will always be able to incorporate multiple formats and technolgies together to migrate old and new. It takes time and sometimes can be costly but it is definatly possible for the consumer. I am unsure of a large enterprise because scale and price is a huge factor.




RE: Not such a problem
By ecbsykes on 4/9/2008 11:18:16 AM , Rating: 3
I agree, even over the last 10-15 years or so, transition from floppy to CD to Flash Drives has been smooth. I really don't see massive losses of data in the future, unless someone keeps a CD full of important things for 30 years...


RE: Not such a problem
By tastyratz on 4/9/2008 11:45:23 AM , Rating: 5
they have been smooth but they have also been progressively swifter.
Back when the 5 1/4 was sliced bread the computer was no where near as large scale as it is today. I would think as the older generation that ignores it passes it will continue to grow in popularity. Newer technologies would then continually receive more funding and grow at an increasingly rapid rate.
Also someone keeping a cd full of important information will be rudely surprised sooner than 30 years if it was burned on CDR. Cd-r's and dvd-r's are generally not archival media and use an organic base dye. The media deteriorates rapidly and tends to only have a usable shelf life of several years at best.
Have any cd you burned when you got your first burner? see if it reads or maybe do a nero cd/dvd test checking for errors. Don't forget cd-r's have probably only really been popular for 10 years or so and most of my media from my 2x burner no longer works.
stamped cds/dvds are a different story and are created in a completely different process


RE: Not such a problem
By tmouse on 4/9/2008 12:21:34 PM , Rating: 3
I agree, I wonder how many people have many priceless memories stored in "instant" pictures. Most of the early ones have long lasting developer on the backs and in as little as a few years would turn almost black. Now people are using the "photo printers" to print hard copies but a lot of the paper people use is pretty poor and the acid content will yellow the paper as the dye ink also fades. The take home message is do not assume anything is permanent and go through your stuff if you do not want to lose it forever, this is a problem it is just not a new problem.


RE: Not such a problem
By Mitch101 on 4/9/2008 2:55:36 PM , Rating: 3
I often found the printer situation kind of funny. A guy I worked with would brag about how the photo he printed had all these advancements so the color would never fade and block sunlight and last for many years. Why not just print another one a few years down the road when the original does fade? Its digital. As long as you make a copy every couple of years you should be ok.

Critical stuff like photos and private movies I tend to burn 3 times over. 1-for general use. 2-goes to grandparents. 3 gets put away in a dark closet. The odds of the 3 copies going bad is pretty slim. I also archive critical items to old hard drives like 20-60gig hard drives. The odds of multiple losses are slim.

I do wish we would get a consumer version of those 100-300gig dvd's we heard about several years back. I would like to see a replacement to the old tape backup systems.


RE: Not such a problem
By Tsuwamono on 4/9/2008 6:14:06 PM , Rating: 3
when you can get 80gb drives for 20$ at a local store why bother.. thats what i did. I backed up all kinds of data on a bunch of 80gb drives and store them on a shelf in their anti-static bags


RE: Not such a problem
By Mitch101 on 4/10/2008 9:56:03 AM , Rating: 2
Cool $20.00! Every time I check they want $40.00 for an 80 gig.

I actually have a hard time buying small hard drives maybe if I got them for $20.00 I would.

Like now I would buy a 500-750gb and replace a 200gb drive in one of my pc's and like you said backup the important stuff and stick the old 80-120gb drives in the closet. I now have a stack of 8 drives in the closet from 17.4gig to 80gig.

200 gig is now the smallest drive I have in the PC's. The next 500gb+ upgrade the PC will replace a 200 gig. The 200 gig will replace the Tivo's 120gig and the 120gig will backup the important stuff and go in the closet.

Its the hard drive circle of life.


RE: Not such a problem
By Starcub on 4/9/2008 10:59:29 PM , Rating: 2
http://www.kmpmedia.com/premium/gold-preservation-...

Not sure how expensive they are, but they are "comsumer" products. I think it's unfortunate that the ISO standard for DVD archival grade media has taken so long for the industry to approve. At this point hard drives are cheap enough to buy strictly for storage purposes, and would therefore be a much better choice for the average consumer. With USB I wouldn't be too concerned about interface obsolecence either.


RE: Not such a problem
By sheh on 4/9/2008 9:27:41 PM , Rating: 3
AFAIK all my CD-Rs from 10-11 years ago are okay. Perhaps it depends on how your store them? I keep them in standard jewelcases. Another factor might be the media quality.


RE: Not such a problem
By goku on 4/10/2008 4:20:53 AM , Rating: 2
Check them with a utility such as Nero CD-DVD speed, even though I can read my old discs with out errors, if you scan them, even those with out scratches as they were stored in jewel cases, you'll find tons of C1 errors and in my case a lot of C2 errors. You can still take the data off of it even with C2 errors but C2 errors are considered the worst and past that you'll get CRC errors which means you're SOL with that data.


RE: Not such a problem
By DCstewieG on 4/9/2008 12:18:13 PM , Rating: 2
Back when I was in middle school (12 years ago now, geez) we made a time capsule using a new-fangled CD burner. Now I wonder (1) is the CD still readable and (2) what format files were put on it?


RE: Not such a problem
By AlphaVirus on 4/9/2008 12:27:05 PM , Rating: 2
(1) It would simply depend on how it was stored. I have some cd's that I burned in Middle/High school that still work but thats because they were tucked away on the back of a shelf, or closet, in a cd binder that fully zips.

This of course does not guarantee anything because some of those cds scratch much easier than cds of today so you have to be much more careful.

I have since then copied those cds to a hard drive of some sort and transfered to a high capacity DVD, and once again stored somewhere. lol


RE: Not such a problem
By omnicronx on 4/9/2008 12:25:46 PM , Rating: 2
Flash Drives ARE NOT a viable format to use for backups. A simple shock of static electricity, or disconneting it incorrectly from your computer can result in data loss. Flash drives should be used as a temporary storage format, not a permanent backup device.

That being said, I am sure we will have drives that read the CD format for quite some time, so this is not a problem. I would be more worried about how long the media itself will last. I have heard stories about audio discs from the late 80s, and CD backups from early CD burners failing to work after around 15 years. Of course CD technology has gone a long way since then, but it does make you wonder ;)


RE: Not such a problem
By theeq on 4/9/2008 11:22:01 AM , Rating: 2
True, but finding floppy drives is becoming slightly more difficult and some OEM's that I have seen no longer offer floppy drives (though some conveniently package it into a combo card reader deal). I think a bigger issue is the format and what programs can read that format. At the school where I work, we have a veritable dinosaur in the back room that we keep because it is the only one that can read some important files we have. Updating what file is what format is tedious and a lot of people just don't want to do that. It must be really frustrating for some people to suddenly need something on an old disk and discover their computer doesn't recognize the file type.
I'm sure there's some sort of fix for this, whether it be converters or just finding an old copy of the program you need (something that would suck for programs like Office).


RE: Not such a problem
By AlphaVirus on 4/9/2008 11:34:15 AM , Rating: 2
quote:
Updating what file is what format is tedious and a lot of people just don't want to do that.

That is the real problem, laziness.

Technology progresses at a pace that it keeps itself in balance. We will not build the car before building the wheels (logistically speaking).
Although it is a timely process, its easy enough to plop an old floppy into a computer and transfer files to the harddrive/network and simply burn to a cd.

At one point I worked in Records management and would find several companies holding on to floppy disks because they wanted to have access to the information on them. We would always just say "Duh, get a clue, copy them to another space." But that costs money (labor, time, etc) so most companies are not worried about it.


RE: Not such a problem
By murphyslabrat on 4/9/2008 1:54:16 PM , Rating: 2
OK, so paying someone at least $12 an hour (maybe closer to $20 or $30) to make sure that data that cost a great deal to obtain is acceptable? You pay, and then you pay again. I would think that outsourcing the storage to a third-party media-storage service would be the simplest solution.


RE: Not such a problem
By bhieb on 4/9/2008 12:40:05 PM , Rating: 3
I don't think his concern is really that short term. Lets just look at a photo album analogy. Say my grandfather dies, and I find a box of black and whites of my great grandparents or even my great great grandparents. It is something phyisical that still survivived, and is instantly important to me (even tho it was not before I discovered it). Now let say instead I find not a box of floppies as you suggest, but a box of punchards (i know they did not really store pictures). Can I run down to Fry's and find a usb puchcard reader? And not just the device even if i could read it, is there any software that can decompile it to be read on modern equipment.

That is what the concern is not, whether my kids find an old CD with pictures on it, but what happens when my great grand kids do? In that case his point is probably valid, there may be some very intresting and personally important data on that disk, but it will probably just be discarded because the means for retreval 50 years from now just won't be worht the effort, so the data goes to cyber heaven never to be useful again ulike a box of actual pictures.

Of course I don't think my great grand kids really need to see all that porn anyway!


RE: Not such a problem
By bhieb on 4/9/2008 12:46:57 PM , Rating: 3
FYI this is one of the many reasons I sit down for an hour or two each year and create a year in review photo book off of a popular photo site. Not only do I get mega brownie points from the wife, but even if that gets stuck in a box in the attic it will survive many years. Sure the quality may not be that great, but it is something I'm sure I'll treasure later in life when the kids are grown and gone.


RE: Not such a problem
By HOOfan 1 on 4/9/2008 12:56:33 PM , Rating: 2
this is really nothing new to he corporate/government world. Sure the problem is just growing exponentially, but there are places with rooms full of punchcards with nothing to read them. Older generation Optical disks and Backup tapes with no machines to read them. I work with microfilm and we have a vault full of not only microfilm, but also VHS tapes, Betamax tapes, older Optical backup disks and tape backup and 16mm movie reels. Format migration is preached heavily in the archives/record world, yet often people in charge of keeping and maintaining records are the low men on the totem poll and the money and time is never allowed to follow through with the necessary steps.


RE: Not such a problem
By AlphaVirus on 4/10/2008 10:38:07 AM , Rating: 2
quote:
I work with microfilm and we have a vault full of not only microfilm,

This was the case with 1 project I had, vaults and cabinets and shelves of microfilm/fiche. They had about 3 readers but only 1 would power up. Not only was reading them difficult, but the weight is horrindess.
quote:
Format migration is preached heavily in the archives/record world, yet often people in charge of keeping and maintaining records are the low men on the totem poll and the money and time is never allowed to follow through with the necessary steps.

Ya, this is the major problem. Big wigs are worried about the "bottom line" and spending $100,000 to get a 3rd party contract company to handle all the records it just not in the budget. No matter how beneficial it is, they will not bother with it.


By Dribble on 4/9/2008 11:18:37 AM , Rating: 2
Surely anyone with half a brain stuck it on their pc hd years ago, and have been copying it/backing it up with all their pc data for years.
Because storage space is always expanding the amount of data that you want to keep from old machines hardly makes a dent in it. I personally have an oldpc folder full of names like:
AMD466
Ath900
P4
...




By roastmules on 4/9/2008 12:00:23 PM , Rating: 2
Agreed. I generally do the same thing.

I did go back and go through a lot of the old archives, such as from the 486 - Pentium II days, and delete nearly everything.

At first, I was reluctant to delete archives of school work, particularly term papers, but then I figured that the paper is too old, there's something newer, and I'm a much, much better writer now then 10 years ago.

Recently, I've started creating archive directories on current computers, so that when it's time to move to a new one, I only copy the archive directories. I'm glad that Microsoft created the My Documents folder, but I'm angry that Outlook uses a pseudo-hidden folder for itself.


By AlphaVirus on 4/9/2008 12:34:17 PM , Rating: 2
quote:
Who actually keeps something they want on 5 1/4 floppies?

Thats like asking
Who still has 8 tracks, who still has vinyls period, who still has cassette tapes, who drives a car older than a 1990, who carries a cell phone that only displays black/white colors, who has VHS in their movie collection, who still owns a Nintendo or Atari game system, who still has 'bunny ears' to watch tv...the list goes on and on

Basically there will always be people "out of the loop" or just too broke to afford newer things, also people who dont know there is a better/newer option. And we cant forget the people who just dont like change.

Its not about having a brain or not, its about personal lifestyle and preference. While I may prefer watching BluRay/HDDVD, someone may think DVD is the only viable option. :)


By murphyslabrat on 4/9/2008 1:59:24 PM , Rating: 2
quote:
who carries a cell phone that only displays black/white colors

No, that's just plain idiotic. Nowadays, you can get a cheapo cell-phone for free with a two-year subscription to just about any mobile service.


By Nacho on 4/9/2008 11:47:20 PM , Rating: 2
I still have a Nokia 1100. B/W display, talk, sms, and it even has a flashlight.


By root mean sq on 4/10/2008 1:22:43 PM , Rating: 2
Not everyone lives in the "golden-age-of-cellphones-opolis" like you do. B/W cells are still the norm here.

and that nokia 1100 pwns!!! those things is frickin' indestructible.


By bhieb on 4/9/2008 12:54:38 PM , Rating: 2
Really? So surely you with "half a brain" has scanned in and backed up all your old physical pictures right? I mean they cannot be lost or damaged either. /sarcasm

It is not really about what is important to you, you may not think it is that big of a deal to toss the CD in a box with the pictures from your kid's first tball game, afterall you were there and now the kid is 16 (life has moved on).

However the point is the data may not be relevant to you, but it would still be relevant to say your great great grand kids escpecially if that tball kid went on to pitch a no hitter in a world series. When they run accross the CD do they even bother trying to see what is on it if they have to pay 10K to some archive retreival firm 50 years from now?


By roastmules on 4/9/2008 2:18:53 PM , Rating: 2
Now there is a good point, which I had not previously considered...

Do they sell archival quaility DVD's and/or CD's? -- Yes.
http://adterrasperaspera.com/blog/2006/10/30/how-t...

I figure that CD/DVD/HD/BR/etc discs will have drives for at least 20+ years.

As for the progression of technology, I see it slowing down further, and further... There are lots of physical limits. Moore's prediction started out at 18 months, and now Intel and others are more in the 24 month range.

Progression from floppies to disc was a MAJOR improvement, as CD/DVD/etc are more reliable, faster and hold more data. Since I can put so much data on a DVD/BR disc, I'm less inclined to upgrade. (Similar situation with DVD transition to hi-def -- I've not gone with HD or BR discs, as the cost is high, and the marginal benefit over DVD is low.)

Look up S-Curve. http://en.wikipedia.org/wiki/Diffusion_of_innovati...


By Dribble on 4/10/2008 7:44:43 AM , Rating: 2
That wasn't the point of the article but if you do scan your best physical pictures you know they will be there for your great grandchildren to look at. If you don't chances are they will get lost of damaged. You are obviously aware that you have physical and digital pictures that are important - instead of sitting here using /sarcasm why not go scan them in, and put em on your pc now. Then presuming you have some method of backup for important pc data you won't loose them.

In a moment of forward thinking I got a kodak photo cd burned in 92 with some of my favourite pictures, which I subsequently put onto my pc, which in turn is securely backed up. Who knows where all the other pictures I took then have gone, but those ones are likely safe for as long as anyone cares too look at them.


By JasonMick (blog) on 4/9/2008 11:12:18 AM , Rating: 2
Its been a long time since I read the Foundation trilogy, but I remember the basic premise was that all the information in the world was stored digitally and when society collapsed, this would all be lost. They had to figure out a way of preserving the knowledge of society, making a "foundation".

Asimov was one those prophetic types, predicting a lot of things before their time.




By MrBlastman on 4/9/2008 11:25:21 AM , Rating: 2
He also brings up a great point in his writings. I've always been fond of his books and just to think that with information become more and more digitally oriented, library attendance dropping - it _could_ happen someday...

A time where books are no longer held in print form and everything is held electronically. If society did collapse and our power infrastructure was compromised by some catastropic event such as nuclear war, there really would be no easy way to access the data. Perhaps in bits and pieces scattered about.

Maybe they are archiving data in Cheyenne Mountain for all we know.

Conspiracy theories aside, it does give room to consider the implications if such a thing were to happen.


By snowbro on 4/9/2008 1:43:26 PM , Rating: 2
True... Medical libraries are pretty much all electronic nowadays... Some new journals are even "ONLINE ONLY" and do not create a printed edition...


"Living archives"
By therealnickdanger on 4/9/2008 11:24:18 AM , Rating: 2
When I think about my archives or backups, I try to view them as something that I'll always have to be diligent with. Every couple years, back them up again onto a new technology, be it floppy-->CD, CD-->DVD, DVD-->HDD, HDD-->SSD, SSD-->DNA?




RE: "Living archives"
By Nik00117 on 4/9/2008 11:44:41 AM , Rating: 2
Just need to contiue to mirgrate form technology to technology.

Recently a teacher of mine got a lot of valuable information from a bunch of 5.25 drives. He made it a project to recover that material and he did.

Question of the matter remainsiss do poeple do this?

I can also think of a situation where a company keeps a extermely old computer running simply because it has one critical application which works on that and no other PC for some odd reason.


RE: "Living archives"
By Golgatha on 4/9/2008 12:19:45 PM , Rating: 2
I think of my backups more like a master copy, and copy of the master copy scenario.

My HDD (MC=master copy) MC goes to other MCs or recordable media (CC=copy of the master copy). In my backup strategy, another MC is made on an internal data only HDD, and also an external HDD of the same capacity as the internal drive. The internal data only HDD is a daily backup, and the external HDD MC is updated periodically (about monthly for me). The external HDD MC is also stored offsite from the other MCs. I suppose a nuclear blast or natural disaster could wipe out both my computer at home and the offsite location of the external MC. However, in these extreme situations I would have had no prior warning to relocate, and would have bigger problems than just data loss; I basically feel my backup solution is sufficiently robust.

Now CCs are made to CD-R, DVD+/-R, USB flash drive, DVD-RAM discs, etc. for specific types of data I might like to conveniently take with me somewhere (e.g. family photos to my parents house, which is out of town).

I do worry much more about the lack of programs able to read the file formats say 100years from now, but I honestly think virtual machines on home computers, or some enterprising company will take care of those issues as well.


By psychobriggsy on 4/9/2008 2:03:56 PM , Rating: 2
This is why I approve of simple data backup systems, from in-house solutions (one click backup tools for Windows, Apple Time Machine, etc), to web based systems (which in 5 year's time will be an awesome digital archive of your data, and able to present it to you intelligently rather than as a data dump).

Just burn a few DVDs (today), BD-ROMS (tomorrow) or Holographic Discs (next week) every year to keep your own local archive of everything. Hardly worth worrying about last year's $1 disc is it?

If you die, then most of that media will only have been personally important to you anyway so who cares if it is lost. "Ooh, it's Uncle Psycho's porn collection, wow, it's only 1080p! Screw that! What, 41.1kHz stereo audio files, who cares!"

Internet backup will mean the end of risking your life to save the photo album's if your house is burning down. Similarly, maybe you should stick your home backup drive near your front door, or in a waterproof container in the shed.




By pgpswiss on 4/9/2008 4:57:16 PM , Rating: 2
Nowadays, there are piles of private files seeming totally uninteresting. But... who can judge what will be interesting 500 or even 100 years from now. Recently, I found old photos taken by my great-grandfather. Certainly totally uninteresting at the time (landscapes and houses near my hometown, nothing special). But a museum was quite happy when they got them...
Contrary to that, ALL current digital information will probably be lost, unreadable or so in a few hundred years. In this case, I wish good luck to the archaeologists of year 3500 or 4000 (if they ever exist) : to understand how the world was in year 2000, they'll have less clues than we have now to understand ancient Egypt...


By ninjit on 4/9/2008 5:26:22 PM , Rating: 2
I've already moved to all internet for a backup solution
$5 a month for unlimited backup with Mozy.com.

But we still need gobs of local storage readily accessible for immediate random use (vs. having to fetch it first from somewhere online).


Archived
By v1001 on 4/9/2008 12:41:56 PM , Rating: 3
I just archived all my data on the brand new HD-DVD format. It just came out it's sure to be safe and be around for the next few decades.




RE: Archived
By snowbro on 4/9/2008 1:41:10 PM , Rating: 2
I echo the comment above...

Back in the day, I used to store data on 3.5" floppies (unreliable, etc) so I usually had 2 backups of anything...

Then, as time progressed, the data got larger, but so did the storage media.... CDs! Well, I started archiving all my pictures, data, etc on CDs...

Well, guess what, DVDs came out.... so now I can fit nearly all my "important" data on 2 DVDs! By important, I mean pictures primarily.... Music can always be re-downloaded, etc, etc...

So, I have a consistent stream of backups since before.... Does it matter that my 3.5" floppy no longer works... No! Cuz any important material from then, would have made it on to a CD from my next gen of backups... and so on and so forth... I have a feeling that my CDs will still work, even DECADES from now, but will that matter.... NOPE

By that time, all my data will likely be on a mini-SD-type card capable of storing a Terabyte, etc... but as usual...

Any consumer who doesn't back up their photos, personal data, on more than just a USB stick and their laptop hardrive is looking to lose it all when the cat spills the cup of coffee on their laptop (and they also happen to lose their USB stick) True story....


trim off the replaceable
By dflynchimp on 4/9/2008 4:43:12 PM , Rating: 2
quote:
The world’s data is estimated to reach 1.8 zettabytes by 2011.


here's an idea, you don't bother to archive the portions of that 1.8 zettabytes that contain porn (leave it to the personal computers to do that) and you should get it down to a much more manageable sub-1.0 zettabyte ;)

who looks at 70's porn anymore anyways?




The 78 record?
By jmunjr on 4/9/2008 7:25:57 PM , Rating: 2
I have some 78 records. Finding a player that supports them is not that difficult. It is called Ebay...and even without that I can still find a player.




A/D Archive...
By teckytech9 on 4/10/2008 1:30:24 AM , Rating: 2
There will be a time in the future where all the data the average consumer wants to archive today will be stored in a tiny spec of material that has not yet been invented.

The digital data that will survived is that which has been carefully and selectively archived over the generations by each individual to pass along to future generations. The analog data that exists today will simply perish in underground vaults, and have succumb to natural decay.




"So, I think the same thing of the music industry. They can't say that they're losing money, you know what I'm saying. They just probably don't have the same surplus that they had." -- Wu-Tang Clan founder RZA

DailyTech Poll
Do you use copy/paste on your smartphone? 




16 Comments









botimage
Copyright 2010 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki