backtop

Print E-mail del.icio.us 43 comment(s) - last by root mean sq.. on Apr 10 at 1:22 PM
Digital media poses significant challenges for archiving data

Today we really don’t put much thought into the way we store our data. Yet people all over the country and the world save data that is important for many reasons each day onto media that may not last the decade, much less the century.

For a lot of people, the most important bits and bytes of data are more personal in nature. Things like digital images for our children, family and friends -- the sort of data that only 20 years ago would have been on film or a photo produced via traditional processing methods. Today we archive our photos on external hard drives, CDs and DVDs.

The problem with the archival formats we use today is twofold. The first issue concerns whether the media itself will last long enough for data integrity and secondly, even if the media will last for the next decade or more, will the needed hardware and software to read the files be available.

Vita Paladino, director of Boston University’s Howard Gotlieb Archival Research Center told The Boston Globe, “Who knows how long they're [digital media] going to last - how much time before the information on a zip disk just goes into heaven, cyberspace heaven."

A perfect example is the 5.25-inch floppy disk. Around the country there are untold numbers of these discs with potentially valuable information on them that may well still be viable. The problem is reading the discs; 5.25-inch floppy readers simply aren’t widely available.

This exact problem poses a significant obstacle for data archivists. Francine Berman, director of the University of California San Diego Supercomputer Center and head of a digital protection taskforce told The Boston Globe, “You can file and forget a book, but our storage media will see the next generation every two to five years, and if we want to keep that material we have to carefully migrate it from one generation to the next."

Archiving digital data and migrating it to newer forms of storage as the need presents will present a daunting task over the next several years to those tasked with archiving it. The world’s data is estimated to reach 1.8 zettabytes by 2011.

Comments     Threshold


This article is over a month old, voting and posting comments is disabled
Not such a problem
By AlphaVirus on 4/9/2008 11:13:52 AM , Rating: 3
quote:
Archiving digital data and migrating it to newer forms of storage as the need presents will present a daunting task over the next several years to those tasked with archiving it.

Maybe for enterprise and company users but not so much consumers.
A consumer can purchase a PC with both a floppy drive (old age) and a cd/dvd burner (new age). They can also visit Frys and pickup a Zip drive (old age).
For your standalone players, you can purchase a combo VCR+DVD player, for a little extra VCR+DVD burner.
CD/DVD burners (old age in a few decades) can copy to SSD (new age).

I think we will always be able to incorporate multiple formats and technolgies together to migrate old and new. It takes time and sometimes can be costly but it is definatly possible for the consumer. I am unsure of a large enterprise because scale and price is a huge factor.




RE: Not such a problem
By ecbsykes on 4/9/2008 11:18:16 AM , Rating: 3
I agree, even over the last 10-15 years or so, transition from floppy to CD to Flash Drives has been smooth. I really don't see massive losses of data in the future, unless someone keeps a CD full of important things for 30 years...


RE: Not such a problem
By tastyratz on 4/9/2008 11:45:23 AM , Rating: 5
they have been smooth but they have also been progressively swifter.
Back when the 5 1/4 was sliced bread the computer was no where near as large scale as it is today. I would think as the older generation that ignores it passes it will continue to grow in popularity. Newer technologies would then continually receive more funding and grow at an increasingly rapid rate.
Also someone keeping a cd full of important information will be rudely surprised sooner than 30 years if it was burned on CDR. Cd-r's and dvd-r's are generally not archival media and use an organic base dye. The media deteriorates rapidly and tends to only have a usable shelf life of several years at best.
Have any cd you burned when you got your first burner? see if it reads or maybe do a nero cd/dvd test checking for errors. Don't forget cd-r's have probably only really been popular for 10 years or so and most of my media from my 2x burner no longer works.
stamped cds/dvds are a different story and are created in a completely different process


RE: Not such a problem
By tmouse on 4/9/2008 12:21:34 PM , Rating: 3
I agree, I wonder how many people have many priceless memories stored in "instant" pictures. Most of the early ones have long lasting developer on the backs and in as little as a few years would turn almost black. Now people are using the "photo printers" to print hard copies but a lot of the paper people use is pretty poor and the acid content will yellow the paper as the dye ink also fades. The take home message is do not assume anything is permanent and go through your stuff if you do not want to lose it forever, this is a problem it is just not a new problem.


RE: Not such a problem
By Mitch101 on 4/9/2008 2:55:36 PM , Rating: 3
I often found the printer situation kind of funny. A guy I worked with would brag about how the photo he printed had all these advancements so the color would never fade and block sunlight and last for many years. Why not just print another one a few years down the road when the original does fade? Its digital. As long as you make a copy every couple of years you should be ok.

Critical stuff like photos and private movies I tend to burn 3 times over. 1-for general use. 2-goes to grandparents. 3 gets put away in a dark closet. The odds of the 3 copies going bad is pretty slim. I also archive critical items to old hard drives like 20-60gig hard drives. The odds of multiple losses are slim.

I do wish we would get a consumer version of those 100-300gig dvd's we heard about several years back. I would like to see a replacement to the old tape backup systems.


RE: Not such a problem
By Tsuwamono on 4/9/2008 6:14:06 PM , Rating: 3
when you can get 80gb drives for 20$ at a local store why bother.. thats what i did. I backed up all kinds of data on a bunch of 80gb drives and store them on a shelf in their anti-static bags


RE: Not such a problem
By Mitch101 on 4/10/2008 9:56:03 AM , Rating: 2
Cool $20.00! Every time I check they want $40.00 for an 80 gig.

I actually have a hard time buying small hard drives maybe if I got them for $20.00 I would.

Like now I would buy a 500-750gb and replace a 200gb drive in one of my pc's and like you said backup the important stuff and stick the old 80-120gb drives in the closet. I now have a stack of 8 drives in the closet from 17.4gig to 80gig.

200 gig is now the smallest drive I have in the PC's. The next 500gb+ upgrade the PC will replace a 200 gig. The 200 gig will replace the Tivo's 120gig and the 120gig will backup the important stuff and go in the closet.

Its the hard drive circle of life.


RE: Not such a problem
By Starcub on 4/9/2008 10:59:29 PM , Rating: 2
http://www.kmpmedia.com/premium/gold-preservation-...

Not sure how expensive they are, but they are "comsumer" products. I think it's unfortunate that the ISO standard for DVD archival grade media has taken so long for the industry to approve. At this point hard drives are cheap enough to buy strictly for storage purposes, and would therefore be a much better choice for the average consumer. With USB I wouldn't be too concerned about interface obsolecence either.


RE: Not such a problem
By sheh on 4/9/2008 9:27:41 PM , Rating: 3
AFAIK all my CD-Rs from 10-11 years ago are okay. Perhaps it depends on how your store them? I keep them in standard jewelcases. Another factor might be the media quality.


RE: Not such a problem
By goku on 4/10/2008 4:20:53 AM , Rating: 2
Check them with a utility such as Nero CD-DVD speed, even though I can read my old discs with out errors, if you scan them, even those with out scratches as they were stored in jewel cases, you'll find tons of C1 errors and in my case a lot of C2 errors. You can still take the data off of it even with C2 errors but C2 errors are considered the worst and past that you'll get CRC errors which means you're SOL with that data.


RE: Not such a problem
By DCstewieG on 4/9/2008 12:18:13 PM , Rating: 2
Back when I was in middle school (12 years ago now, geez) we made a time capsule using a new-fangled CD burner. Now I wonder (1) is the CD still readable and (2) what format files were put on it?


RE: Not such a problem
By AlphaVirus on 4/9/2008 12:27:05 PM , Rating: 2
(1) It would simply depend on how it was stored. I have some cd's that I burned in Middle/High school that still work but thats because they were tucked away on the back of a shelf, or closet, in a cd binder that fully zips.

This of course does not guarantee anything because some of those cds scratch much easier than cds of today so you have to be much more careful.

I have since then copied those cds to a hard drive of some sort and transfered to a high capacity DVD, and once again stored somewhere. lol


RE: Not such a problem
By omnicronx on 4/9/2008 12:25:46 PM , Rating: 2
Flash Drives ARE NOT a viable format to use for backups. A simple shock of static electricity, or disconneting it incorrectly from your computer can result in data loss. Flash drives should be used as a temporary storage format, not a permanent backup device.

That being said, I am sure we will have drives that read the CD format for quite some time, so this is not a problem. I would be more worried about how long the media itself will last. I have heard stories about audio discs from the late 80s, and CD backups from early CD burners failing to work after around 15 years. Of course CD technology has gone a long way since then, but it does make you wonder ;)


RE: Not such a problem
By theeq on 4/9/2008 11:22:01 AM , Rating: 2
True, but finding floppy drives is becoming slightly more difficult and some OEM's that I have seen no longer offer floppy drives (though some conveniently package it into a combo card reader deal). I think a bigger issue is the format and what programs can read that format. At the school where I work, we have a veritable dinosaur in the back room that we keep because it is the only one that can read some important files we have. Updating what file is what format is tedious and a lot of people just don't want to do that. It must be really frustrating for some people to suddenly need something on an old disk and discover their computer doesn't recognize the file type.
I'm sure there's some sort of fix for this, whether it be converters or just finding an old copy of the program you need (something that would suck for programs like Office).


RE: Not such a problem
By AlphaVirus on 4/9/2008 11:34:15 AM , Rating: 2
quote:
Updating what file is what format is tedious and a lot of people just don't want to do that.

That is the real problem, laziness.

Technology progresses at a pace that it keeps itself in balance. We will not build the car before building the wheels (logistically speaking).
Although it is a timely process, its easy enough to plop an old floppy into a computer and transfer files to the harddrive/network and simply burn to a cd.

At one point I worked in Records management and would find several companies holding on to floppy disks because they wanted to have access to the information on them. We would always just say "Duh, get a clue, copy them to another space." But that costs money (labor, time, etc) so most companies are not worried about it.


RE: Not such a problem
By murphyslabrat on 4/9/2008 1:54:16 PM , Rating: 2
OK, so paying someone at least $12 an hour (maybe closer to $20 or $30) to make sure that data that cost a great deal to obtain is acceptable? You pay, and then you pay again. I would think that outsourcing the storage to a third-party media-storage service would be the simplest solution.


RE: Not such a problem
By bhieb on 4/9/2008 12:40:05 PM , Rating: 3
I don't think his concern is really that short term. Lets just look at a photo album analogy. Say my grandfather dies, and I find a box of black and whites of my great grandparents or even my great great grandparents. It is something phyisical that still survivived, and is instantly important to me (even tho it was not before I discovered it). Now let say instead I find not a box of floppies as you suggest, but a box of punchards (i know they did not really store pictures). Can I run down to Fry's and find a usb puchcard reader? And not just the device even if i could read it, is there any software that can decompile it to be read on modern equipment.

That is what the concern is not, whether my kids find an old CD with pictures on it, but what happens when my great grand kids do? In that case his point is probably valid, there may be some very intresting and personally important data on that disk, but it will probably just be discarded because the means for retreval 50 years from now just won't be worht the effort, so the data goes to cyber heaven never to be useful again ulike a box of actual pictures.

Of course I don't think my great grand kids really need to see all that porn anyway!


RE: Not such a problem
By bhieb on 4/9/2008 12:46:57 PM , Rating: 3
FYI this is one of the many reasons I sit down for an hour or two each year and create a year in review photo book off of a popular photo site. Not only do I get mega brownie points from the wife, but even if that gets stuck in a box in the attic it will survive many years. Sure the quality may not be that great, but it is something I'm sure I'll treasure later in life when the kids are grown and gone.


RE: Not such a problem
By HOOfan 1 on 4/9/2008 12:56:33 PM , Rating: 2
this is really nothing new to he corporate/government world. Sure the problem is just growing exponentially, but there are places with rooms full of punchcards with nothing to read them. Older generation Optical disks and Backup tapes with no machines to read them. I work with microfilm and we have a vault full of not only microfilm, but also VHS tapes, Betamax tapes, older Optical backup disks and tape backup and 16mm movie reels. Format migration is preached heavily in the archives/record world, yet often people in charge of keeping and maintaining records are the low men on the totem poll and the money and time is never allowed to follow through with the necessary steps.


RE: Not such a problem
By AlphaVirus on 4/10/2008 10:38:07 AM , Rating: 2
quote:
I work with microfilm and we have a vault full of not only microfilm,

This was the case with 1 project I had, vaults and cabinets and shelves of microfilm/fiche. They had about 3 readers but only 1 would power up. Not only was reading them difficult, but the weight is horrindess.
quote:
Format migration is preached heavily in the archives/record world, yet often people in charge of keeping and maintaining records are the lo