Digital storage is problematic

To talk about anything else
Post Reply
User avatar
D.T. Nethery
Posts: 2340
Joined: 27 Sep 2006, 19:19

Digital storage is problematic

Post by D.T. Nethery » 04 Jul 2007, 23:42

Sobering dose of reality to those of use working in a "paperless" , digital environment :



http://news.bbc.co.uk/2/hi/technology/6265976.stm

The National Archives, which holds 900 years of written material, has more than 580 terabytes of data - the equivalent of 580,000 encyclopaedias - in older file formats that are no longer commercially available.

Ms Ceeney said: "If you put paper on shelves, it's pretty certain it is going to be there in a hundred years.

"If you stored something on a floppy disc just three or four years ago, you'd have a hard time finding a modern computer capable of opening it."

"Digital information is in fact inherently far more ephemeral than paper," warned [chief executive Natalie] Ceeney.

She added: "The pace of software and hardware developments means we are living in the world of a ticking time bomb when it comes to digital preservation.

"We cannot afford to let digital assets being created today disappear. We need to make information created in the digital age to be as resilient as paper."

But Ms Ceeney said some digital documents held by the National Archives had already been lost forever because the programs which could read them no longer existed.
And this:

http://www.variety.com/article/VR1117963533.html
Digital proves problematic ;
Industry lacks method to store footage


By DAVID S. COHEN -- "Variety"

As far as movies are concerned, digital, like diamonds, was supposed to
be forever.

No more dyes to fade, no more film stocks to decay or catch fire. Just
pristine digital data, preserved for all time, and release prints as
clear and sharp as the images caught by the camera.

Just one problem: For long-term storage, digital is -- so far --
proving to be a time bomb, more permanent than sand painting but not much else.

Simply put, there's no generally accepted way to store digital
"footage" for more than a few months. After that the industry is using a hodgepodge of improvised solutions, some rather costly, others not very reliable.

That looked like a small problem when digital filmmaking was limited to low-budget indies, animation houses and tech pioneers like James Cameron and George Lucas.

Now, though, that small problem is growing geometrically as the major
studios shift away from film to digital capture. Such recent releases
as "300," "Apocalypto," "Zodiac" and "Superman Returns" were shot on
digital. Their digital masters could be seriously degraded if the problem isn't addressed quickly.

In fact, the problem is so severe that the Academy of Motion Picture
Arts & Sciences' Science and Technology Council warned in 2005 that within just a few years films shot with digital cameras could be lost.

Two years on, digital is going mainstream, but "The problem is still
there," says Phil Feiner, chairman of the Acad Sci-Tech Council's archiving committee. And those few years the council warned of are nearly up.

It's not that there's no way to store digital data. On the contrary,
there are dozens of ways to store it, most of which go obsolete in just a few years. Remember 5" floppies and Zip disks?

And the disks that have stuck around? Not so reliable.

Data tapes are balky and can fall apart. Data DVDs and CDs have a history of "rotting" and can't be counted on to last as long as their commercially pressed cousins.

Plus there's no reason to expect that the computers of 20 years from
now -- never mind 100 -- will be able to plug in to today's hard disks.Some private companies are jumping in as awareness of the problem grows, and Feiner's committee will be launching several initiatives over the months to come.

But the amount of digital footage that needs to be archived is growing
faster than ever.

More than one tech expert, including the Academy's Sci-Tech Council
director Andy Maltz, told Variety they had found archival tapes unreadable just18 months after they were made.

Feiner, the former longtime prexy of Pacific Title, says when he worked
on studio feature films he found missing frames or corrupted data on 40% of the data tapes that came in from digital intermediate houses.

The tapes were only nine months old.

"On certain pictures we had to go into the DI negative and re-scan the
data," he says. "You couldn't retrieve it. Gone."

Milt Shefter, who is a team leader on Feiner's digital archiving
committee, warns that "Long term, it's possible that we're looking going back to the early days of motion pictures, where films are made, put out for a week or two, then thrown away."

With acetate or polyester film, the typical approach to archiving has
been summed up as "store and ignore."

Color film can be turned into black-and-white color separations on
polyester stock. Properly stored in cool vaults at low humidity, such film can last centuries. But there's no way to "store and ignore" digital.

Instead, digital data has to be copied, or "migrated," to new storage
every few years. Migration, however, takes computers, an IT staff, software and a lot of labor. In short: money.

While indies may lack the funds to do regular migration, studios are
plunging in.

Sony's VP of asset management and film restoration, Grover Crisp, says
the studio has put in a program of migrating every two to three years.

"The motion pictures and original material, those are primary assets of
the company," says Crisp. "We all want to do whatever we can to protect those assets."

Disney's VP of production technology Howard Lukk, says as the studios'
digital archives grow, migration becomes a bigger job.

"It's like painting the Golden Gate Bridge and it getting a foot longer
every year."

Not only are more films shot digitally now, but digital filmmaking
encourages directors to shoot more footage.

"The technological issues here are not going to be solved by the
entertainment industry," says Shefter. "It's going to take big
business, big science and maybe big government."

In the meantime, the Academy is stepping in to make the motion picture industry's voice heard in any big business initiative to solve the problem.

The digital archive project is the broadest initiative launched since
the Academy decided in 2003 to fund the current incarnation of the Science & Council.

Maltz expects a report that will pin down what the industry needs to do
to be released in a few months.

Meanwhile, private industry is attacking the problem of digital archiving, too, with at least one announcement in the field planned for NAB.

At NAB, Elektrofilm Digital Studios and Sun Microsystems announced a
service to manage and archive the vast amounts of video from feature film production.

Many tech experts expect the studios to eventually outsource all their
archiving and migration to companies like Elektrofilm rather than try
to do it themselves. Feiner says what is happening is, in effect, the birth of a new business: digital archiving.

He speaks from experience. Earlier this year, three companies received
Science & Technology Awards for their work on archiving. Feiner and his
Pacific Title team were among the winners.

Their solution takes the data from a digital intermediate and turns it
into three-color separation negatives. In other words, they take the digital movie and turn it into good old-fashioned film.
MAC OS 10.11.6 , Macbook Pro Retina 15-inch, Mid 2015,
2.5 GHz Intel Core i7, 16 GB Memory , TVPaint PRO 11.0.6, 64bit -
Wacom Cintiq 21 UX - Wacom Intuos Pro 5 - driver version 6.3.20-8

User avatar
ematecki
Site Admin
Posts: 1944
Joined: 15 Feb 2006, 14:32

Post by ematecki » 05 Jul 2007, 08:03

It hasn't to be as bad as described in these articles...

See http://www.archive.org/about/about.php#storage for examples of how to do it.

Use standard file formats, whose description is publicly available. In the worst case you can hire a programmer to write some piece of software to read your data back. And use preferably open-source standards like png, you'll always find some program able to read it, even if only in an emulator, say in 50 years when windows is hoppefully gone !!!

Also, store all these standard description and software WITH your data, so you can access them as long as you can access your data.

The software side being taken care of, now on to the hardware.

First of, store your data in at least two physically distinct sites, so when one burns down, you still have a second copy. Books in libraries don't have this chance.
Then, as technology progresses, you'll have to migrate your data every 10 to 20 years to more modern hardware. Just remember 20 years ago harddrives capacities where in tens of MegaBytes, today we are close to one TeraByte... You could store eveything you stored 20 years back on 10000 drives on one single drive today. You HAVE to do that to save on space, energy, and maintenance cost. Not forgetting magnetic fields fade slowly out, and some day you can't read them back...

A little bit of software again... while moving your data physically, you can convert it to more modern file formats, and this doesn't add any time to the transfer, the 20 years old drives are soooo slow...

The same is true for tape or CD/DVD backups. Move them to newer and bigger supports before the information stored on them fades out (for mag tapes) or get scratched (for CD/DVD).
And keep a few drives (and computers) for these in working order to be able to read them back in 20 years, just try to locate an 8 inch floppy drive today ! And software too to be able to interpret these data.

How many invaluable books have been destroyed accidentaly (by fire or flooding), or voluntarily (i.e. in war time) ???
With electronic storage and elementary precautions, they would still be around !
You can choose your friends,
but you can't choose your family.

User avatar
fabrice_
Posts: 1146
Joined: 08 Feb 2006, 14:57
Location: France

Post by fabrice_ » 06 Jul 2007, 14:02

Personally, I use to keep all the things I do with my computer like this :

CD/DVD (optical) and HardDrive (magnetic)
If possible : Digital and Paper ( f.i : the user-manuals )
I also use different formats : (tvp and psd) , (odt and pdf) , (audio CD and mp3), etc ...
And as Eric said, I store my most important datas in two physically distinct sites.

I succeeded a few years ago to convert my Amiga floppies (1990) into .adf files, which can be read today with an emulator (UAE).
But yes, all digital datas have to be copied to a new storage every few years ... and IMO, this is very time consuming, and tedious ...

It is certainly a very big challenge for hollywood films. I just can't imagine the amount of datas they have to manage ! :roll:

On the other side, is it possible to get back the old paper storage ?? and is it more ecological to use one harddrive or million books ... ?

It is very difficult to answer to all those questions ... :|

Fabrice Debarge / Beta-Team member / Author of the user-manual.

User avatar
D.T. Nethery
Posts: 2340
Joined: 27 Sep 2006, 19:19

Post by D.T. Nethery » 06 Jul 2007, 14:13

The irony is that Hollywood productions are increasingly switching over to shooting in digital video (which makes sense : smaller cameras, less cost shooting on digital tape instead of 35mm film stock, etc.) but to preserve the finished product they're using 3-strip Technicolor separation negatives for long term storage ! 1930's technology to the rescue.

Thanks ematecki and fabrice for your tips on safeguarding and backing-up files .
MAC OS 10.11.6 , Macbook Pro Retina 15-inch, Mid 2015,
2.5 GHz Intel Core i7, 16 GB Memory , TVPaint PRO 11.0.6, 64bit -
Wacom Cintiq 21 UX - Wacom Intuos Pro 5 - driver version 6.3.20-8

sury rony
Posts: 19
Joined: 01 May 2006, 02:35

Post by sury rony » 13 Jul 2007, 19:19

i think it's like evolution , in a survival of the strong kind a way , i mean evantually if it's important enough u'll find a way to recover the data.
if it's not important enough... too bad .
if u look at digital information A.I (kinda far i know) as a new speciment or entity it's logical .
well to me , anyway.

i recently bought a new PC , and i had to backup files and old works.so i had to browse through all the junk i made and saved and drew ...bottom line : i had to throw away half my files because they weren't " important" enough.

User avatar
D.T. Nethery
Posts: 2340
Joined: 27 Sep 2006, 19:19

Post by D.T. Nethery » 13 Jul 2007, 19:30

I don't think the concern here is with saving every single drawing , animation, or motion picture footage ever made ... there is a sort of intelligence at work in selecting that which is worth saving and that which should be discarded (although that is a bit subjective ... one man's trash is another man's treasured antique) . My concern was with the projects I DO want to save and the apparent lack of consistency with current digital storage methods (such as finding corrupted data in digital storage tapes 9 to 18 months after they were first made) . The irony of backing up productions shot on digital video by saving them to 3-strip technicolor separation negatives (35mm film negs) was rather interesting to me .
MAC OS 10.11.6 , Macbook Pro Retina 15-inch, Mid 2015,
2.5 GHz Intel Core i7, 16 GB Memory , TVPaint PRO 11.0.6, 64bit -
Wacom Cintiq 21 UX - Wacom Intuos Pro 5 - driver version 6.3.20-8

sury rony
Posts: 19
Joined: 01 May 2006, 02:35

Post by sury rony » 15 Jul 2007, 01:03

i meant the same thing , once me and my friend shot a dv film of a week long journey we took through the country , the result was a very nice punk\art video thingie that we were very proud of.
imagine our frustration when all the editing apps and modules couldn't handle our movie because we used an ancient digi-design editing pc.
we've been through hell to extract it to Digital video and DV ,at the end of it we managed to hunt a computer wizard who magiclly saved us.
but it was important enough so we managed .
other project are probably just pieces of plastic in my cabinet.
so now my punk-video-art thingie will live a long life.

User avatar
Peter Wassink
Posts: 3495
Joined: 17 Feb 2006, 15:38
Location: Amsterdam (Basse-Lotharingie)
Contact:

Post by Peter Wassink » 15 Jul 2007, 06:04

sury rony wrote: so now my punk-video-art thingie will live a long life.
will it...?

i think this is exactly davids point.
how do you know it will live longer then 18 months when you only have it on a dvd and how long does a dv tape last?
i assume you didn't print it to film?
Peter Wassink - 2D animator
• PC: Win10/64 Home - Xeon Quadcore E5410@2.33Ghz - 12Gb RAM
• laptop: Win10/64 Pro - i7-4600@2.1 GHz - 8Gb RAM

User avatar
ematecki
Site Admin
Posts: 1944
Joined: 15 Feb 2006, 14:32

Post by ematecki » 16 Jul 2007, 07:27

Then again, tapes or DVD, when properly stored, will last 10 (DVD) to 20 (tapes) years without problem, probably much more.
On the other hand, film, when improperly stored, will propably not last more than 18 month.

So it's not only the medium, but also how you handle and store it !
Big studios store they film in waults, with temperature, moisture and whatever else control. Most of us can't afford that.
We'll have to live with it... and make copies from time to time to 'refresh' the data, and take advantage of the 'better' technology that just hit the market.

And, if film is so good, why not make DIGITAL prints ? Printing a set of dots on film which represents the bits of a digital file ? Would take more film for the same movie, but the final result would be much better quality. And by using high contrast B&W film, the bits would be very well defined and readable even from a somewhat aged film.
You can choose your friends,
but you can't choose your family.

User avatar
Peter Wassink
Posts: 3495
Joined: 17 Feb 2006, 15:38
Location: Amsterdam (Basse-Lotharingie)
Contact:

Post by Peter Wassink » 16 Jul 2007, 07:45

"Digital prints"
Eric, maybe you should apply a patent for this idea ?

i wonder though how much larger(in lenght) will this print have to be?
if the dots become too small, you might again encounter information loss
Peter Wassink - 2D animator
• PC: Win10/64 Home - Xeon Quadcore E5410@2.33Ghz - 12Gb RAM
• laptop: Win10/64 Pro - i7-4600@2.1 GHz - 8Gb RAM

User avatar
evar
Posts: 172
Joined: 10 Feb 2006, 18:40
Location: nb, canada
Contact:

Post by evar » 16 Jul 2007, 10:14

I used to do this with my Amiga back in the day, sortof - the VBS (video backup system) - would generate black square patterns output to video and you could them capture it and upon playback, retrieve data. Same idea, but using vhs tape. It was pretty effective, but I doubt I could hook it up and see my backups now though....
Evar Simon

User avatar
Peter Wassink
Posts: 3495
Joined: 17 Feb 2006, 15:38
Location: Amsterdam (Basse-Lotharingie)
Contact:

Post by Peter Wassink » 16 Jul 2007, 10:30

hehe , i remember that system
very clever

except that magnatic tape has about the worst reputation when it comes to longevity
Peter Wassink - 2D animator
• PC: Win10/64 Home - Xeon Quadcore E5410@2.33Ghz - 12Gb RAM
• laptop: Win10/64 Pro - i7-4600@2.1 GHz - 8Gb RAM

sury rony
Posts: 19
Joined: 01 May 2006, 02:35

Post by sury rony » 18 Jul 2007, 12:06

Tantalus wrote:
sury rony wrote: so now my punk-video-art thingie will live a long life.
will it...?

i think this is exactly davids point.
how do you know it will live longer then 18 months when you only have it on a dvd and how long does a dv tape last?
i assume you didn't print it to film?
oh...crap , i gotta go do somthing ...film we say eh ?
film is expensive , indeed we backuped on every possible media ... but film is a bit too much.
i guess i see your point.

User avatar
IceDelight
Posts: 22
Joined: 06 Nov 2007, 19:46
Location: UK

Post by IceDelight » 08 Nov 2007, 14:51

I just accidently deleted a massive message I wrote here describing my intricate thoughts on this subject. I wrote about my personal history and experiences with backing up data, and how everything eventually dies and turns to dust... ALL LOST!!!!!!!!!!!!! I HATE CLICKING PASTE INSTEAD OF COPY BY MISTAKE!...

I guess it's ironic that I lost my message.

The last point in my message was that I want to get an Ultrium tape drive, the tapes are priced ok, but the drives are really expensive. Also, it looks like they are a lot harder to use than, say, an external USB HDD. Tape drives are apparently good for archiving stuff. But they of course also don't last forever, and I'm not saying they're the answer to film studios problems. But perhaps one would be good for personal backup. I will still get one if I'm rich one day.

(( I just realized I think I could have right-clicked and chosen undo, but it's too late now because I wrote something else :( ))

Post Reply