Preserving digital content and converting historical analog content to digital form will be the single largest user of digital storage capacity.  Much of the storage for archiving will be on removable media such as tape and optical disks that can be put on a shelf or a library until needed.  Digital preservation allows content to be available for research and distribution.  Future content businesses will be based on the availability of vast amounts of historical digital content.Following are some examples of historical analog content that should be converted to digital form for preservation.

1. Following are some examples of historical analog content that should be converted to digital form for preservation.

   1. 100,000 hour CNN libraryStock Material at Major Networks
   1. ABC, 1,037,000 films/tapesCBS, 1,045,000 tapes and more than 150,000,000 feet of film
   2. NBC, 600,000 film reels (currently estimated at 100,000,000 feet) and 1,600,000 videotapes
   2. Materials accumulated by Major Studios
   1. Disney, 6,500 TV programs on 80,000 reels and tapesFox, 54,000 TV programs on 780,000 reels and tapesMCA/Universal, 18.000 (through 1994) TV programs on 217,000 reels and tapesParamount (Viacom), 8,000 TV programs on 1,200,000 reels and tapesSony/Columbia, 35,000 TV programs on 600,000 reels and tapesTurner Entertainment, 20,000 TV programs on 337,000 reels and tapes
   2. Warner Brothers, 28,000 TV programs on 1,000,000 reels and tapes
   3. Worldwide estimates are at least 10X this number of historical content material
   4. The older film stocks and substrates are aging rapidly making a major content preservation effort of the greatest priority

Despite the possibility of eventual payback for digitization of historical content as well as the value of preserving irreplaceable historical content it appears that only a small percentage of this content will be converted to digital form and preserved.  In the case of the 100,000 hour CNN analog tape library apparently only 2% of the content has been converted despite diligent efforts.

The reason why digital content preservation is occurring so slowly is that it is generally a very manual process.  Older tapes must be handled carefully and often given special treatment (such as baking in an oven) before being played.  Also many of these tapes are in such poor condition that they can only be played back once before they fall apart and for magnetic media the signals may decay beyond the capability of recovery.

We will first look at technologies for archiving, then at methodologies and standards for archiving, and finally look at ways in which digital conversion and preservation can most effectively be carried.

Archiving Technology

Magnetic tape is the most commonly used data center archival technology. Magnetic tape cartridges are generally believed to last from 15 to 30 years, in ideal environmental conditions. Even in an era of significant emphasis on compliance and records retention, that is long enough to make storage administrators comfortable that the media will last. However, even though the digital media can be read many years from now, the rate of change for new storage technologies make the media obsolete in less than 10 years. Finding replacement parts for tape drives, trained maintenance personnel, diagnostics, and operating systems that support old devices now mandate conversion to a new archival technology well before its rated useful life is over.   Digital preservation requires a plan for format conversion as older storage media becomes obsolete.

Remote electronic tape libraries used as vaults and true offline tape storage remain useful for archival of records that are seldom accessed, offering additional geographic protection against disasters.

ATA disks are increasingly being used for cost effective storage of static (seldom changed) content but disk drives are generally not considered good archiving media because of the expected lifetime of the drives.  However, a new concept of MAID (Massive Arrays of Idle Disks) storage may change this for some applications and users. MAID is similar to the RAID concept except that in a MAID storage array, all disks are not spinning all the time. With a MAID subsystem, disks remain dormant (powered off) until requested.

Power-up time for ATA disks takes about 10 seconds. MAID storage is somewhat analogous to an automated tape library with the exception that disks are substituted for tape cartridges. MAID can be viewed as a library of disks.

By reducing the number of disks that are concurrently active, the overall storage subsystem costs can be significantly lowered by simplifying controller complexity. The financial savings increase as storage environments get larger. MAID provides traditional levels of RAID data protection capability, important for ATA drives, to enable higher availability similar to current disk arrays. MAID may be best suited to addresses mid-term archival and lower activity fixed-content data.  Two start-up companies are currently offering MAID products.

Optical disk capacities have been favored by some entertainment users although traditional optical recording has not kept pace with the storage capacity growth of either disk or tape.  There is an optical storage technology that could change that however, holographic storage.  Although holographic storage has been in development for many years it may finally be reaching commercial success.  InPhase Technologies, has demonstrated the first fully-functioning prototype of its Tapestry holographic drive at the 2005 Storage Visions conference held in Las Vegas. The company's holographic storage media stores data in 3-D holograms cut into a polymer material that's 1.5 mm (0.06 inch) thick and is placed between two 130 mm (5.1-inch) plastic discs. Since holographic devices are able to store data in 3D 'pages,' they should have a greater capacity than today's CDs or DVDs that store data only on the disc's surface (except for dual-layer DVDs-they have two layers of data). InPhase plans to ship 300 GB holographic write once disks by mid-2006.  By 2009, InPhase hopes to be shipping drives that can store as much as 1.6 TB on a single disc.

Optware is another company working on holographic storage products.  Optware has helped to form a Holographic Versatile Disk (HVD) alliance.  Six companies have collaborated to accelerate the development of the HVD technology, including CMC Magnetics, Fuji Photo film, Nippon Paint, Optware, Pulstec Industrial and Toagosei.  Optware is developing technology that puts 100 GB-1 TB of data, with data transmission speeds of 100 Mbps-1 Gbps, on discs that are of the same diameter as today's CDs and DVDs.

Jeff Ubois, a staff research associate at UC Berkeley’s School of Information Management and Systems, and a co–chair of the Association of Moving Image Archivists’ Television Interest Group, recently wrote a paper addressing issues in television archiving and digital video.

He cited four areas for potential cooperation to save television shows: cataloging; technical standards; legal strategies for access and rights clearance; and building a social consensus about television archiving

.He said operators of television archives need to devote more of their budgets to cataloging activities since lack of such logs is a major barrier to assessing the overall state of television archiving. Common technical standards for archival preservation are also necessary. Online access to text depends on the use of common standards.

Although legal strategies involving television a complex, all sides in the copyright debate could benefit from the creation of a system that would simplify access to and reuse of archival footage, and streamline rights clearance procedures, Ubois said. An organization that handles rights for television, something like ASCAP or BMI does for the music industry, could reduce the barriers to access and use, and perhaps even help to fund preservation efforts.

And, finally, Ubois said the most important missing ingredient needed for the creation of a comprehensive, broadly accessible system of television archives is a social consensus that 1) television broadcasts are an important part of our culture deserving of systematic preservation and widespread access, just like books, periodicals, sound recordings, and film, and 2) that such a system is technically, legally, and economically feasible.

Given the difficulty in converting analog media such as tape to digital forms how can this activity be made more efficient and less costly.  In addition are there ways to finance archiving? 

Given the tender loving care that digital conversion and preservation of aging analog content requires it is difficult to see how the costs of doing this can be reduced except by increasing the volume of activity and developing standardized methods for conversion and (equally important) generation of content metadata.  Perhaps there is even room for dedicated and skilled amateurs to play a role (can you image a Society of Amateur Archivists?).

With regards to how this conversion can be paid one suggestion that I made in 2003 is to place a contribution that was part of the purchase price of new content.  For instance if content distribution companies were to say that e.g. 50 cents from the sale of each CD or DVD that they make would put into the preservation and digital conversion of their cultural archives (but not take away from their on-going spending for archiving and preservation). This would be a great PR move, would be seen as a great boon to mankind and demonstrates enlightened content stewardship.

The individual companies could use this fund for digital preservation or perhaps they could pool the resulting funds to create an industry wide archiving activity.  For the data storage vendors preservation means digitization and this effort would result in a great boast of the sales of static-content data storage and archiving devices as well as storage media such as tape, optical disks, and ATA-based drive arrays. Digital preservation of the current store of rich media archives would amount to several Exabytes of data storage**.

Tom Coughlin
President
Coughlin Associates
www.tomcoughlin.com