Digital Image Storage

This article first appeared in the BT Journal from Wildlife Research Photography in 2001.--David

It was with some trepidation that I agreed to write an article on digital image storage for the Journal.  After 20 years of building a reputation primarily in the computer industry the last thing I wanted was to be pigeonholed as a pixel-pusher in the photography world.  But Moose, as usual, did have a point.  When I moved from film to digital gone was the slight uneasiness I'd always felt with darkroom chemicals and the mysteries of paper contrasts.  It was fantastic to be able to use my tool of choice-the computer-to edit, file and print my images. However, so many things we took for granted with film now require re-thinking and some work with digital.  One of them is image storage.  With film-slides in particular-our work was limited to in-camera duplicates and what type of 'archival' hanging file organization we wanted to use.  The rest took care of itself.  We confidently believe that in 20 or 50 years our chromes will still be there and still be usable.  Of course we'll never really know if they have faded since we can't go back in time to look at the original and our eyes are lucky if they can remember accurate colors for a few minutes let alone a few years.  And if we use those slides in projectors for shows that eats into their lifetime as well.  Or in the worst case if our 'image of a lifetime' was scratched while being drum scanned or reviewed by an editor-or not returned at all-we were just plain out of luck.

Image 1: taken with Nikon D1. Downsized to 300x450 Copyright David J. Cardinal

In digital we've replaced all that with the magic of infinite copies and seemingly infinite options.  We'll start by talking about the file formats available to you, then we'll talk about archiving options, and finally discuss a system that will help you prudently protect what will quickly become your largest asset as a digital photographer-your digital image library.

File Formats

JPEG is the most loved and hated file format on the planet.  Everyone loves the amazing compression they gain by using it.  Camera manufacturers love the fact that the resulting smaller images can be quickly written to digital film and later transferred to your computer easily.  But as we all know, JPEG brings with it that terrifying word "lossy".  That means that an image compressed using JPEG will never be identical to the original image.  And each time we recompress it it gets a little worse.  But boy does it work well for compressing images!  So how do we live with it?

First, let's look at how much 'harm' JPEG compression really does.  Image 1 is an image of a Bald Eagle taken with the Raw mode of the Nikon D1.  Image 2 shows a very small area of that image enlarged to reveal the image detail.  Image 3 is an enlarged area of the image compressed using JPEG the same way the camera does in Fine mode (about the same as High in Photoshop).  This level of enlargement is the equivalent of using a loupe on an 8x10 enlargement.  Unless you're planning to scrutinize your images this way you can see that you're fine using the high-quality JPEG compression in your camera.  If we repeat the process with the next level of compression, BASIC, (similar to Medium in Photoshop) we get Image 4.  Now you can start to see some real problems or 'artifacts' of the compression.  Except in some very rare cases where you are only capturing images where you'll only ever need small versions for use on the web I'd recommend staying away from all but the highest quality JPEG compressions settings on your camera-D1 or otherwise.  After all, once you've captured the image you can't go back and add more detail, but you can always compress it more later.

Second, let's discuss the issue of re-compressing JPEGs.  You don't hurt them by reading them from disk.  However, if you use an image editing program that re-writes the file, or you deliberately save the file, you are normally creating additional problems for yourself with each 'generation'.  Image 5 shows our FINE mode image saved 4 times and Image 6 shows it saved 8 times-each time with no editing, just re-saving the file.  You can see that while the effects are subtle this isn't something you want to do to the digital originals of your valuable images!  So I'd recommend keeping your original JPEGs in all cases.  If you need working files for a particular project, keep them in a separate folder.  If you have images on which you are doing active editing on, you're better off keeping those copies in a 'lossless' format like TIFF or Photoshop.  They'll be much larger but your edits won't hurt the image.  When you're finished with editing you can either flatten the file to save some space or perhaps re-compress it once into a high-quality JPEG for 'archival' storage.  I always use the "High-quality" setting in Photoshop when saving JPEGs.  Anything lower and you are damaging your image more than you need to.  You can use "Maximum", but the increase in image quality is almost invisible and the space required is much greater.

Third, there is one very common and important editing function you may not even know you perform.  That's rotation.  To make all those nice verticals easy to view you need to rotate them in your image editing program.  Most editing programs-including FotoStation-need to read and re-save your image to rotate it.  This counts as adding a generation to the file and will cause some slight artifacts.  If you are using a program that does rotation this way I'd recommend only rotating the preview or thumbnail images and leaving the original JPEG alone on your disk.  There are software programs which use some imaging tricks to implement truly lossless rotation.  If you have one of those you can gleefully rotate your original images!  You should also make sure that whatever program you use to add caption information to an image isn't rewriting the image each time you add caption or other text to the image.  Fotostation seems fine on this one, but if you are using a new program with your valuable images you might want to verify this with the software developers or by experimentation.

Finally, any discussion of JPEG needs to clear up some of the concerns about "bit-rot".  Various folks are fond of reporting stories of JPEG files mysteriously going bad.  Frankly a bit is a bit is a bit.  JPEG files are no more likely to go bad than any other image file.  Your storage devices have some redundancy built into them so a single-bit error on your disk will normally be corrected invisibly to you.  Multiple bit errors can start to create problems for any file format.  That's one reason why later on we'll talk about re-recording your data periodically.  However, there is some reason to be more suspicious of JPEG than of an uncompressed format.  Because of the highly compressed nature of JPEG files a small error can render the file unreadable to most imaging programs.  That isn't to say that all of the image data disappears, but the algorithms used to de-compress the file are often programmed to give up when they detect an error.  So you may lose much of your image even though a small part is damaged.  The secret is not to let your files get damaged, which we'll talk about below.

Shooting Data

There is another important piece to this workflow that is one more reason not to mess with your original camera files.  That's your shooting data.  Almost all digital cameras record a wealth of shooting data but no two of them do it the same way.  Except for the manufacturers' own software it is unlikely that any other package truly 'understands' 100% of the shooting data.  As a result if you use any other tool to convert or re-save your images you are almost certainly going to lose some of the information.  That may or may not be important to you, but it is something to think about.  A full discussion would take an entire article, but I'll use the shooting data recorded by the D1 as an example.  The D1 records some standard information in what is called EXIF format.  This uses the industry standard TIFF comment structure to record a set of standardized information-including f-Stop, focal length, time, date, and shutter speed-with the image.  (This standard allows for GPS information as well and is hopefully where the newly announced GPS-interface for the new D1X and D1H will write their information!)  Some image editing programs support the EXIF standard and will maintain this information when they edit or annotate your image but many (including versions of Photoshop pre-6.0) will not.  However, there is additional shooting data-including the white balance, tone, and AF settings-that is recorded in a way completely unique to the D1 or whichever specific camera you are using-in fact Nikon itself records this information in 3 different ways among its Coolpix and D1 product lines.  Unless an imaging program is written for your particular camera this is the type of information that any conversion of the file will almost certainly destroy by not preserving it during the conversion.

Other file Formats?

Your camera or scanner will almost certainly support other image formats, including TIFF and possibly one or more "Raw" formats.  TIFF is an excellent choice if you have the space to save the images.  It is a common format that has plenty of industry support and can often be sent directly to service bureaus for printing.  TIFF files, however, are almost always too large to send across the Internet unless both parties have a very high-speed connection and a robust mail system!  TIFF files support annotations and layers, although not in the same way as JPEG files, so you do risk losing your annotations and the camera shooting data when you convert from JPEG to TIFF or the reverse.

Raw files represent the ultimate in flexibility.  By capturing and recording the image almost as it leaves the CCD raw files provide you with the most options for later processing.  Specialized software for the Nikon D1 and Canon D30, for example, can actually change the white balance used in the image capture by processing the Raw file.  However, in addition to being larger and slower, Raw files have one sizable disadvantage for archival storage.  They are entirely proprietary to the specific manufacturer and camera used.  So unless you are very careful to keep a fully functioning version of the specialized software you use to read the Raw files around for the life of your library you will lose the ability to read those files at a future date.  This may be harder than it sounds at first.  Think back to the first PC software you got in the 80s.  Now imagine needing to re-construct its data files.  You'd probably need to scrounge up a compatible version of DOS, a 5-1/4" floppy, and hope that your backups of the software didn't go bad.  That's what it'll be like in 20 years when you try to go back and get that Raw file you shot yesterday!

Future File Formats

There are some emerging new file formats, such as JPEG2000, that can provide intermediate levels of compression with less image deterioration than JPEG.  These are just emerging as standards so they aren't practical alternatives today.  As they are supported by camera manufacturers and software developers one or more of them may become an attractive alternative to JPEG.  But don't count on that happening for at least a year.

Archival Storage

Okay, so now that you've sorted out what format you want to use for your long term image storage, how do you actually put a system in place that makes sure your images will always be there.  This is separate from the magic of image filing-about which Moose has written plenty.  This is about making sure that you still have readable files.

The first challenge is what medium to use to record the images.  The first truth is that there is no perfect answer-only a series of tradeoffs.  We'll talk about a few of the most common options:

CD-RW and CD-R

The most discussed and possibly most common image backup system is to some form of CD.  CD writers are common and inexpensive and CD-R disks can keep over 600MB at under $1 per disk.  Kodak and others often tout their media as having truly astonishing archival lives of up to 100 years.  Sounds perfect!  Unfortunately there are a couple problems.  First, 600MB doesn't go as far as it used to.  Saving 100,000 Fine mode D1 images requires nearly 200 CDs!  What a pain.  And you need to make sure that your image indexing system will help you get to the right CD.  Second, the archival life of the media is only part of your problem.  Since you're probably writing the CD on a consumer quality writer that has been shipped around the country and doesn't get calibrated by a lab tech-and no doubt later reading it back on something similar-you're not going to get anywhere near the practical lifetime that the raw media is capable of.  Your mileage may vary, but for now using an excellent blank disk like the Kodak Gold disks and assuming that you'll get around 5 years safely is probably a pretty good rule of thumb.  Re-writing the disks is also another chance for errors to creep in.  For the safest archival storage you're best off writing the entire CD at once, verifying it, and then storing it properly.

DVD-RAM and DVD

Okay, so what's like CD except bigger?  DVD.  Until this month the only version of DVD you could author at home was DVD-RAM.  These aren't true DVDs, since they can't be played in a regular DVD player.  But they do hold 5.2GB per cartridge and are re-writable.  However the cartridges are expensive and the software drivers for DVD-RAM can be a pain to get configured.

Fortunately for all of us, 2001 will be the year that true DVD authoring comes to your desktop.  Apple is starting to bundle writable DVD with some configurations and Pioneer has announced they'll have a DVD writer for under $1000 by mid-year.  It's too early to say whether these products work well, since they are just becoming available, but their 5+GB per disk capacity makes them a no-brainer as the next generation solution to the proliferation of CDs!  Now our 100,000 image library can fit on 20 DVDs.  Even if you assume you'll want to re-record them every 5 years that is starting to sound like a manageable process (especially since in 5 years you'll no doubt be re-recording your images onto a much larger capacity medium!).  Sony has also introduced a next generation CD with 1.3GByte capacity-double that of a standard CD-this will be a nice upgrade to the current CD technology but the relatively small increase in capacity won't help in a major way.

Tape Cartridges

Tape cartridges are a very effective alternative for disaster-recovery storage.  Because they are not direct access media it can take minutes to find a particular image-so they are not very useful as a working library-but their capacity of 20GB and up makes them ideal as a backup system.  Remember that since JPEG images are already compressed you'll only be getting the lower or "un-compressed" capacity of your backup tapes-as the backup system won't be able to compress your images any further.  That's 20GB on a DDS-4 tape, not the 40GB the manufacturer might claim!  However you manage your image library I highly recommend getting a state of the art backup product-I use software from Veritas-and keeping a revolving set of backups that includes permanently stored full backups every few months.

Off-site Storage

We don't like to think about them or plan for them, but catastrophes can happen.  Having all your backups right in your Studio is not a great idea.  Keeping a duplicate copy of your image library in a safe deposit box at your bank-or somewhere else safe and offsite-is an excellent precaution.  One easy way to do this is every quarter when you make your permanent full backup tapes take your last set down to the bank.  Tapes can go bad so you're best off to keep a few sets there in case one of them develops a problem.

Internet Backup

There are many Internet backup services available.  These used to be offered only by startup .coms-some of whom are now out of business, but now large players like Oracle are getting into the act.  Frankly they make me a little nervous and I'm not inclined to use them as a substitute for my own archival storage.  However, if you have a very fast Internet connection they may be something you want to investigate further.

An Archival System

As you can see from the description of your options there is no 'fail-safe' medium that allows you to just 'file and forget'.  If you are truly concerned with the permanence of your image data you need to actively manage your library.  Commercial institutions re-write their important data every couple years.  They literally pull the tapes off the shelf and write the contents to fresh tapes.  This is important for three reasons.  First it protects against the deterioration of the tape over time.  Second, it ensures that they have software that can read and write the tape formats-as they may write out to a newer tape format and thus keep their library 'up to date'.  Third, if there is a problem with the data this system allows them to figure it out relatively early, while there might still be other places they can recover the data.

Coupled with this, all commercial libraries make sure that all data is recorded on at least two different physical media and preferably in two different physical locations.  Since you are not the Pentagon or a Bank you may not need quite the same level of precision as they have for maintaining an active library.  But your digital images will quickly become your most valuable photographic asset so you should have an active management program-whether you re-read and re-write every 2 years or 5 years is up to you-and an offsite storage program.

The Punch Line

So, how to make sense out of all this?  A few simple rules of thumb will help.  First, use the best quality JPEG you can to capture your images-unless you have a specific reason not to.  Second, keep the original file so that you've always got it to go back to.  Third, don't re-save your JPEG files.  Use a format like Photoshop or TIFF for your working images.  Fourth, keep some form of backup, two if possible-preferably in two different places.  If you don't have the space to keep your Photoshop files, save them out as a High quality JPEG files-but don't over-write your originals.  Have fun going digital!

--David Cardinal