Incredible Shrinking Music

If it gets any more compact it might just disappear ...

Back when the millennium rolled over from 1 to 2 I was still using a computer with a mere 10 gigabyte hard drive. But newer drives were becoming available with 60 or 80 gigabytes, and 100 was soon to follow. At this point I turned my eyes to my music collection -- a set of shelves holding a few hundred CD's -- and began to wonder ...

Wouldn't it be great if I could copy all those disks to a big hard drive someday? Then I could organize them into folders by composer and album title, even into larger categories like Classical, Jazz, Rock, and Folk, and I'd be able to find what I wanted to hear with a few mouse clicks instead of tilting my head sideways to read the tiny print on the edges of the disk boxes. Instead of loading my 5-disk CD changer I'd be able to have a list of music that would play all day, or all weekend if I wanted to.

Then I did the math. Allowing about 600 megabytes for each CD (because many of them were not the full capacity of 700 megs or 80 minutes) and multiplying by the approximate number of disks on my shelves I got the discouraging result of nearly 400 gigabytes. (I have a LOT of music.) To contain my whole collection I would need an array of at least five 100 meg hard drives. At the going rate of about $200 each, plus a rack of RAID hardware to organize them all, it would have cost over $1400, even with no room to grow.

But of course all I had to do was wait a few years. Three things happened that made my dream system not only doable but easy and cheap.

First was advances in sound compression technology. The huge WAV files on all those CD's were like the pig-sized BMP images that take up so much space because they make no attempt to optimize the way they store their contents. MP3 compression was one answer, of course, and I came to consider it good enough for at least some forms of music (e.g. rock) where the subtlest shades of tone might not be critical. But I wasn't satisfied with that for my classical collection. Having shelled out all those bucks for the best possible sound I wasn't about to run it through a meat grinder before listening to it.

Fortunately FLAC came along -- a "lossless" form of audio compression that can shrink a WAV file to about half its size and then blow it up again with every bit intact. If this format had been used on CD's you could have bought the complete Brandenburg Concertos on a single disk, or all the Beethoven Symphonies and overtures on a 4 disk set instead of 7 or 8. Flac has the further advantage of being open source and free to use. This instantly reduced the amount of storage I would need down to a more manageable 200 gigabytes.

By this time hard drives had grown to 500 gigabytes (the original size of my estimate), with 750 and 1000 -- a full terabyte -- soon to come. Not willing to wait any longer, but still cost conscious, I grabbed a 320 gig drive on sale for $89. Quite a bit better than $1400!

The last ingredient that pulled all the pieces together was software that automated the process of converting the CD's to flac files, and adding them to a searchable database complete with categories that I could edit as I chose. There are many programs to organize music collections, but my favorite is the open source Amarok. Originally for Linux only, and now also available for Mac and Windows, Amarok works well and makes it trivially easy to find, manage, and play your music.

It was a short step from there to acquiring new music in pure digital form. I do this legally through eMusic, a subscription service that lets you budget how fast you want to grow your collection by choosing from a variety of monthly plans and payment options. Depending on your selection you can get your downloads for as little as 43 cents per track, with most full albums (if that word still has any meaning) only a bit above $5.

eMusic convinced me that with a high enough sample rate MP3's were satisfactory even for classical music with its subtle nuances. This launched me into a new acquisitive phase. I started buying more music than I had done at any time since the advent of CD's caused me to replace my LP collection. (Music publishers please take note.) And of course the MP3 albums were even more compact than all those flacs I had pulled off my CD's, so I was able to pack an amazing amount of additional music onto my 320 gig drive and still have space left over.

Digital Peril

What about disaster recovery, you may well ask? With so many hours of effort invested in this project, not to mention the financial investment in new music, what would I do if the hard drive failed?

In the case of the CD's I still had the optical disks for backup. At the beginning I started burning at least the best of the new MP3's to disk for protection, but this seemed to defeat the purpose. It now began to appear as a liability to have to store all the disks. In the end I decided that once I outgrew the old hard drive I would copy all the files to a newer and bigger one, then put the old drive in a drawer as a backup copy. At least that would give me a fallback position so only the files that I added to the new drive would be lost if it failed. (Notice, by the way, that my shelves full of disks had been reduced to a package about 4x6 inches in size.)

I planned to move next to a 500 gig drive, but what happened is that my computer died instead of the hard drive. I ended up moving to a new machine with a full terabyte instead -- a system I would have drooled over ten years ago, but which is now pretty standard. I partitioned the drive into two 500 gig halves, donated one half for my wife and grandchildren to live in under Windows, and moved into my own Linux partition on the other half. After restoring all my files from the old drives, including all the music, I still had almost 200 gigs left over. Room to grow.

The old music drive is tucked into that drawer to give me some peace of mind. Next will be to set up another 1 or 2 terabyte drive in an external enclosure to serve as backup. The thing to keep in mind is that the growth of storage space and the dropping of its prices have not slowed down yet. That old 10 gig drive cost me $160 back in 1998, or around $16 per gigabyte. Now you can buy a terabyte drive for around $90, or about 9 cents per gigabyte. In five years we should be up to 8 or 10 terabyte drives with a cost per gigabyte of about a penny. After that we'll have to start watching the terabyte prices fall until we're up in the petabytes.

The same drive that holds all those music files also contains a few "virtual machines" -- files that contain a simulated computer complete with its own simulated hard drive. They can be opened in their own windows (using VirtualBox), copied, and moved to other hard drives as easily as you move a photo from one folder to another. Each of my virtual machines has more power and storage space than my whole computer back in 2000. The new computer swallows them whole and has space for them to swim around in.

Long ago I got used to the idea that everything I had ever written, if suitably compressed, would fit on a single floppy disk. Now it seems I can look forward to a time when all the files I've ever worked with -- all the images, software, music, etc -- can be saved and copied just as trivially. But by then we may no longer be storing anything ourselves ...

[Coming soon: Life in the clouds.]