LTO Tape Archiving on the Cheap

We’ve been told by Larry Jordan and others that archiving our one-of-a-kind media files on hard drives is just waiting for disaster to strike. Eventually the hard drive will fail. Most people recommend using LTO tape as an archiving medium since it’s estimated to last 30 years or more. But even with new hardware like the mLogic mTape Thunderbolt drive, you’re still talking about $4,000 for the hardware and software. I needed a cheaper solution.

In response to an article by  called “Affordable, Easy LTO Archiving” on CreativeCow I posted this reply:

I’ve been archiving original camera files to Blu-Ray, but needed some way to archive FCPX projects which are about one TB each, on something other than a pile of hard drives.

I just took the plunge into the murky world of LTO and cobbled together a system for under $1500. Since we’re talking about “affordable” I thought I’d share my journey, since I couldn’t find an off-the-shelf LTO 5 or 6 Thunderbolt solution for under $3,500.

I found a Tandberg LTO 5 tape drive on eBay for $750 (probably a one of a kind deal), bought a used ATTO PCIe SAS card from B&H for $199, installed the card into a Akitio Thunder2 Thunderbolt to PCIe box ($219 on Amazon, including a Thunderbolt cable), found a 2 meter HP mini-SAS cable on eBay for $10, and downloaded the demo version of BRU PE for the Mac.

LTO-5_HH_SAS_external_sm

Then I found a guy selling new HP LTO 5 tapes in bulk on eBay for $15 each so I bought 20 of those. Imagine my surprise when I plugged the system into my 5K iMac and it all worked!

So far I’ve copied about 16 TB of data onto tape. I found out that the 1.5 TB tapes will actually hold about 1.35 TB before BRU asks for another tape. And it’s really slow — it takes about 3 hours to record that much to one tape, and another 3 hours if you want to verify data. I don’t know if the standalone systems are faster, but I won’t have a ton of archiving to do after the first pass, so I can live with it. I’m storing the tapes off-site and will keep the hard drive collection for quick restoring until they fail.

The only other expense will be the BRU license for $499, if I decide to keep using it. BRU kindly allows demo users to restore files even after the 30 day trial. I appreciate its ease of use but wish they offered a “light” version for us freelancers who just need to record a few dozen tapes per year. I tried LTFS but so far found software that will recognize the drive but it won’t format a tape. If anyone knows of a reliable, cheap or free LTFS solution that will work with my hardware setup on a Mac I’d appreciate hearing about it.

So it’s not elegant and it took a lot of research, but I’ve got my data archived on tape at last. Good luck to all in finding your own solutions.

Archiving Video Projects to Blu-Ray

I’ve had a few hard drives fail in the past but luckily didn’t lose anything too valuable.  But I get paranoid that something I put a lot of time into will be lost forever with a drive crash. I routinely back up original media to a separate drive, and the Final Cut Autosave Vault has saved me many times, but when I’m done with a project I needed a way to archive everything.  A long-form video project can eat up from 300 to 600gb so I needed something that can handle that amount of data, be relatively stable and cheaper than just buying more 3tb hard drives.

A lot of pros recommend archiving to LTO tape systems, but those drives are over $1000 and the tapes aren’t cheap either.  So I started archiving video projects to Blu-Ray discs as a cheaper alternative.  They hold up to 25gb per disk (50gb on double layer disks) and are rated to last up to 50 years — a lot longer than hard drives or tape.

I picked up an internal LG 12X Blu-Ray burner ($87 from 3btech.net) and installed it in the second bay of my Mac Pro in about 10 minutes.  After researching different brands of media it looks like Sony Blu-Ray discs are consistently rated highly, and I found 50-disc packs for about $75.  I used Toast Titanium 10 to gather all the files from various on-line drives, and set them up for burning.  Toast will automatically span as many discs as needed for a large archive.

To keep things to a reasonable size I archive the original camera and audio files and everything from the project except render files and transcoded media since those can be recreated from the originals if needed.  A recent long-form project with about 3.5 hours of final video took 14 Blu-Ray discs to archive, costing about $21 for materials.   Each disc takes about 15 minutes to burn and verify, but you can work on other things while waiting.

A boring task, but I figure it’s good for clients to know you’ve got their material archived, and it’s good for your own peace of mind.

 

Faster YouTube Upload Success – Fiber Optic Internet

I have a potential client who needs one-hour HD videos uploaded to YouTube within a very short time window.  After compressing a test video with the Matrox CompressHD card to 720p h.264 with Compressor (45 minutes) the file size was 2.32gb.  Uploading through my Comcast residential connection at about 4Mbps took over 4 hours.

I tried a Comcast business connection which Speedtest.net said was uploading at 9.1 Mbps, but that still took almost 3 hours.  I tried at different times of the day but although Speedtest said some upload speeds were faster than others it didn’t make a lot of difference in the time it took to upload.

After an extensive search and posting on several forums for help I finally found a local office building which is home to many non-profit organizations, and which has a fiber optic connection to the internet.  Plus they have free public wi-fi.  The upload speed is still only 9.9 Mbps, but the YouTube upload of the same file took only 38 minutes!

I don’t know why there is such a huge difference in speed.  Is Comcast somehow choking the upload process, or does a fiber optic connection make it faster even at the same upload speed?  More research is needed, but at least now I know I can meet the client’s need for speed.

Project Management for Video Files: Post Haste+DiskTracker+Voyager

I’ve been using Post Haste since I read a tip from EditingWhiz about keeping your video files organized so you can find what you’re looking for.  It’s a free program from Digital Rebellion that has templates which automatically set up folders for different media from FCP, After Effects, Photoshop, etc., and creates a dummy file in each folder with a consistent naming convention based on customer name and date for that project. It’s customizable for your own situation. I’m on a Mac but I think they have a Windows version too.  Coupled with a disk cataloging system like DiskTracker, you can find files easily across multiple internal and external drives.

I’m using a Newer Technologies Voyager quad interface external drive dock and about a dozen (and growing) 1 to 3tb drives that I use to archive projects.  The Voyager has Firewire 800 & eSATA which speeds up write times a lot over USB.

Compressor: Looking for Speed — Update: Matrox CompressHD

In the continuing quest for speed using Compressor, after testing to find the fastest workflow for me I started looking for hardware acceleration.

It looks like the main player in this is Matrox with their Max Technology as an option on their hardware In/Out boxes and cards.  The cheapest implementation is the Matrox CompressHD card ($499 retail) which only gives you H.264 acceleration, no I/O.  After finding several bad reviews of the video capabilities of the next cheapest product, the Matrox Mini02, I decided to just try the CompressHD card.

I found one on eBay for about $100 less than retail.  Being used, I was taking a chance, but the guy said it tested perfectly.  The card arrived today in a plain brown box with lots of bubble wrap but nothing else.

I installed it in an empty PCI-e slot in the Mac Pro and fired it up.  Then I went to the Matrox site to download the software.

(Note to self:  Write down the serial number of a used card before installing it!  Matrox required the serial number to download the software & I had to open the Mac back up and pull the card to find it.)

There were about a dozen choices of software to download, based on the Mac OS you’re using and whether you’re using Compressor or Adobe Media Encoder.  None of them matched OS 10.6.8 that I’m using, but I picked the closest one before it.  Software installation was straightforward but caused some anxiety when it automatically updated firmware on the card, since I didn’t even know if the card was working yet.  Luckily things went smoothly and after a restart I was ready to go.

The software installs the Matrox Max H.264 settings into the Custom folder of Compressor’s settings.  I loaded up the 90 second ProRes LT test file from my previous tests, grabbed the High Quality 720p/29.97 Matrox setting and started the compression.

Previous fastest encode was 03:07.  Encoding with the Matrox Max CompressHD card:

01:22 — More than twice as fast with the card, and 8 seconds faster than real time!

I also tried Fast Encode, which finished in 50 seconds but the quality suffered a bit.  I could see some stairstepping in the titles and the image appeared softer.  So I’ll probably stick to High Quality unless it’s just for a quick approval clip.  Good to know it’s there if I need it.

Matrox Max High Quality Encode - 720p 100% Crop

 

Matrox Max Fast Encode - 720p 100% Crop

I’ll try a longer file soon, but this is very promising.

Tascam DR-40 Sync Drift / FCPX Audio Fix Missing

I was editing an hour+ long performance video in FCPX using audio from both an audience mic into a camera, and audio from the front-of-house board recorded on a Tascam DR-40.  I synced everything with PluralEyes, but I noticed an echo toward the end.   Zooming in to the waveforms I could see that the two audio tracks were in sync at the beginning, but slightly out of sync after an hour.

I couldn’t find a way to change the speed of the Tascam track in FCPX.  In FCP7 there’s a Change Speed feature where you can input the percentage or duration change down to 3 decimal points.  I searched the forums and blogs and it looks like FCPX can’t do it.

I went back to FCP7 and re-synced everything there.  Zooming in tight to the waveforms and using Modify: Change Speed (command + J) I adjusted the speed of the track with trial & error until it matched up.  Speed change was 100.008 percent.

Out of curiosity I ran a test with a Sony HDR-AX2000 camera, a Zoom H4n and the Tascam DR-40.  I found that the Tascam track was 9 frames slower than the audio from my camera after one hour.

Sync Drift After One Hour: Tascam +9 Frames, Zoom +2 Frames

Interestingly, the Zoom H4n was 2 frames off too, so eventually it would also need to be adjusted.  But at one hour the sync difference was not noticeable.

I went to Change Speed and entered 100.008 %, re-synced at the beginning and got the Tascam track to within 1/2 frame.

Sync After Speed Change

Interestingly, I couldn’t change the speed of the Zoom track less than 100.005%, so it couldn’t be improved.

So for sequences of less than 15 minutes or so the Tascam should be ok, but for long form shoots (which I do a lot of) it’s kind of a pain.  As much as I like the DR-40 (for the line/mic switch, the locking XLRs, and light weight) I think I’ll have to get rid of it and stick with the Zoom H4n.

Compressor: Looking for Speed

Apple Compressor has always been a mystery to me.  Sometimes it takes forever to transcode, but when I’m in the middle of a job I don’t have time to figure out why.  I just plan my workday to transcode overnight and hope nothing crashes at 4am.

So I took some time today and did some tests with a 90 second clip from FCP7.  Send to Compressor 3 and transcode to H.264/720p took 09:14 with a lot of other open programs on my 8 Core Mac Pro — 6.15 times the length of the clip.

After restarting and having only FCP7 and Compressor 3 on, same file took 08:54

Saving the Compressor 3 settings and reopening with Compressor 4 with Qmaster on (7 instances) yielded 09:01 — Seven seconds worse than Compressor 3 without Qmaster (I’ve never been able to get Qmaster to work with Compressor 3 — another story).

Next I went back to FCP7 and Exported a QuickTime file with current settings, saved as a self-contained movie.  It took 00:33 to export, or about 1/3 of the length of the clip.

Transcoded the file with Compressor 4 with Qmaster on.  03:15

Transcoded with Compressor 3 without Qmaster: 03:07

So the best workflow for me right now is:

1.  FCP 7: Export QuickTime file with current settings.  Turn off FCP and any other open programs

2.  Transcode file with Compressor 3

This yields a total transcode time of about 2.5X length of the clip, so a one hour video should take 2.5 hours to transcode.

Update:  A one hour clip took 02:40:30 to create the QT file and transcode.   A lot better than before.  But I’m going to look into hardware acceleration.

Compressor: Reset Background Processing

I’ve been working on a large video project and have been frustrated by extremely long Compressor sessions.  Taking a one-hour video from FCP 7 edited in ProRes LT to H.264 720p was taking 15 hours or more in Compressor 3.53.  I tried a trick (from CreativeCow Forums) to take the file to Compressor 4 by saving the Comp 3 file and reopening it in 4, but it was worse — the time remaining to finish kept growing until I finally canceled it when it grew to 30 hours.

Tried another suggestion from CreativeCow to Reset Background Processing and that helped a lot.  Transcode times dropped to 8 hours for a one hour video.