LTO Tape Archiving on the Cheap

We’ve been told by Larry Jordan and others that archiving our one-of-a-kind media files on hard drives is just waiting for disaster to strike. Eventually the hard drive will fail. Most people recommend using LTO tape as an archiving medium since it’s estimated to last 30 years or more. But even with new hardware like the mLogic mTape Thunderbolt drive, you’re still talking about $4,000 for the hardware and software. I needed a cheaper solution.

In response to an article by  called “Affordable, Easy LTO Archiving” on CreativeCow I posted this reply:

I’ve been archiving original camera files to Blu-Ray, but needed some way to archive FCPX projects which are about one TB each, on something other than a pile of hard drives.

I just took the plunge into the murky world of LTO and cobbled together a system for under $1500. Since we’re talking about “affordable” I thought I’d share my journey, since I couldn’t find an off-the-shelf LTO 5 or 6 Thunderbolt solution for under $3,500.

I found a Tandberg LTO 5 tape drive on eBay for $750 (probably a one of a kind deal), bought a used ATTO PCIe SAS card from B&H for $199, installed the card into a Akitio Thunder2 Thunderbolt to PCIe box ($219 on Amazon, including a Thunderbolt cable), found a 2 meter HP mini-SAS cable on eBay for $10, and downloaded the demo version of BRU PE for the Mac.

LTO-5_HH_SAS_external_sm

Then I found a guy selling new HP LTO 5 tapes in bulk on eBay for $15 each so I bought 20 of those. Imagine my surprise when I plugged the system into my 5K iMac and it all worked!

So far I’ve copied about 16 TB of data onto tape. I found out that the 1.5 TB tapes will actually hold about 1.35 TB before BRU asks for another tape. And it’s really slow — it takes about 3 hours to record that much to one tape, and another 3 hours if you want to verify data. I don’t know if the standalone systems are faster, but I won’t have a ton of archiving to do after the first pass, so I can live with it. I’m storing the tapes off-site and will keep the hard drive collection for quick restoring until they fail.

The only other expense will be the BRU license for $499, if I decide to keep using it. BRU kindly allows demo users to restore files even after the 30 day trial. I appreciate its ease of use but wish they offered a “light” version for us freelancers who just need to record a few dozen tapes per year. I tried LTFS but so far found software that will recognize the drive but it won’t format a tape. If anyone knows of a reliable, cheap or free LTFS solution that will work with my hardware setup on a Mac I’d appreciate hearing about it.

So it’s not elegant and it took a lot of research, but I’ve got my data archived on tape at last. Good luck to all in finding your own solutions.

Mac Pro Upgrade – Mountain Lion, SSD, eSata, USB 3

My main editing computer has been an early 2009 Mac Pro 8-core Intel machine with 14gb RAM and an NVIDIA GeForce GT 120 graphics card, running OSX 10.6.8 Snow Leopard.  It’s been a workhorse but it was getting tired, taking a long time to boot up and do disk-intensive tasks.

I decided it was time to look into some upgrades to speed things up.  The first thing I did was to try using USB 3 on the Mac Pro to get some speedy external hard drives.  I found a Sonnet Allegro PCIe USB 3 card on Amazon.  It works with a 3 tb WD MyBook USB 3 drive getting up to 92mbs sequential write and 85mbs read.  But the two-slot card will only work with one device at a time and the drive will mysteriously disappear sometimes, only reappearing when I reboot.  I give the setup a 5 out of 10 because it works but is a dead-end since I can’t add a hub or another drive.

Next I tried a eSata card to get better speed from my Newer Voyager and NexStar external  drive docks which I’ve been running with Firewire 800.  I decided on a NewerTech MaxPower eSATA 6G PCIe 2.0 RAID Capable Controller Card from OWC because it supports RAIDs and they claimed it would run on the Mac Pro.  Speed increase was not dramatic, but I’m getting about 22% faster read/write from external drives.   I give it 7 out of 10.

I’m kind of late to the party with Solid State Drives (SSD) but from many accounts putting the system and main software on an SSD boot drive would be the best and speediest upgrade.  I decided to upgrade to OS 10.8 Mountain Lion first to see if it was stable, then install an SSD.

I ran Mountain Lion on a separate drive for a couple of weeks, only copying over software that I actually used during that time to see if I could eliminate a lot of the clutter in my Applications folder that I had accumulated over the years.  Plus Mountain Lion was not compatible with some of the old programs so I wanted to see what needed to be updated.  All the main programs I was using — Final Cut Pro 7 and X, Photoshop 5.1, After Effects, Compressor, ClipWrap, MPEG Streamclip, MS Office — seemed to work fine under Mountain Lion.  I had to upgrade Pluraleyes to the new Red Giant version, and I had to update drivers for Matrox Max, the Sonnet Allegro USB 3 card and a few other utilities.

All was good except that if the system went to sleep it would not wake up when you hit the keyboard or mouse, and I’d have to do a hard restart.  I didn’t lose much work because I usually save everything before getting up from the computer, but it was really annoying.  After checking a lot of posts by people with the same problem, I stumbled on a solution posted by Bonfire to upgrade the Nvidia graphics card drivers.  After downloading and installing the QUADRO & GEFORCE MAC OS X DRIVER RELEASE 304.00.05, everything is working fine.

Next the SSD.  I  picked up a Sandisk Extreme 480gb SSD on eBay for about $300.  Mounted it in an Icydock 2.5 to 3.5 converter box.  Loaded Mountain Lion and essential applications and swapped out my old boot drive in bay 1.  Startup time went from almost 2 minutes down to 45 seconds.  It’s reading/writing about twice as fast as my old drive according to Xbench.  9 out of 10 with just a knock for the expense.  I hope SSDs will start to drop in price like SD cards have.

So I’m happy with the upgrades.  For about $400 in parts it feels like a new machine, which will suffice until or if Apple decides to update their Mac Pro line.

 

Hacked and Fixed

This blog was hacked last month.  Google said it was blocked because of malware.  I never heard of this before, but I downloaded the files and checked them with ClamXav, a free virus checker, and found that 673 out of 1670 files were infected with something called Trojan.PHP-33.  The support desk at Bluehost where my website is hosted suggested using a malware scanning service to help clean out the bad code.  I decided to contact wewatchyourwebsite.com since it got some good reviews and was considerably cheaper than the alternatives.

Thomas J. Raef at We Watch Your Website had the site cleaned up in one day and now we’re back up and running.  The service monitors the site and will clean it as many times as necessary.  I also installed the anti-malware plug-in Anti-Malware by ELI (Get Off Malicious Scripts) on WordPress to prevent re-infection.  Finally, I contacted Google to unblock the site and it was cleared in a few hours.  All in all it was a pain to deal with.  I’m not sure why anyone would want to hack this little blog, but I hear that it’s pretty common.  I’m glad I was able to get rid of it.   If you are using WordPress I’d suggest at least installing the Anti-Malware plug-in, and subscribe to We Watch Your Website if you want to be proactive.

Archiving Video Projects to Blu-Ray

I’ve had a few hard drives fail in the past but luckily didn’t lose anything too valuable.  But I get paranoid that something I put a lot of time into will be lost forever with a drive crash. I routinely back up original media to a separate drive, and the Final Cut Autosave Vault has saved me many times, but when I’m done with a project I needed a way to archive everything.  A long-form video project can eat up from 300 to 600gb so I needed something that can handle that amount of data, be relatively stable and cheaper than just buying more 3tb hard drives.

A lot of pros recommend archiving to LTO tape systems, but those drives are over $1000 and the tapes aren’t cheap either.  So I started archiving video projects to Blu-Ray discs as a cheaper alternative.  They hold up to 25gb per disk (50gb on double layer disks) and are rated to last up to 50 years — a lot longer than hard drives or tape.

I picked up an internal LG 12X Blu-Ray burner ($87 from 3btech.net) and installed it in the second bay of my Mac Pro in about 10 minutes.  After researching different brands of media it looks like Sony Blu-Ray discs are consistently rated highly, and I found 50-disc packs for about $75.  I used Toast Titanium 10 to gather all the files from various on-line drives, and set them up for burning.  Toast will automatically span as many discs as needed for a large archive.

To keep things to a reasonable size I archive the original camera and audio files and everything from the project except render files and transcoded media since those can be recreated from the originals if needed.  A recent long-form project with about 3.5 hours of final video took 14 Blu-Ray discs to archive, costing about $21 for materials.   Each disc takes about 15 minutes to burn and verify, but you can work on other things while waiting.

A boring task, but I figure it’s good for clients to know you’ve got their material archived, and it’s good for your own peace of mind.

 

Faster YouTube Upload Success – Fiber Optic Internet

I have a potential client who needs one-hour HD videos uploaded to YouTube within a very short time window.  After compressing a test video with the Matrox CompressHD card to 720p h.264 with Compressor (45 minutes) the file size was 2.32gb.  Uploading through my Comcast residential connection at about 4Mbps took over 4 hours.

I tried a Comcast business connection which Speedtest.net said was uploading at 9.1 Mbps, but that still took almost 3 hours.  I tried at different times of the day but although Speedtest said some upload speeds were faster than others it didn’t make a lot of difference in the time it took to upload.

After an extensive search and posting on several forums for help I finally found a local office building which is home to many non-profit organizations, and which has a fiber optic connection to the internet.  Plus they have free public wi-fi.  The upload speed is still only 9.9 Mbps, but the YouTube upload of the same file took only 38 minutes!

I don’t know why there is such a huge difference in speed.  Is Comcast somehow choking the upload process, or does a fiber optic connection make it faster even at the same upload speed?  More research is needed, but at least now I know I can meet the client’s need for speed.

Compressor: Looking for Speed — Update: Matrox CompressHD

In the continuing quest for speed using Compressor, after testing to find the fastest workflow for me I started looking for hardware acceleration.

It looks like the main player in this is Matrox with their Max Technology as an option on their hardware In/Out boxes and cards.  The cheapest implementation is the Matrox CompressHD card ($499 retail) which only gives you H.264 acceleration, no I/O.  After finding several bad reviews of the video capabilities of the next cheapest product, the Matrox Mini02, I decided to just try the CompressHD card.

I found one on eBay for about $100 less than retail.  Being used, I was taking a chance, but the guy said it tested perfectly.  The card arrived today in a plain brown box with lots of bubble wrap but nothing else.

I installed it in an empty PCI-e slot in the Mac Pro and fired it up.  Then I went to the Matrox site to download the software.

(Note to self:  Write down the serial number of a used card before installing it!  Matrox required the serial number to download the software & I had to open the Mac back up and pull the card to find it.)

There were about a dozen choices of software to download, based on the Mac OS you’re using and whether you’re using Compressor or Adobe Media Encoder.  None of them matched OS 10.6.8 that I’m using, but I picked the closest one before it.  Software installation was straightforward but caused some anxiety when it automatically updated firmware on the card, since I didn’t even know if the card was working yet.  Luckily things went smoothly and after a restart I was ready to go.

The software installs the Matrox Max H.264 settings into the Custom folder of Compressor’s settings.  I loaded up the 90 second ProRes LT test file from my previous tests, grabbed the High Quality 720p/29.97 Matrox setting and started the compression.

Previous fastest encode was 03:07.  Encoding with the Matrox Max CompressHD card:

01:22 — More than twice as fast with the card, and 8 seconds faster than real time!

I also tried Fast Encode, which finished in 50 seconds but the quality suffered a bit.  I could see some stairstepping in the titles and the image appeared softer.  So I’ll probably stick to High Quality unless it’s just for a quick approval clip.  Good to know it’s there if I need it.

Matrox Max High Quality Encode - 720p 100% Crop

 

Matrox Max Fast Encode - 720p 100% Crop

I’ll try a longer file soon, but this is very promising.

Tascam DR-40 Sync Drift / FCPX Audio Fix Missing

I was editing an hour+ long performance video in FCPX using audio from both an audience mic into a camera, and audio from the front-of-house board recorded on a Tascam DR-40.  I synced everything with PluralEyes, but I noticed an echo toward the end.   Zooming in to the waveforms I could see that the two audio tracks were in sync at the beginning, but slightly out of sync after an hour.

I couldn’t find a way to change the speed of the Tascam track in FCPX.  In FCP7 there’s a Change Speed feature where you can input the percentage or duration change down to 3 decimal points.  I searched the forums and blogs and it looks like FCPX can’t do it.

I went back to FCP7 and re-synced everything there.  Zooming in tight to the waveforms and using Modify: Change Speed (command + J) I adjusted the speed of the track with trial & error until it matched up.  Speed change was 100.008 percent.

Out of curiosity I ran a test with a Sony HDR-AX2000 camera, a Zoom H4n and the Tascam DR-40.  I found that the Tascam track was 9 frames slower than the audio from my camera after one hour.

Sync Drift After One Hour: Tascam +9 Frames, Zoom +2 Frames

Interestingly, the Zoom H4n was 2 frames off too, so eventually it would also need to be adjusted.  But at one hour the sync difference was not noticeable.

I went to Change Speed and entered 100.008 %, re-synced at the beginning and got the Tascam track to within 1/2 frame.

Sync After Speed Change

Interestingly, I couldn’t change the speed of the Zoom track less than 100.005%, so it couldn’t be improved.

So for sequences of less than 15 minutes or so the Tascam should be ok, but for long form shoots (which I do a lot of) it’s kind of a pain.  As much as I like the DR-40 (for the line/mic switch, the locking XLRs, and light weight) I think I’ll have to get rid of it and stick with the Zoom H4n.

Tascam DR-40/A-Data SDHC FAIL

I bought a Tascam DR-40 audio recorder to try out after reading some good reviews.  It is the cheapest recorder ($165 on eBay) I could find with XLR inputs that would record to 48kHz/16bit to match my cameras.

There’s a lot I like about the DR-40 like the mic/line switch for external inputs, running off of USB bus power to save batteries, ability to record at two levels simultaneously to avoid distortion, etc.

I used it on a long performance shoot last week to take audio from the board.  I was recording to an A-Data 16gb SDHC card which had been reliable up until now.  During the shoot I had it set to automatically start a new file after recording 1gb.

After about the first hour I looked at the Tascam and it had stopped recording.  There was an error message saying “Write Error – Press Enter.”  I pressed enter and kept getting the error.  Finally I powered down and restarted the recorder but THE FILE WAS NOT SAVED!  When I got back to the studio and checked the card there was a file name but 0 kb in the file.  I tried a couple of file recovery programs but there was nothing there.

Luckily I had backup audio from one of the cameras, but lost a lot of confidence in the Tascam and A-Data.  I’m not sure which was the problem, and haven’t been able to duplicate the issue.  I think I’ll stick to the Zoom H4n after this.

Next:  Tascam DR-40 Sync Drift

 

Why I can’t use the Canon T3i

I picked up a Canon T3i (D600) last year in a bundle deal that dropped the cost of the body to under $500 after I sold off the rest of the bundle.  I was hoping to use it as a backup to my 5DmkII and was excited about the added capabilities like the 1.6 crop and digital crop — giving me a lot more range with my lenses. The T3i has been well-reviewed by many and is a very capable and affordable DSLR for general use.

Unfortunately I shoot mostly in low-light situations and the T3i doesn’t hold up.  Shooting video at ISO 1600 (1/30 sec, f2.8, L lenses) the blacks are noisy and ugly and it just doesn’t intercut with video from my other equipment.  Too bad.  I’ve seen some very good low-light footage shot with this camera but for my purposes it just doesn’t cut it.

I sold it and picked up a Canon HF-G10 prosumer camcorder instead.  It’s pretty great for a sub-$1200 video camera.  More on that later.

Wasabi Power Batteries

I was looking for camera batteries to replace the aging OEM Canon and Sony ones I have and came across these Wasabi Power ones on Amazon.  For half the price of a new Canon LP-E6 you get two of these plus a charger.  I’ve purchased some cheapo aftermarket batteries on eBay before but the 5DmkII doesn’t recognize them so you can’t see how much power is left.  Worse, Magic Lantern won’t work with the cheapos because when you try to change the firmware the camera hangs up.

The Wasabi’s show up fine in the 5DmkII, either alone or in the OEM battery grip (not in a cheapo after market grip though).  They charge fast and hold up well.  Sure hope they will work in the 5DMk3.

I just ordered one to replace a Sony NP-F970.  More on that later.

Recommended.