1. Reminder: Please use our affiliate links for holiday shopping!

Favorite backup strategies and products

Discussion in 'The Watering Hole' started by mattia, Feb 24, 2014.

  1. mattia

    mattia Mu-43 Hall of Famer

    May 3, 2012
    The Netherlands
    Inspired (or perhaps 'scared out of complacency' would be a more accurate description) by Betty's recent post on hard disk failure, I thought it might be a good idea to start a specific discussion on the backup strategies everyone uses, pros and cons, and the associated costs. The fact I've had two burglaries in the past few years doesn't help my peace of mind with everything backed up *in my house* either. Right now, I'm still looking around for an 'optimal' backup solution; I haven't really settled on the details yet, so I would appreciate comments, experiences, etcetera.

    The background: I've got just under 2 terabytes of photographs and videos (mostly RAW) shot over the past 11 years stored at home. There's a decent amount of redundancy:
    - Primary drive (non-secure) is a RAID 0 pair of WD Black drives, for speed
    - Internal backup to a WD Geen
    - External backup to a RAID-0 2 disk NAS (bit long in the tooth, only 2 TB capacity), on-site
    - Most of my 'most important' shots, in JPEG, are on Flickr (pro member, so unlimited storage).

    My worries:
    - I don't have a good off-site backup of the RAW files in case of theft or fire
    - What to do with the old NAS? i.e. replace drives with bigger ones, buy a second NAS?
    - Still need offsite backup for the entire computer system.

    I live in a mostly mac (all my machines)/little bit windows (my SO's laptop) household, if that makes any difference. Right now, I'm looking at cloud-based backup services, but due to the amount of data, means most cloud options aren't cheap. I'm currently sifting and re-organizing my entire image catalogue, getting rid of stuff I know I'll never look at again (i.e. blurry shots out of a series that I haven't looked at in years and similar), so I'll probably only bite the bullet when it's clear exactly how much data I need to back up, and whether something like Glacier makes more sense than CrashPlan (or similar). I'm interested in hearing about people's experiences with various products and solutions.
     
  2. Reflector

    Reflector Mu-43 Veteran

    406
    Aug 31, 2013
    I own a lot of harddrives so I keep some offline and I keep data I value across redundant systems and drives: So my photos reside on more than one computer and more than one drive on some of those computers, in different rooms.

    Worse comes to worse: Get a set of external harddrives or a hot swappable 5.25" HDD bay, backup as routinely as you can and put them into a bank storage box. Then you won't have to worry about intellectual theft on "cloud storage" ("Storage as a service") or fire/water/etc. If you have enough money in some banks, they'll offer this to you free of charge.
     
  3. mattia

    mattia Mu-43 Hall of Famer

    May 3, 2012
    The Netherlands
    I do want a cloud solution - most are encrypted, and all are far more secure than even a multi-drive, RAID-10, off-site DIY setup would be, and I'm not terribly paranoid about the 'intellectual theft' issue considering there aren't any reported incidences of that that I'm aware of, nor do I think anyone will care enough about my pictures to bother with that. I don't consider any single drive (or pair of single drives) to be a particularly secure form of long-term storage...
     
  4. Reflector

    Reflector Mu-43 Veteran

    406
    Aug 31, 2013
    Funny, I've actually have heard far too many incidents of projects being placed on cloud storage solutions and then "finding that a new, upstart competitor suddenly has equal if not further progress" in non photography related business cases. Even more funny is that some of those companies often "welcome a member" of the company that so happens to of formerly worked at another firm as IT.

    The one that they also happened to hold the encryption keys for and/or master access.

    But what you do with your money is your choice.
     
  5. HappyFish

    HappyFish Mu-43 Top Veteran

    983
    Sep 8, 2012
    Chad
    some thoughts
    I have a post production company and still shoot for a living
    my main machine is a mac pro with Areca raid card for main storage
    I use a PM setup for backing up and off line etc..
    about 50 TB of attached data on my main computer :) so a bit of storage
    running 3 mac pros at the moment in our office so my needs might be more but its my income :)

    I like raids but not 0 for working data ? would prefer 10 if you are doing that or raid 1 since it gives you some safety ?

    a single HDD at about %60 full is about half as fast ? so raid is nice to keep speed up through say %80 like raid 0 can do so not knocking your raid 0 :) just if I go down its money and to me raid 1 or other forms like 10 or 5 or 6 are like run flat tires if you have a hardware failure you can keep working and won't loose your current stuff you are working on

    now I like to back up from hardware failure and human failure and catastrophe

    so hardware I prefer raid 1 or something depending on funds these days with 4TB drives raid 1 is nice and simple
    for backing up my working drives I like something close to as fast as the main working drives so I can keep going the way I am if I need to

    backing up for human failure I like time machine or incremental backup so i can go back and grab stuff at stages through the day

    offline for me online connection is to slow and I have to much data to move around so rely on sending off HDDs or storing at friends :)

    I am very curious about this unit now ? not a fan of drobos in general but this is kinda cool
    http://www.filetransporter.com


    I love the idea of NAS devices for backing up to :) for me to much money at the moment to change since I would need at least 40 or more TB so my storage needs are again something I think of slowly hahahahahah

    I think online stuff is really cool and the future and for personal stuff or smaller amounts if you have good connections is where the future is :)

    with 2 TB online would be a good option and some kinda incremental and secondary still at home :)
     
  6. mattia

    mattia Mu-43 Hall of Famer

    May 3, 2012
    The Netherlands
    My main data is on a Mac Pro which is configured as follows:

    - 120GB SSD for operating system / applications
    - A RAID 0 pair of 2TB drives (striped) divided into a 'fast' partition (first 2TB) and a 'slow' section that I don't really use for much at all.
    - Two separate internal 2 TB hard disks, one with an extra copy of key data, one as an internal Time Machine backup. Short of installing a third party RAID card I can't really do RAID 5 in-computer, although the thought has crossed my mind :)
     
  7. RickinAust

    RickinAust Mu-43 Regular

    119
    Feb 9, 2013
    Australia
    I am in a similar position. Have a NAS that stores all my files, have about 2.7TB of storage and only about 400GB of pics so no problem yet. My main concern is if a disaster struck like fire, theft or flood. If the NAS was say stolen I don't have a back up. A possible solution is portable hard drives stored off site.But this is a pain to maintain. A cloud storage backup would be great but a few problems:
    1 cost - most are too expensive. Some don't cater for RAW.
    2. Speed - Would take months to upload. Unless they offered a "send in a disk" option for initial upload.
    3 Recovery - same problem as for uploading.
    4. Stability of service - By this I mean will they be around in a few years time?
    I am considering a DIY Amazon S3 approach. It seems fairly priced and there are a few GUI client front ends that can be used.
    However, I would be very interested in hearing of any solutions that people have?
     
  8. HappyFish

    HappyFish Mu-43 Top Veteran

    983
    Sep 8, 2012
    Chad
    not sure what you need the speed for or just trying to fill them up and gaining a bit extra ? I don't do video much but for PS and LR s single drive is plenty fast enough won't matter if you have faster for those photo apps ?

    for cache or scratch drives I use SSD
    same goes for boot SSD

    IMHO the whole partition thing really does not matter anymore about fast and slow areas did a long time ago I would just use the whole drives if you are doing raid 0 and no partitions on them
    macs are pretty good about laying down and keeping data on the drives so they will stay on the faster outer parts anyway so might as well leave more room empty :)

    no shortage of options what to do and configure though :) I would say do a online backup for stuff since 2TB is not to much really and the jpg online gallery is handy I know a few folks that do that also :)
     
  9. HappyFish

    HappyFish Mu-43 Top Veteran

    983
    Sep 8, 2012
    Chad
    some of the online backup won't care about raw ? some can send you a seed disc to help with the initial bit of data ? when figuring monthly cots vs not buying the HDD not a bad deal ?
    good point about will they be around and I think for getting your data back they charge to send a disc out !


    for off site you really need at least two drives and bring that one with from home and change out with one at location and bring back not trying to bring back and forth a single disc other downside is most just don't do it or set expectations to high but hey once a month is better than never for offsite :)

    option if you have friends or parents or kids :) do a yearly backup and just use the current year for online stuff could save space and time etc. all your data is safe also that way
     
  10. alex66

    alex66 Mu-43 Top Veteran

    715
    Jul 23, 2010
    I have a simple ish back up system in place all photos are on at least 3 hard drives, might be 4 with older replaced drives, also 2 sets of DVD's, one HD back up is upstairs at my dads house and a set ov DVD's live at my brothers. There is frequent swapping out of the HD at my fathers, the next step I am considering is a set of USB sticks that I will keep on my person for the intermediate files, i.e. between drive swaps. Also I buy a drive every year or so with greater capacity than the archive, on line cloud whatever storage is not an option my internet speed is a joke.
     
  11. Ian.

    Ian. Mu-43 All-Pro

    Mar 13, 2013
    Munich
    Ian
    2 thoughts on off-site storage.
    You could have 2 drives for off-site. One you take to a relative. Thus securing your data against flood and fire. It would have to be someone you visit weekly to swap them. You are also secure against data-loss through burglary. Unless both locations get robbed on the same day. It does not cost much.
    You could also synchronise to a friend or family member online. Via say Dropbox or Ubuntu one. Also free. And they can sync back to you. So you both have offsite storage.
    "Hi Grandma! I'm just going to set up a server in you spare bedroom. It uses loads of electricity, but it will keep you warm in the winter." I think she'd prefer the drive swap option. Then you'd be forced to visit her more often. "..he comes around for a little chat every week now. He doesn't stay long though..."
     
  12. Hagane

    Hagane Mu-43 Regular

    88
    May 31, 2013
    Limburg, Netherlands
    My stuff goes to my NAS which has a RAID 1 with 2x2 TB harddiscs. Every 1-3 months I backup the NAS to an external USB-drive which stays in a small fireproof safe. I'm still looking for a convenient off-site storage solutions (and no, not something in the cloud...)

    Btw, don't trust a NAS alone, friend of mine lost all data on his NAS last year because lightning struck his home and afterwards the NAS wasn't functioning anymore and he needed help from a date rescue company.
     
  13. Amin Sabet

    Amin Sabet Administrator

    Apr 10, 2009
    Boston, MA (USA)
    My strategy:

    A) All RAW files get simultaneously imported to 2 internal locations:
    1) Drive E: 2 x Seagate Barracuda 3TB in RAID ) (for speed)
    2) Drive G: 1 x Seagate Barracuda 3TB (for backup of RAW files)

    B) The directory containing my images on drive E is continually, incrementally backed up to a USB 3.0 Seagate Backup Plus 3TB using Acronis True Image Lite

    C) My most important images are put on Flickr in JPEG format and often printed as well

    D) About once every 2-3 weeks, I do a bulk upload to Amazon Glacier of all the files from that 2-3 weeks. I use a client called FastGlacier (Pro Version). Using this client on my FIOS connection, it takes about an hour to upload 15GB of files. It is dead simple to use - just select your Glacier vault, click on Files -> Upload Folder and then choose the folder you want to upload.
     
  14. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    Just a few things to remember:

    - Any internal or permanently mounted drive that can be written just like any other drive is at risk from crypto-locker malware. It is, however, a good way to make sure you've got live local backups. So it can be part of a backup solution. To protect against viruses/crypto lockers your backup solution needs to be incremental (i.e. doesn't overwrite previous data) and have some protocol between the computer being backed up and the backup media more complicated than just a mounted filesystem. Things like NAS with rsync or even time machine can do this. So can the CrashPlan software which can do non-cloud backups for free. There are other solutions as well. Bottomline, if your backup media is persistently mounted to your machine that backup will be taken out by malware along with your system.

    - Not to belabor the obvious, but CD/DVD is not a reliable backup medium. There is no way to verify correct burns. Verifying immediately after burning is meaningless, you can burn a disk that reads back fine for a week and is blank in a year. If not burned exactly right the dyes will fade very quickly but read back just fine for a few days or weeks after the initial burn. CD/DVD is a semi-useful way to send data to someone. It should have no role in backups as it is a horrifically unreliable media.

    - Brew your own because of fears someone will steal your data or that Amazon Glacier is unreliable is, well how to put this diplomatically... Hmmm, there isn't a way actually. It is just plain stupid. Unaudited and infrequently verified backups are for fools. Might as well smoke menthol cigarettes to cut the risk of lung cancer. Seriously though, every part of a multi-location backup scheme is going to have some home brew aspect to it - I mean you need to set it up and monitor it even if you use a canned commercial solution. The thing is this is a solved problem and one that people screw up every time they try to reinvent it. Have at least one reliable off site, usually commercial cloud based, that is run by software used by a large user base. Yeah if you are savvy you can just rsync to an old Linux box you setup at a friends house. Ask me how many extremely talented technical people I know who have discovered too late that what they thought was simple and robust wasn't. Never, ever forget you don't want a single point of failure in your backup strategy and that includes you as a single point of failure. Lean on the experience and products (including FOSS) of the community.

    - Audit your backups. To the degree feasible attempt a recovery - frequently. Obviously do this in a way that doesn't jeopardize the data while you do the test!!! For a Mac person you could try to restore to an external hard drive (Macs can do that easily). For photos recover a random set of photos (especially very old and very recent) to a temporary location. Again, many people's first test of their backup is when they need it and sadly that often ends badly.

    - RAID of any form is not a backup at all. If you think you have any sort of extra copy because you run a RAID you are fooling yourself. RAID can *sometimes* protect against a HD failure and that is it. No protection from your own mistakes (accidental format/deletion), viruses (crypto-lockers or worse) or even a bad RAID controller or other hardware other than the drive. Worse many RAID solutions never scrub the RAID (i.e. verify the redundant data is actually still valid) and so when a HD does fail you suddenly discover that there was already a corruption problem before that failure and boom no data. RAID is for availability only, it does not provide any real data security.

    - Lastly, and already alluded to, you want not just more than one copy but more than one method of creating copies. You aren't just protecting against your house burning down, you need to protect against one of your methods being flawed. So while something home brew might be a dumb thing to rely upon on its own it can be a very sensible thing to pair with an external cloud based solution. While the cloud solution is likely far more reliable than your home brew you do want something covering in the rare case the cloud solution has a problem. Belt and suspenders and all that.

    One basic premiss to all this - doing it right and doing wrong take about the same amount of effort and dollars. So take the time to think it through and do it right.

    As to my preferred solutions at the moment I use CrashPlan for my cloud based solution. Both my working machine and my NAS backup to the CrashPlan cloud. Give CrashPlan a good look because you can actually use their software for free to set up offsite back-ups to friends and families houses. Software being used by lots of people so more likely if something breaks you will hear about it and if you don't want to pay for the cloud just throw hardware at someone else's house.

    Locally my work machine (a Mac) Time Machine's to the NAS, but I also have a lot of data directly on the NAS that isn't on my work machine. The NAS data is presently duplicated on an old Windows machine but I'll be changing that to external HD on the NAS that is periodically taken off-site.

    In short both my work machine and my NAS data is duplicated at least two places - offsite with CrashPlan and on site as well. My on site backups do not use the CrashPlan software so that is redundancy of method as well.

    I am thinking of seeing if I can do a separate backup directly to Glacier from the NAS. I think that will actually be a bit more sensible than using external HDs taken offsite physically by me periodically but I haven't decided one way for sure.

    Lastly I do about monthly spot check my backups and methods are still functioning properly.

    EDIT: On Glacier one more thing - this can be very costly to do brew your own. Glacier pricing shoots through the roof if you don't use it correctly and even some software packages don't do it correctly. The low price is for the storage, if you create too many transactions during the upload you can get charged quite a bit extra. Looking around it appears the Synology NAS software for doing Glacier backups is flawed and a bunch of people have been surprised by over $100 charges on their initial uploads to Glacier. So again, use something proven when possible rather than roll your own!
     
    • Like Like x 2
  15. mattia

    mattia Mu-43 Hall of Famer

    May 3, 2012
    The Netherlands
    Thanks for the comments, food for thought!

    Re: RAID and speed, honestly te RAID0 is more for my other hobby (audio recording, multi track) than the photo end of things. And I know the RAID1 is just redundancy :)
     
  16. greenlight

    greenlight Mu-43 Regular

    56
    Nov 16, 2012
    UK
    Colin B
    I haven't used them, but last time I researched it Backblaze looked like a good cloud option - no storage limits.
     
  17. stw

    stw Mu-43 Regular

    76
    Jul 23, 2013
    I am currently using Backblaze and have been pleased with it. That said, I have nowhere near the storage that some of you are mentioning. A quick login this morning shows that I have just over 320GB worth of files on Backblaze. If you do have some sort of catastrophic loss, you can either do a restore over the 'net, or have them send a harddrive to you for a fee (I just checked and it's $189 for a 3TB drive).

    The restore process is a bit clunky, but it works fine. You choose what you want to restore, then you will get an email with a link to a zip file. I restore a couple folders every other month to spot check to see that everything is working fine.
     
  18. dav1dz

    dav1dz Mu-43 Top Veteran

    926
    Nov 6, 2012
    Canada
    Everyone seem to have a solution that works for them. For me it's much simpler and I've never had any problems. The trick is to have a lot of external drives. I have assembled a few of my own from left-over 2.5" drives, and bought some packaged ones from WD, Seagate and the like. I only use 2.5" drives for backup, I know the capacity is often smaller but I don't want use and keep yet another power supply that is often required for 3.5" drives. Nice thing of assembling your own is that you can update the interface by buying another enclosure, so you're not bottle-necked by USB 2.0 of FireWire 800 when your new machines support USB 3.0 or Thunderbolt. Though I haven't found a reasonably priced Thunderbolt enclosure yet.

    I'm on a MacBook Pro with a 240 GB SSD, this is an aftermarket upgrade and does not support TRIM (I'm not going to enable it either using an "utility", no thank you). But it does have its own background garbage collection process, which I found to work great. Either way, I have to ensure my backups are proficient enough to not cause any headaches in case the SSD goes haywire. Living on the edge without TRIM!

    Time Machine, and a bunch of external drives. Haha that's it. You can use Areca Backup if you're on Windows or another platform. Any of these applications that supports delta backup (incremental changes) without overly complicated compression method is great. When using Areca Backup I don't even use any compression, straight data exchanges makes recovery easier. I like Areca Backup because it's cross platform and open source.

    Here's my method:

    1. Only have 1 year worth of photos in production on the primary SSD keep recovery quick and backups small.
    2. Load anything >1 year onto an appropriately marked external drive, a backup image of that drive is made using Disk Utility or Windows Backup and stored separately on a dedicated external drive drive.
    3. Time Machine or Areca Backup the production drive every few weeks onto a 3rd external drive. Just like my grandma.

    Done! Sometimes I get lazy but Time Machine has annoying pop-ups to keep me vigilant.

    Recovery is done using Restore from Time Machine during a fresh OS X install, or using Areca Backup in Windows once I load the necessary drivers and applications after fresh installs.

    I would use off-site backup, but my family's financial records, passports, tax returns, and other legal documents are more important than my photo archive. So if I rent a safety deposit box type of thing, those would probably have precedence over a bunch of external hard drives.
     
  19. dornblaser

    dornblaser Mu-43 Hall of Famer

    Aug 13, 2012
    Chicago-area
    David Dornblaser
    I just switched over to Backblaze and really like it. I found that Crashplan and Carbonite slowed down my Mac noticeably. I opted for cloud backup instead of swapping out backup drives to our safe deposit box. Like others, I am Mac based and have G-Tech drives in RAID 1 and 0 for local backup and as a work drive. It is nice that Time Machine can now back up to two drives.
     
  20. darrellc

    darrellc Mu-43 Regular

    112
    Aug 21, 2012
    Working drive is 4tb ext enclosure, gets backed up to NAS everytime I transfer files from a camera, every quarter that gets backed up to hard drives that go into bank safe deposit box and between quarterly offsite backups all incremental files are backed up automatically nightly to Amazon AWS.