Backup options

Discussion in 'Open Discussion' started by Ramsey, Dec 14, 2015.

  1. Ramsey

    Ramsey Mu-43 Top Veteran

    719
    Jan 9, 2013
    Zagreb, Croatia
    As they say, you will eventually lose all of your files.

    Sorry for the upcoming wall of text.

    And that happened to me few weeks ago, with the only option of professional specialized data retrieval company trying to save some data, which i will do this week. I don't care about music and movies, i only care about my photos (the external hdd had my photos back from 98-99). I'm not a pro, the photos are mostly snapshots of friends/family, with the most recent ones (including pics of my family) still on laptop but still, they're treasured memories.

    So i'm currently in the process of gathering information about future backups and would like some opinions and experiences.

    I only have a laptop. I want 99% secure (i know there's always that 1% chance that everything will be lost) backup. I backup once a week (if we're talking about external hdd). I don't believe in cloud (call me paranoid, i can't help myself) and i don't feel like paying for cloud services on a monthly basis. Before i found out that the external HDD was going to be sent to the data retrieval company, i thought about NAS with RAID 1. I'm not an expert per se in those things, but i know my way around computers and am willing to learn. WDs My Cloud mirror or QNAP 268 were my preferred options, with the thought of upgrading to something with 4 or more slots in the future (the ability to access data remotely, with the option of hooking up the NAS to the TV making it a media center had me excited like a little boy). Now, with the additional cost of saving the data, wifey is not that keen on that big of a budget hit (prices in Croatia are higher than in the US, so we're talking about 50% of the average monthly salary for a QNAP with two 3TB red WD drives vs 25% for 2 external 2Tb HDDs).

    I'm not keen on, nor do i have the room for installing a dedicated server with racks.

    So i'm thinking about two external HDDs. All the most often recommended drives have a certain failure rate. Toshiba Canvio basics, WD My Passport Ultra, Samsung M3, Seagate backup plus slim. Reviews on amazon like "I will never buy another [name brand] disk, because i had 6 in the last 2 years and all failed" are not that reassuring. True, the 1 star negative reviews are like 5-10% of all reviews but i'm terrified of the option that both drives fail.
    Should i be that worried?

    Is it safer to get
    A) two of the same external HDDs,
    B) one e.g. Toshiba and one WD or
    C) two red WDs with enclosures bought separately?

    I don't carry HDDs around, they are stationary, in a drawer (dry, no shaking, room temperature). It doesn't matter if it's 3,5" or 2,5", though for convenience sake, it would be good if it doesn't need it's own power supply (i'm willing to forego this if 3,5" and a separate power supply device has a much lower failure rate).

    So, give me your recommendation/experience/advice.

    TL;DR: Failed external HDD. Probably lost most of my photos. Money is tight (when is it not?). Would like to hear recommendations for future backup (two or more external HDDs, NAS, sth else).
     
  2. pasisti

    pasisti Mu-43 Regular

    66
    Dec 16, 2013
    Helsinki, Finland
    I'm sorry for your bad luck, I know the feeling..

    Maybe it will be helpful if I explain my backup routine and the logic behind it!

    In short I have my PC, two Synology NAS's and a non-powered external HDD. I have all my RAW and edited photos on my computer. From there I backup them to my on-site Synology NAS, where they are safe from anything happening to my computer or its HDD. They are not safe from any large scale accidents such as thunder, fire or water accidents or robberies. For that reason I backup the edited photo (they are more important, anyway..) to my previous NAS which I didn't sell when getting this new one but instead placed at my mother's house. I do the routine of downloading the files there about once a month or two. On top of that, I have an external HDD sitting in a drawer somewhere. It is currently safe from any power or thunder accidents but not from anything else. It is mostly for my convenience in case of emergency.

    So I think it's crucial to think of everything that can go wrong, such as power, fire and water accidents and robberies. A backup at parents' place is pretty much safe from all of these! Of course in my situation there is the off-chance of a fire accident happening in both my place an my mother's place at the same time but I'm kind of optimistic about my chances there :)
     
    • Agree Agree x 1
    • Appreciate Appreciate x 1
  3. listers_nz

    listers_nz Mu-43 Veteran

    256
    Nov 22, 2013
    Christchurch, New Zealand
    Simon
    I'll say it again - RAID is not a backup. It provides high availability of your data, i.e. if one drive fails you can swap out the drive and have it rebuilt automatically while styill having access to your files. The problem with RAID is that there are a lot of events it won't cover you for or actually makes things worse, e.g. if you inadvertantly delete half the files on your NAS, then RAID will faithfully mirror that across both drives in the array.

    You want a minimum of two copies of your data on separate disks, e.g. separate enclosures or separate NAS, etc, and periodically copy from one to another. You don't have to use NAS drives, USB3 is pretty quick. NAS does make it easier in that they are generally always on and connected, whereas USB drives tend to get put in a drawer and fogotten about (guilty!). Any backup routine which requires you to manually do stuff will suffer from the "I can't really be bothered" syndrome :)

    What I do is as follows:
    I have a reasonably large drive in my laptop where I will generally have a lot of recent stuff - basically I've taken the DVD drive out and put another drive in there.
    Everything is backed up (automatically) to a server on the network (could be a NAS, mine isn't its my home theatre PC)
    Data is copied periodically (once per day, overnight) from the primary server to a second NAS. If I screw something up then I can always cancel the process and fix it.
    I also periodically back up the important data to a 2.5" USB drive that lives in my work bag (I used to leave a drive at work, but it doesn't work as a backup if you don't bring it home periodically).

    Provided you use separate drives for your backups, and you have more than one copy, it doesn't really matter what brand the drive is. If it fails, toss it, buy a new one and copy the data back onto it from your other backup. The chances of two identical brand drives failing at exactly the same time are somewhat remote. But no drive, no matter what brand, will last forever.

    (And for anyone who is interested in doing cloud storage your own way, on your own hardware, I recommend you have a look at ownCloud (ownCloud.org). )
     
    • Agree Agree x 2
    • Appreciate Appreciate x 1
  4. bigboysdad

    bigboysdad Mu-43 All-Pro

    Aug 25, 2013
    Sydney/ London
    You've overcome the first step by accepting that all data will eventually be lost. I'm truly sorry that your valuable data may have been lost. My thought process is, from now on, what's the worst that could happen? Your house will burn down taking your data with it. So you back up to the cloud to prevent data loss in the event that your external backup drive(s) get melted. But you don't want any cloud services. Ok, then you'll need to buy at least 2 hard drives and rotate the backups (running a backup at least once a week), ensuring that one of the drives is always somewhere else and will not be destroyed. Respect this routine, do not be lazy with it. 2 cheap portable drives are fine, the brand is not the most important thing. What's more important is that the drives should never be over 70% full as having a drive near capacity is dangerous. They should also be retired and replaced say every 18 to 36 months. These are basics, but can never be reiterated enough. A key thing also is software. Do not be cheap about this, your data is too important for that. Don't just copy files across, archive them, so you have copies of changed or deleted files in your backup drives, just in case. I say this, as data loss is mostly because of user error, not the computer. I don't use a Windows machine so can't really make any recommendations there but if you're a Mac user, pay the $40 USD for Chronosync. The developer is not some new kid on the block, but has been developing and updating this software for over 20 years. He knows about backup. It has never ever let me down. If you use a Mac, I don't think there's anything more secure than a password protected Disk Image which you can put on the external drives you're backing up to (obviously, the password need not be "1234" or "open sesame"!)
     
    Last edited: Dec 14, 2015
    • Appreciate Appreciate x 1
  5. Ramsey

    Ramsey Mu-43 Top Veteran

    719
    Jan 9, 2013
    Zagreb, Croatia
    Thank for the replies guys. I'll probably get two separate external HDDs (Seagate has a higher failure rate, so WD it is) for now, backups manually bi-weekly. As soon as i can swing it, one dual bay NAS with automated backup daily. As soon as i can swing it, again with a 4 or more bay device, the first one is repurposed as "most important files backup" at another location (parents or otherwise).

    Good point. I plan on assigning privileges on a NAS, read-only folders for parents and wife (not sure how she will take it, but "Honey, it's a security issue, you don't want us to lose the photos, do you?"). I don't to bulk delete, and always check before deleting anything. Still, something to consider.



    currently, this would be overkill for my needs. But i applaud your dedication to security :)

    Thank you for the tips. The capacity and replacement rates are something i'll take into account in future. Don't worry, in case of fire, my first concern is my family and HDDs :)

    Archiving is not my priority currently, as i rarely perform that kind of operations for home usage. For work needs, i can understand it, and will keep in mind.
     
    • Like Like x 1
  6. Speedliner

    Speedliner Mu-43 All-Pro

    Mar 2, 2015
    Southern NJ, USA
    Rob
    Nothing like off-site, automated, daily backup.

    Also like Iosafe external drive or NAS. They may endure fire and water damage. The best backup scheme doesn't help if all is lost in a fire.

    I don't know anyone who actually moves external drives between multiple locations reliably. Everyone eventually gets lazy.
     
    Last edited: Dec 14, 2015
  7. dornblaser

    dornblaser Mu-43 Hall of Famer

    Aug 13, 2012
    Chicago-area
    David Dornblaser
    + 1
     
  8. b_rubenstein

    b_rubenstein Mu-43 All-Pro

    Mar 20, 2012
    Melbourne, FL
    I don't like USB external HD's for a few reasons:
    - They will be close to the computer and exposed to physical knocks and bangs.
    - Anything physical that happens to the computer will probably also happen to the drives due to their close proximity
    - External drives typically don't have very good cooling and the drives can run hot.
    - They Don't have very good failure rates

    I believe Network Attached Storage (NAS) is a better solution. You can buy a NAS, or they can be homemade. Small Net Builder has a lot of information ready made and home brew ones. A server doesn't require "racks of equipment". I built a Windows Home Server (WHS) PC using a M-ITX board very small case a couple of 2TB storage drives and a SSD for the OS. What is really needed is good, reliable backup software that automatically backs up the computer. I like WHS 2011 because it wakes up and runs an automated incremental backup on all the computers in my house every night. It can even do a Bare Metal Restore (replace a failed HD with a new one and the server can restore the OS, programs an data to the new hard drive(s).

    Whatever backup system you decide on, make sure that you can actually restore files.
     
    • Agree Agree x 1
    • Informative Informative x 1
  9. budeny

    budeny Mu-43 All-Pro

    Mar 4, 2014
    Boulder, CO
    In my home PC I use LSI 9260 raid card (real hardware raid) with 4 green WD drives 3TB each (disabled idle power off) in RAID10 configuration. This gives me 6TB of good speed (200 MB/s) storage and for backups I use offsite USB drive that I refresh every month.
     
  10. tkbslc

    tkbslc Mu-43 Hall of Famer

    You really need to get your backups offsite to protect against theft, fire, flood, etc. So you can either do that with a cloud service (mine was under $100 for 2 years) or you can buy two external drives for backups and use some type of weekly rotation. I used to do this and would keep the backup at work in my office. But when I realized I was spending $100-150 or so on backup drives every other year, I decided cloud service was just as cheap and I don't have to remember to move them offsite regularly. And if I'm honest, my "weekly" rotation ended up being more like every 3-4 weeks. It's easy to forget or procrastinate.

    I do a cloud backup every other day to idrive.com. On the other days I do a local robocopy backup to a second drive in my computer. THis happens automatically every evening and since I'm only doing new/changed files it takes about 30 minutes tops. So if I have a localized issue (bad drive, accidental delete, etc) I can just keep working from my second drive. If I have a major system failure or theft, I can restore from idrive to my new computer. I also try not to reformat my SD cards for a week or so after an important shoot so I can make sure they've hit both backups.

    I personally don't think NAS is a good value unless you have a LOT of data. For the cost of a decent NAS, you can pay for a decade of cloud backups. And your NAS will go in a theft or fire too.
     
    • Informative Informative x 1
  11. SVQuant

    SVQuant Mu-43 Top Veteran

    877
    Sep 20, 2015
    SF Bay Area, California, USA
    Sameer
    Sorry to hear about your files. I am helping a friend recover his photos drives (raid failure, unknown causes) and can imagine how you feel. Lots of great suggestions above. I will second the off-site backup idea especially if you have a fast upload internet connection.

    If you do go with a pair of local (USB/NAS) drives, another small suggestion would be to buy them from different vendors. That way, you can reduce the risk of buying disks from a bad manufacturing batch. FWIW, I have installed multiple local RAID and NAS for my work and personal use and have had no issues with WD drives so far.
     
    • Agree Agree x 1
  12. Ramsey

    Ramsey Mu-43 Top Veteran

    719
    Jan 9, 2013
    Zagreb, Croatia
    Thanks everyone for answers and suggestions.
    Not sure if i'm being too carefree, but this first step of backup/security will probably not include the mitigation of risks of fire/theft/flood. We didn't have a flood in my (part of) town for over 50 years. The houses here are built with brick and concrete, so the fire risk is much lower. In case of a gas leak/other explosion, my photos are probably the least of my concerns. Theft risk is always present, but that is one thing i will try to take care in the future with off site backups (NAS or otherwise).

    The good news, which i found out yesterday is that i did a sort of a backup of some older photos (even the ones from early 2000s) on my Playstation 3 two or three years ago. It was done mostly for showing to my wife that the PS3 "was not only a gaming machine, it is also a media center". Now i can say it is also a backup option (i don't think anyone, not even the people in Sony thought of that purpose).

    RAID 10 with Red WD drives was my plan if i went with the four bay slot NAS (QNAP TS-451 was in mind). The best of both RAID 1 (redundancy) and RAID 0 (speed). RAID 6 will be implemented if i get something with more than four slots (losing two slots, not half).

    Good suggestion for different vendors, the bad batch issue was my concern. I've already made my mind on WD drives. Regarding HDDs in NAS/enclosures, i've done some google-fu and found that Backblaze (cloud provider, 41,000 HDDs in use) analyses their HDD failure rate and the results are heavily in favor of Hitachi. I know, i know, anecdotal evidence is the best kind of evidence, but still, interesting read (Who makes the most reliable hard drives? (updated) | ExtremeTech)

    Backblaze-2015.
     
    • Informative Informative x 1
  13. barry13

    barry13 Super Moderator; Photon Wrangler

    Mar 7, 2014
    Southern California
    Barry
    Lots of good points already.

    FWIW, my strategy is as follows:
    1. HDD in PC (which will probably become a RAID1 soon). ~200GB of photos at the moment

    2. regular backups to the NAS (FreeNAS with ZFS/ZRAID1) and to the HTPC. Don't forget to copy the catalog files for the DAM (I use DigiKam)

    3. regular backups to a portable drive, which goes in a fire safe when not actually being backed up to.

    4. occasional rotations of a pair of portable drives to a fire safe off-site

    5. documents also copied regularly to Google Drive (not currently copying many photos there; maybe I will start copying my favorites at least)

    6. to-do: off-site sync over a VPN to a friend's house; evaluating software but may just copy photos and use encrypted RAR files for documents at first.

    I keep multiple versions of my collection, in case of accidental deletion or corruption (I did have a drive go bad once that corrupted some of my music collection). One way to do this if you have enough space on the backup medium is to simply make a folder for each backup, e.g.
    Photos 2015-10-01
    Photos 2015-11-01
    ...

    Be sure to TEST your backups regularly! Make sure you can read them and find your files, and that the files are viewable.
    This is especially important with Cloud services.

    MS SyncToy is a easy way to schedule backups on Windows, fwiw.
    For manual copies I like to use Total Commander as it gives lots of control over timestamps and has a good sync tool to see what you've deleted (or lost) locally.

    Note that most RAIDs cannot detect corruption until it's too late; that is one of the big advantages of using ZFS (or possibly BTRFS) with regular 'scrubs' and SMART checks on your NAS.

    Barry
     
    • Appreciate Appreciate x 1
  14. SVQuant

    SVQuant Mu-43 Top Veteran

    877
    Sep 20, 2015
    SF Bay Area, California, USA
    Sameer
    Barry,

    That's pretty much what I do as well. Small differences: I am currently backing up my photos onto Microsoft OneDrive and other documents onto Google. I have a VPN backup to my office, but VPNs are slow and I don't do photos on it. I have looked at dedicated cloud backup options such as Carbonite, but I have not pulled the trigger on any of them.

    I currently have a Raid 1 in my PC (2TB WD Black drives) and and Raid 1 Synology NAS (2TB WD Red drives) to which I backup nightly. Also I have a Raid 5 Media Server on which I put the keepers for slideshows. All were initially populated with WD greens, but have been migrated over the last few years.

    I believe that Hitachi does not make hard drives any more. They sold their hard drive division (which they had bought from IBM) to WD some years ago. There may still be some Hitachi branded stuff out there, but Seagate and WD are pretty much the only games in town now.

    I would echo Barry's suggestion of off-site/off-line backups for your favorites by putting them on an SSD or a bunch of DVDs and keeping those in a bank locker. I think that it is an excellent way to make sure that memories are not lost.
     
    • Like Like x 1
  15. PakkyT

    PakkyT Mu-43 Top Veteran

    767
    Jun 20, 2015
    New England
    Hitachi is now HGST and is owned by WD (as you mentioned) and still put out their own line of HGST drives. The question is if they still do their own separate R&D and manufacturing or if WD is starting to make them use WD designs. So yes you can still by "Hitachi" drives but unknown how different they are from the WD lines.
     
    • Informative Informative x 1
  16. SVQuant

    SVQuant Mu-43 Top Veteran

    877
    Sep 20, 2015
    SF Bay Area, California, USA
    Sameer
    Thanks for correcting me. I wrote without doing a product search. Living here in Silicon Valley, one sometimes hears rumors and I had thought that HGST was going to be phased out by earlier this year. Clearly incorrect information. I used to know R&D engineers at both, so maybe I will try and find out if the R&D is still independent.
     
    • Useful Useful x 1
  17. siftu

    siftu Mu-43 Top Veteran

    646
    Mar 26, 2015
    Bay Area, CA
    siftu
    Nice Barry, your setup is similar to mine. Instead of VPN I have reverse ssh tunnels to family members houses. I have a bit of the process here.

    Backups.. but I'm lazy. // SIFTU's Systems
     
    • Useful Useful x 1
  18. barry13

    barry13 Super Moderator; Photon Wrangler

    Mar 7, 2014
    Southern California
    Barry
    I don't think I mentioned this here already:

    I have nothing against using DVDs or Flash (SSD or SD or thumbdrives) for backups, but don't leave them for years and expect them to still be good...

    DVDs and CDs, especially ones you 'burn' yourself, often go bad in 2-10 years. I've seen a large batch of CD-Rs (labeled Sony) at a former job site that went bad in less than a year after recorded to, burned on various drives and stored properly.
    Exposure to sunlight (UV) will greatly increase risk of loss.

    Flash media will eventually erase itself; iirc some enterprise-grade drives are guaranteed good in storage for 5 years, consumer media will be much less... the solution for Flash is to rotate it and re-write the data at least once a year.

    Barry
     
    Last edited: Dec 15, 2015
    • Agree Agree x 2
    • Informative Informative x 1
  19. siftu

    siftu Mu-43 Top Veteran

    646
    Mar 26, 2015
    Bay Area, CA
    siftu
    All good points. I think of data as a living breathing thing. If it's not online somewhere and accessible then who knows what happens to it. I too have had data become corrupt over the years (minor files) thus have been using ZFS as the filesystem for maybe the past 8+ years.

    For the average user I think the cloud storage is the safest and in the past year or two it has become affordable for the amounts of data we consume. The providers have the resources to monitor and replace equipment as needed and you don't have to think about it (this is worth $$). The problem with home based solutions like I run is all the work falls back on me. As it happens I don't mind that so much but it's certainly not for everyone.
     
    • Agree Agree x 1
  20. griswoldo

    griswoldo Mu-43 Regular

    81
    Sep 27, 2011
    Sorry, but I'm gonna be "that guy" and call you paranoid. You could backup all your data to multiple cloud services many times over for what you will spend in money, time, and effort trying to back things up on your own. Do yourself a favor and look at Crashplan, Backblaze, etc. For ~$50 per year you get truly secure, redundant, and perhaps most importantly constant offsite backup. Set your account up with a private encryption key, and no one but you can ever see what you're even transferring or storing. There's a reason a lot of governments don't like encryption - I encourage you to do some research if you truly are paranoid about others snooping around in your encrypted backups.
     
    • Agree Agree x 2