Scalable storage solutions. How do you handle dwindling HD space?

Discussion in 'Hardware' started by TRCPhoto, Apr 10, 2015.

  1. TRCPhoto

    TRCPhoto Mu-43 Regular

    82
    Dec 13, 2014
    Princeton, NJ
    Hello all, it's getting to be that time again where my external hard drive (4TB Western Digital) is getting low on space. In the past I've just bought a larger HD, transferred the contents of the full HD to the new larger empty unit, and moved on. Keeping the smaller in an offsite storage unit as an added form of backup. I feel there has to be a better way and I'm curious what everyone here does to manage their libraries? Sorry if this thread already exists in one form or fashion elsewhere.
     
  2. OzRay

    OzRay Mu-43 Legend

    Jan 29, 2010
    South Gippsland, Australia
    Ray, not Oz
    You must take and keep an awful lot of photos. I'm still able to store all of mine in a 1TB HDD with room to spare.

    Personally, I don't like the idea of storing things in ever bigger drives, if it fails, you've lost plenty. Even if you have other backups, Murphy is always waiting. If you need that amount of storage, I'd be looking at a suitable NAS, using multiple drives and backing up proportionally. NAS units are quite inexpensive nowadays and very fast with say USB3.

    I have an old Western Digital one that is a network NAS, but don't use it now as it's slow and a pain to use. It now simply houses two HDDs that are backups and I also use a drive caddy that houses another HDD.

    Cloud services are always another option.
     
    • Like Like x 1
  3. Ricoh

    Ricoh Mu-43 Top Veteran

    906
    Nov 2, 2013
    UK
    Steve
    Apart from a cull, have you considerd splitting your catalogue.
    4TB, that's a lot of photographs, especially if they're all keepers.
     
  4. griswoldo

    griswoldo Mu-43 Regular

    81
    Sep 27, 2011
    If you want to keep and store that many photos, just keep buying hard drives and use an online backup service like Backblaze (what I use) or CrashPlan. That way it doesn't matter if you lose a 4tb drive or just a few gigs. Everything is already backed up redundantly for you in a secure, offsite facility.

    CrashPlan is nice because you can run their client locally on your own network in addition to backing up all your stuff on their infrastructure. They'll also keep everything you back up indefinitely - regardless of whether you still have it locally. Backblaze is a mirroring service - they'll delete your files about 2 weeks after you delete them from your local storage. Backblaze has been fine for my purposes, but if I ever get the urge strongly enough one day I'll probably switch to CrashPlan. With the amount of data I have backed up though, switching would be quite a time commitment.
     
    • Informative Informative x 1
  5. JBoot

    JBoot Mu-43 Regular

    106
    Dec 4, 2012
    Scotch Plains, NJ
    Jerry
    Add a small RAID NAS and a little process change and you are good for the long haul. Just upsize the RAID box for future expected growth.

    In general good a disaster recovery process will provide you with at least two local and an one offsite copy. My process is below.
    1. I start by working locally on my laptop for ease and portability
    2. Crashplan backs up my laptop continuously (as per griswoldo)
    3. Once done editing I ensure Crashplan has completed (offsite /online copy)
    4. Then I backup to dvd (goes into firesafe box - passive protected on site in lieu of physical offsite)
    5. Only then do I clear my memory card (serves as a backup to this point)
    6. Then I move files via Lightroom to main NAS RAID 5 setup (active protected onsite and keeps my database accurate at the same time)
    7. Then backup Lightroom database to NAS RAID 5
    8. Then backup main NAS RAID to single NAS drives (kept in firesafe when not needed. - a more accessible passive onsite copy)
    This process ensures that, once my pics are loaded, I always have at least two copies and once done at least two local and one true offsite copy and my database is also protected.

    I can lose a drive in my RAID 5 and keep going. If my RAID 5 fails, I can connect a single NAS drive and keep going, etc.

    This is just what I do. You need to assess the cost of protecting your images yourself. In either case, build a process that works over time as TB grow quickly and hard drives fail without notice along with other disasters.

    Paranoid? Yes! But with all invested to take these pics I feel it's worth it.
     
    • Like Like x 1
    • Informative Informative x 1
  6. PacNWMike

    PacNWMike Mu-43 Hall of Famer

    Dec 5, 2014
    Salish Sea
    guess?
    Cloud backup is a practical solution only if you have fast internet. Getting 'em up there is the easy part even if slow. If I had to last resort to a 4gb cloud backup hard drives would be obsolete before I could recover my photos.
     
  7. Repp

    Repp Mu-43 Top Veteran

    501
    Jan 27, 2011
    Oak Harbor, WA
    I run a Synology 9tb NAS (also as a media server), and back up online to amazon (free with prime = $99, or unlimited photo only for $12 a year). I can access the NAS while traveling via Laptop or IOS app.

    So, my personal system is as follows.
    Local: Import to Lightroom with Smart Preview, Edit, Move RAW files to NAS, Monthly back up to Amazon and a USB external HD. This gives me 3 full copies and the smart previews.

    Traveling: Import to a 126 Gig iPad mini, sort and rate via Photosmith, Back up to NAS via Synology "DS Photo+" app if I have wifi, and leave the files on the SD cards. When I get home I import everything into Lightroom via Photosmith, and then continue as normal. This also gives me 3 copies of everything, 2 if I don't have internet access.
     
    • Like Like x 1
    • Informative Informative x 1
  8. TRCPhoto

    TRCPhoto Mu-43 Regular

    82
    Dec 13, 2014
    Princeton, NJ
    Good stuff guys, thank you all. I've been looking at the Synology and like what I see so far. I've been using back blaze for a year or so now, but was not aware of CrashPlan and will check that out asap. It sounds more to my liking.
     
  9. Repp

    Repp Mu-43 Top Veteran

    501
    Jan 27, 2011
    Oak Harbor, WA
    I think Synology is a great system, but if you've never used it before, there is a learning curve. On the positive side, I've managed to google or use their forums to overcome any issue I couldn't figure out on my own; but I also learn something new about it every few months that changes the way I use it.

    They are also pretty good about software and security updates, and there is free antivirus that can run in the background. My only regret is that I didn't buy one with more bays from the start. I originally bought a 2 bay 712+, and have already bought the 2 bay expansion for it. I'd recommend starting with a 4-5 bay one.
     
    • Informative Informative x 1
  10. Speedliner

    Speedliner Mu-43 All-Pro

    Mar 2, 2015
    Southern NJ, USA
    Rob
    I have all of my files, documents as well as photos and videos on a raid array on a file server in the basement. All of us have our own folders on the shared drive. I used CrashPlan for backup. The very first run took almost two weeks to back 400GB to the cloud. Since then, nightly runs take minutes to a few hours. It has been completely reliable. It's a nice setup because none of us lose anything important if a system crashed, we just rebuild the OS and reinstall apps and are ready to go with all of our "stuff" out on the shared RAID drive.

    I re-install Windows on all of our PCs every 18-24 months, so this arrangement makes it painless.

    It is VERY nice knowing that your pictures and things are safely backed up in the cloud.
     
    • Informative Informative x 1
  11. TRCPhoto

    TRCPhoto Mu-43 Regular

    82
    Dec 13, 2014
    Princeton, NJ
    Do you have time to elaborate a bit more? What sort of issues did you run into?
     
  12. TRCPhoto

    TRCPhoto Mu-43 Regular

    82
    Dec 13, 2014
    Princeton, NJ
    Ha, Ray I believe in Clarke's Law which says Murphy was an optimist! Anyway, I do shoot a good deal of events that produce anywhere from 500-2000 RAW files. I agree about having all of ones eggs in the same basket (I don't even use memory cards over 16GB) so in addition to the full HDs in storage I use Backblaze and I upload all my client's selects to my zenfolio site for proofing, delivery, and purchasing. This adds another layer of backup as well. Additionally over the last two years it seems every one of my corporate and editorial clients ask for files through Dropbox, which adds yet another level of backup of the selects.

    From what I've read here I think I'm going to sign up for CrashPlan and invest in a RAID NAS system.
     
  13. Repp

    Repp Mu-43 Top Veteran

    501
    Jan 27, 2011
    Oak Harbor, WA
    Sure, the biggest issues I've had were mainly due to me being a novice using the technology. The basic file system and it's use as a NAS are pretty easy and straight forward. Things I ran into were things like accessing the system via the internet and apps (quick connect in 5.1 made this very easy), hosting issues, port forwarding, media server, and streaming services. With a bit of research/edu, all of these problems were fixed.

    I will admit that I'm not really using the system to it's full software capabilities. I don't use it to host a website, email server, music server, or use the photo station all that much.
     
    • Informative Informative x 1
  14. dornblaser

    dornblaser Mu-43 Hall of Famer

    Aug 13, 2012
    Chicago-area
    David Dornblaser
    I have a SSD drive along with my internal SSD that I use as scratch drives. I use Time Machine and Time Capsule for my current work. All my archival material is on a RAID 10 drive. I use Backblaze to back everything up. I used Crashplan in the past but much prefer Backblaze. I also have a 2 TB Dropbox account that I use to share files. I thought about a NAS but prefer DAS. FWIW - I am completely in the Mac/Apple ecosystem.
     
    • Informative Informative x 1
  15. TRCPhoto

    TRCPhoto Mu-43 Regular

    82
    Dec 13, 2014
    Princeton, NJ
    What do you like/dislike about the two David?

    Repp, those sound like roadblocks i'd encounter too. :)
     
  16. dornblaser

    dornblaser Mu-43 Hall of Famer

    Aug 13, 2012
    Chicago-area
    David Dornblaser
    I used Crashplan twice, each time it slowed down my Mac noticeably. Not sure why. Backblaze was originally designed to work with Macs and it does very well. It was the first, or one of the first, to offer unlimited data back-up. Every drive that is attached to my Mac(s) is automatically backed up no matter the size. Finally, it is only $5/mo. For me, my system/network works well. The home network, with iTunes, etc., eliminates my need for a NAS. Backblaze and a large RAID drive provide scaleability.