Search

Photography Blog and Information

Enjoy my Articles, Blog and Photography related informaton

Photography Content for the many

Photography, Videography and Data Management
TrueToad 1857

Photography, Videography and Data Management

Keep or Cull

Our younger videographers and photographers grew up in the world of digital and at the press of a button can take selfies, videos, and photographs, and the data is quickly stored onto the media card, shared on Instagram, Twitter, Facebook or offloaded to local or cloud storage.  

In the good old days If you grew up in the film emulsion era, you most likely had to deal with the physical storage of the exposed film emulsions and archival prints. Photographers in that era exercised caution because film and printing negatives were cost-prohibitive. Do you recall the analog video days? We had drop-in video cartridges. In that day and time, we basically needed "physical" storage space for out tapes.  I had shelves of 35mm negatives, a few hundred 120 negative rolls, and over 100 video cartridges - all taking up physical space.

Timewarp to the present day: I like many who read this have overtime converted their collection of analog video and scanned my film emulsions and stored all that digital output on my hard drives - many, many hard drives and external drives. The transition from analog to digital was a learning experience with a never-ending appetite for more storage space.

After I bought my first digital camera years back and upgrading cameras over the years I found myself only deleting the most blurry or under/overexposed images. I was out of control and had images spread across dozens of storage drives with no true way to know which drives held what images/video unless they were recently taken. I needed to mount the drive and fire up LR to browse endless amounts of images just to find the few I needed - time to change.

The problem with my "keep all" delete few mentalities was - as cameras evolved so did the camera sensor resolution measured in Mega Pixels - My first DSLR was 5MP and starting I only shot JPEGs. Starting my files were less than 1mb - no problem right? I developed bad data management habits and basically kept most of what I shot because it did not take up much space. Then came larger and larger camera sensors and with that came larger and larger images, not to mention going from those low res videos to shooting in 4K then switching to RAW along the way. A single image file from my D850 is close to a whopping 40mb on average. You ask when did I start using larger Mega Pixel cameras? The month the D800 arrived, then D810, my D850, and even larger yet the Nikon Z7 - so you know I was hurting for data management.

Enter Cloud and home Network Access Storage (NAS): My Data Storage Bacon is saved! NOT by a long shot. No way was about to pay for TERABYTES of cloud storage cost and pump years of BAD data upstream from my home internet connection - I was sure I would go broke and the ISP would shut me off due to over bandwidth usage Plus I would probably belong expired before I could move all that data to the cloud. I needed to get rid of the crap files and only manage the images and data worth keeping.

The first step solve the bigger issue: I had no true home redundant backup/storage solution - sure I had all those hard drives lying around but that is not a REAL storage/backup solution just a bunch of band-aids waiting to go bad. I needed a better mousetrap. After some research, I pulled the trigger and purchased a 12 bay Synology NAS and loaded it up with 8 10TB hard drives with an SDD as a cache memory for moving files quicker over the home network. Finally, I had a fully redundant system with true RAID protection. The Synology NAS sends me weekly reports on the health of the hard drives showing bad sectors and warnings if any and should a drive fail I simply pull it out and replace it with my standby cold spare and the NAS rebuilds the data - easy peasy.

Problem One Solved: With the Synology NAS running and tested it was time to start moving my photos, videos, and other important data to the NAS. This time-consuming part of the task is where I eat my own dog food as they say and live the nightmare I created for myself over many years. To make the pain easier, my Synology NAS has a USB port to plug in the external drives to allow copying the data over to the NAS.

The beginning of better Data Management: The quick cull - As each of my external hard drives data was moved to the NAS the days following I pointed  Lightroom to the NAS and created the library.  Using Lightroom to preview the thumbnails and cull out obvious bad images; out of focus - gone, blurry - gone, images of those accidental snaps - gone.  This culling process kept me busy for weeks and this was just the photos.

Under Control Using New Rules: One month later after all my image and video data were on the NAS and the initial culling of bad images was highly needed and conclusive - It was time to deep dive and establish a set of personal data management rules and use these rules against a more aggressive culling of images/data.

  • Rule One - Only keep data you are proud to display or have an extreme emotional bond with like your son/daughter's first day on planet earth.  I do bird photography and may have 50 shots of the same bird - do I keep all those images now? No! I now only keep a small group of the best from the session, the others deleted - that rule just saved me 500mb of space in one day of shooting, and this was repeated from folder to folder.
  • Rule Two - Don't become emotionally attached to images just because you took them - for your travel/family vacation photos try to save those that tell the story and delete the others.  You don't need image after image of the same subject going around on a Farris Wheel.
  • Rule Three - When in doubt toss it out. Seek advice from an unbiased third party to provide their opinion on the worthiness of the image/video.
  • Rule Four - Use meta-tagging for all images you keep and rate the images as you add them.

Building a File Structure: Because I will be using Lightroom and or Capture One to create catalogs and meta-tagging my images I felt the only real file structure I needed was by year and inside the year folder would be the individual months. This structure is basic and allows me to open the NAS folder outside of Lightroom and have a sense of where things are.  Lightroom makes it easy to do all the data management I require, tons of filtering options, search options such as by camera, the date captured, meta tags, lens used, or browse by folder year or month - or the entire catalog. Building an elaborate file structure outside of Lightroom for me was unnecessary and a waste of my time.

Don't Throw Caution to the Wind: If using ANY imaging software that stores your hand jammed metadata, notes, ratings, etc - make sure you back up that catalog to your RAID NAS and keep a dupe on your local PC just in case. A lot of work can go up in digital smoke if your catalog becomes corrupt.

Becoming More Disciplined: Don't wait months to offload your SD cards to your NAS and process them. Waiting when until you have 5000 images to go through can make it seem overwhelming and you might want to revert to bad habits or cut corners and not meta tag or cull out the poor images. Process your images as soon as feasible. Remember sending bad data up to Cloud storage = money wasted.

Years of Work Up in Smoke: Now that you are in control of the digital chaos and spent months beautifying your digital landscape - Consider the years of work in your collections - this can span one or two decades especially if you converted your film emulsion from your SLR days. Although the Synology NAS was built by you to withstand hard drive failures - consider the risk of fire, flood, hurricane, tornado, or some other natural disaster - or worse yet a home break-in. Your entire life's work and memories are gone forever.

Enter Your Spare Tire and Safety Insurance: You need an off-site backup with geographic separation for your data, the age of Cloud Storage will give it to you! Having an off-site backup is your insurance plan from disasters and thieves. Geographic separation means your off-site data is not located near where you live for obvious reasons. if an earthquake occurs and takes out your house along with the data center storing your data - what good is that?  Do some research and find a reputable company that suits your storage needs and pricing that is affordable. Keep in mind you not looking for instant access you need archival for recovery purposes only. In my opinion below is what I determined is a reasonable, reliable, and reputable solution:

My Choice: Amazon Glacier Cloud Storage Services. Amazon Glacier service is for long term archiving - archiving means the data is stored long term and not intended to be accessed except for recovery purposes, remember you hold the originals, and if the originals are lost you have the Amazon Glacier Archives to fall back on, this drives down the pricing drastically.  Below are rough examples of cost between Archiving vs Amazon Storage Gateway - again this is very rough costing - please check with Amazon for your cost.

 Rough Comparision
 Amazon Glacier 4 Terabyte $4.50 monthly approximately
 Amazon S3 4 Terabyte Up to $125.00 monthly
     
 * There is no cost to upload data to either S3 or Glacier, you pay a monthly fee based
on the amount of data your store or need to retrieve over your FREE monthly allocation.

Warning Notice a Volume is degraded
With Amazon Glacier you get up to 10GB of free data retrievals per month - so if you accidentally deleted one photoshoot and it was backed up in Glacier your cost will be FREE if the total is not over 10GB, otherwise you pay for additional retrieval. The other positive from either solution is the data is replicated to other geographic areas.

Good News for Synology NAS Users: Amazon Glacier Services are built right into the NAS you simply load the free application, Create your account on Amazon then set up a backup schedule for your data - what, when, and how often. Pretty simple.

Rate article

5.0
Rate this article:
5.0
TrueToad

True Toad

TrueToad

Other posts by TrueToad
Contact author

Comment

Collapse Expand Comments (0)

You are replaying to

Your comment was added, but it must be approved first.

Please enter your name
Please enter your email adressPlease enter valid email adress
Please enter a comment
Add Comment

Contact author

x
Terms Of UsePrivacy StatementCopyright 2020 by TrueToad
Back To Top