What is Data Decay
“Bit Rot” also means Data Decay, or Data Rot. It happens most visibly in photos because photos have so much data to define them. A one-bit flip in photos can make a disproportionately large change in the look of the image.
Most modern drives have means to recover from Bit Rot. Usually, this happens when the drive detects a potential fault when reading the data, automatically reassigning the data to a better block. Also, copying and rewriting “refreshes” the data, possibly keeping it from becoming a bad bit for longer.
Neither HFS+, Time Machine, nor keeping your files in Dropbox, protects from “Bit Rot”. In all cases, the local copy may suffer corruption, and then it gets copied to the backups, overwriting the originals. However, to some extent, if corruption gets detected early, the good original can be restored to replace the bad copy. Unfortunately, some instances are hard to notice and restoration rarely happens in time.
Its true that running backups regularly will force a read on pictures. This depends on us manually backing up old storage devices (that 2 year old hard drive in the cupboard anyone?).
To be reliably confident that our data is safe, we simply need parity checks and/or storage formats with data integrity checks (e.g. ZFS).
So I had a few hours to learn Golang this evening. I had made a snippet to emulate corruption in the images. It was simply for me to visualise how having messed up bits might change the look of images.
(Note: zoomed into defect on the upper-left image)
The code to emulate bit rot can be found at github.com/nicluo/bitrot
Disclaimer: My code doesn’t flip individual bits but zeros an entire byte. In both cases data corruption still happens, so I won’t try to make sense of how the look might change depending on which parts are corrupted. I assume any amount of corruption is bad.
Many of my old photos from when I was younger has some form of corruption in them. Over time, some drives had failed and I had to use PhotoRec to recover images, I lost backup CDs when I had to move from one place to another. Some storage are simply forgotten. This is sad.
Today I keep my photos in my Mac “Pictures” folder with Time Machine set up. I have a 2TB ZFS drive for keeping important files. Neither is reliable enough. I am still looking for a better answer right now.