max.engineer archives search feed

Backblaze Safety Freeze and Disk Utility Errors

Ran into “Safety Freeze” message by Backblaze backup.

The best explanation of what it really means is here. Pasting it here:

Disclaimer: I work at Backblaze and wrote the Safety Freeze code.

I can’t find any missing data.

Safety Freeze is a very very VERY specific thing. Backblaze maintains an append only log of what has been added to your backup in local files on your laptop called bz_done files, and these files are then sent to the Backblaze servers also. Since these files are append only, they can never shrink. By definition that would lose information, it is impossible. They are called bz_done files because they are what has already been “done”, what the backup has already backed up. There is another list called bz_todo which is what is REMAINING to be done.

Safety Freeze occurs when the Backblaze bz_done files shrink. Or more precisely the bz_done files on your computer are being pushed to the Backblaze datacenter and the Backblaze datacenter already has an even larger copy.

That’s it. That’s literally the only reason it can Safety Freeze. Disconnecting a drive simply cannot cause this, it really can’t. Even if you lose all the data on the drive, that would be a DIFFERENT error, not a “Safety Freeze”. Safety Freeze is highly mis-understood by pretty much everybody - Backblaze support, other Backblaze engineers, customers. They think it’s some “general” thing, but it’s incredibly specific.

Now, why would this occur? Why would the bz_done files shrink? The most common reason is this series of events:

  1. The customer edits a picture or document on their local laptop.
  2. Backblaze backup backs up that new version of the picture or document.
  3. The customer decides they made a mistake and uses a whole disk clone from before #1 occurred to get the original back. Or the customer uses something like Time Machine to roll back time an hour or two hours. This accidentally, unknown to the customer, rolled back time on the “append only” bz_done files also. That just “shrunk” the size of the bz_done file. Boom -> Safety Freeze.

Ok, so what are some other ways Safety Freeze can occur? One is kind of subtle and we’re fixing this to be more clear. If a customer clones their computer or drive, then uses both the “clone” and the “original”, here is the sequence that occurs:

  1. Customer clones laptop1’s boot drive and puts it in laptop2 to hand to their spouse to use going forward. Unknown to the customer, this means laptop1 and laptop2 are attempting to “share” a Backblaze backup.
  2. laptop1 adds a picture or document
  3. Backblaze backs up laptop1’s new picture or document
  4. laptop2 attempts to do a backup - but it results in both laptop1 and laptop2 being “safety frozen” because laptop2’s bz_done files are smaller than laptop1’s bz_done files - but they are SHARING A BACKUP, this confuses the whole situation.

We also have seen that there are some subtle timing bugs (in the past, now hopefully fixed). So one of the most important things you can do is NOT use an old installer - instead make sure you always go to https://secure.backblaze.com/update.htm and use that installer to uninstall/reinstall/inherit backup state.

If Safety Freeze happens ONCE I wouldn’t let it bother you. But if Safety Freeze happens twice in the same year, we need to go figure out what is wrong. Why are those bz_done files shrinking.

The recommended steps to resolve this is here. Namely:

  1. Download the latest backblaze installer.
  2. Launch the Uninstaller from it.
  3. Uninstall Backblaze.
  4. Install Backblaze.
  5. Inherit backup state. (All exclusions should get preserved).
  6. Restart the Mac.

That said, running Disk Utility’s First Aid on Data volume brought up some issues on my drive.

warning: inode (id 230103615): Resource Fork xattr is missing for compressed file
warning: inode (id 230103825): Resource Fork xattr is missing for compressed file
warning: descendants (16969) of dir-stats object (id 264991) is greater than expected (11893)
warning: physical_size (155787264) of dir-stats object (id 587381) is greater than expected (154832896)
warning: physical_size (4349952) of dir-stats object (id 587398) is greater than expected (4075520)
warning: descendants (39) of dir-stats object (id 587432) is greater than expected (37)
warning: descendants (179) of dir-stats object (id 30222494) is greater than expected (138)
warning: descendants (29229) of dir-stats object (id 94676039) is greater than expected (29228)
warning: physical_size (9969881088) of dir-stats object (id 1787190) is greater than expected (380928)
warning: descendants (146790) of dir-stats object (id 5519762) is greater than expected (146789)
warning: physical_size (2103693312) of dir-stats object (id 266529) is greater than expected (86437888)
warning: descendants (2354) of dir-stats object (id 264424) is greater than expected (2253)
warning: physical_size (6385180672) of dir-stats object (id 264424) is greater than expected (777089024)
warning: descendants (291648) of dir-stats object (id 7600991) is greater than expected (291646)
warning: physical_size (11600113664) of dir-stats object (id 587358) is greater than expected (11594612736)
warning: physical_size (52462202880) of dir-stats object (id 262428) is greater than expected (52436328448)
error: doc-id tree: record exists for doc-id 103277, file-id 58461546 but no inode references this doc-id
error: doc-id tree: record exists for doc-id 587753, file-id 215510491 but no inode references this doc-id

First Aid itself succeeds, but issues don’t end up fixed. Some of these I fixed by pressing Alfred app’s settings button to Rebuild macOS Metadata, and deleting .Spotlight-V100. Some errors came from there.

To find out what files belong to what nodes, I used this command:

sudo ls -iR /System/Volumes/Data | grep -B1000 [inodenum]

In the end I wasn’t sure how to deal with these issues, but Backblaze continued working after reinstall.

Here are some resources I found while looking for info on this:

Tags
infrastructure
Date
April 9, 2024
Previously
Rails configuration policies Rails config files tend to get a bit messy, but everywhere I worked they mostly follow a similar set of policies. There are typically 2 kinds of