How to drag Time Machine to its knees

by Dec 2, 2019

I have, without intending to, found a way to drag Apple’s Time Machine backup system to its knees. I thought it would be amusing to explain what I did, and how I resolved the problem once I realized that my Time Machine backups were effectively never quite finishing.

As I was doing prep work to get ready to move from my iMac to my new Macbook Pro, I became aware that basically ever time I looked, my machine was active in a Time Machine backup. That felt wrong, so I started investigating, and in fact, my machine really was spending almost all of its time with Time Machine active, and most of that time in the “Finishing Backup” state.

My backup setup was pretty normal (or so I thought): the iMac mounting a partition on my Mac Mini where the time machine backup lived. The two machines were connected via ethernet, not Wifi, for maximum performance, so networking shouldn’t have been a problem.

Digging into this, I found each Time Machine backup was taking hours to complete, even though there wasn’t much changing between backups. I started to worry I might have some disk problem I hadn’t noticed going on.

I then noticed that my SuperDuper clone backups were taking a long time, too, and so I started worrying even more about disk problems. Then I noticed that SuperDuper was reporting that it was backing up 2,000,000 files as part of the clone of the boot disk.

Two million files. Say, what?

Yeah, two million files. Digging into this, it turns out that a project I’m working on for work is the “culprit” — because I’m taking advantage of the Homebrew package manager to install various tools I need to work with for this project. This project requires, among other things, Python 3 and Ruby, plus rbenv, imagemagick and others, and within the ruby and python environments, a whole bunch of gems and packages. whee! So when I finally tracked it all down, /usr/local (where brew files live) had over 800,000 files in it just on its own.

Why Time Machine freaks out

Time Machine tries to be smart about not wasting disk space in a backup. When it does a backup, it makes a copy of changed files in the backup area, and then uses hard links for the non-changed files. But for my case, this creates a new problem, since it seems that Time Machine now needs to make TWO MILLION individual requests to create hard links in the backup partition OVER THE NETWORK. No wonder it’s spending all of its time in “finishing backup”.

Fixing this problem

Since it’s easy to regenerate the brew installation if you keep a short list of what you need installed, a way to help solve this problem is to exclude “/usr/local” from your time machine backups. The problem: if you go into “System Preferences->Time Machine->Options” and click on the ‘+’ to add /usr/local to the exclusion list, it doesn’t show up in the file picker. MacOS is helpfully hiding that directory as something you generally don’t need to think about from the GUI. Until now. 

The way to get at it is this: switch to the finder and type in “CMD-SHIFT-G”. This brings up the “Go to Folder” dialog. Type in “/usr/local”, and click “Go”. You should now have the “/usr/local” folder up in a finder window. If you click on the folder at the top of the window (next to the word “local”) you can now drag that folder over and install it in your sidebar. Once its in the sidebar, you’ll see it in the file picker in the Time Machine dialog, and you can select and exclude it from Time Machine. 

I think there’s a deeper problem in Time Machine given the large number of files, since it seems like a backup of 2,000,000 files is taking at least 4X the time one with 1,200,000 files in it does, rather than (one might hope) double the time. But dropping those 700,000 or so files out of the backup brought backup times back to something I considered reasonable. 

I did go one step further those, and moved the backups local again using a 2TB Thunderbolt drive, just to get the network delay of that many requests having to cross the wire. And since I did that, everything’s been rock solid and fast again. 

Much as I like Time Machine, it really looks like it isn’t well set up for cases with really large sets of files in a partition, even though (I think) that scenario is increasingly common these days. I’m not entirely sure how to suggest Apple improve this, but I am worried that somewhere between 1.2 and 2 million files there seems to be a performance drop-off well worse than linear given the growth in number of files. 

To me this is more a curiousity than a bug, but I wanted to put it out here in case someone else is running into this kind of problem and has no idea how to fix it. That tweak involving “CMD-SHIFT-G” is a good one to know, but not necessarily obvious until someone shows it to you…