So yesterday or the day before I blogged about rsync and how good it was. And that’s true but rsync is notoriously tricky and today I became very, very aware of rsync’s trickiness.
So my scenario was that I had an old MacBook Air which I was trying to get all the data off of since DropBox seemed to have an incomplete sync. And I figured that if I just got all the files copied up to my big desktop then I could reformat the machine and give it to my wife as a replacement for her laptop on which the keyboard died.
Note: Some of these things are perhaps known issues / not issues because I’m on OSX. What I can say is that as a Mac user some of these things make no damn sense.
Problem the First - Directory Names with Spaces
So here was the first thing that I observed:
rsync -avh -e ssh Dropbox/"Camera Uploads"/ 192.168.1.25:/Users/sjohnson/backups/macbook_air/Dropbox/"Camera Uploads" Password: building file list ... done created directory /Users/sjohnson/backups/macbook_air/Dropbox/Camera ./ .DS_Store .dropbox 2017-07-03 05.30.42.jpg 2017-07-03 07.03.03.jpg 2017-07-09 08.54.25.jpg 2017-07-09 08.54.31.jpg 2017-07-09 11.09.16.jpg 2017-07-09 11.09.23.jpg 2017-07-10 18.48.37.jpg 2017-07-10 18.48.43.jpg 2017-07-10 18.48.47.mov ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.1/rsync/rsync.c(244) [sender=2.6.9]'
So what you’re seeing is that even tho the data came from “Camera Uploads”, it got moved to “Camera”. The issue is obviously directory names with spaces. And since I had started with an scp approach this meant that rsync was copying up data that it already had in place. ARRGHH!
And here’s a second pass with a different approach to space encoded filenames:
rsync -avh -e ssh Dropbox/"Camera Uploads"/ 192.168.1.25:/Users/sjohnson/backups/macbook_air/Dropbox/Camera\ Uploads/ Password: building file list ... done ./ 2017-07-14 18.09.19.jpg 2017-07-14 18.09.37.jpg 2017-07-14 18.09.39.jpg 2017-07-14 18.28.18.jpg [A bunch of filenames omitted for brevity] 2017-07-30 09.27.13.jpg 2017-07-30 09.27.23.jpg ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.1/rsync/rsync.c(244) [sender=2.6.9]
Before the second pass:
ls -ltr Camera | wc -l 15
After the second pass
ScottiMac:Dropbox sjohnson$ ls -ltr Camera | wc -l 36
So even after changing how I handled the destination filename, it still went to “Camera”. Double ARGH.
So what’s the obvious workaround when rsync seems to entirely fsck up OSX filenames with spaces? I’m sure there are other options but my approach was drop dead simple, rename each folder on source and destination from:
and re-run the sync command:
rsync -avh -e ssh Dropbox/Camera_Uploads/ 192.168.1.25:/Users/sjohnson/backups/macbook_air/Dropbox/Camera_Uploads/ Password: building file list ... done .DS_Store sent 18.87K bytes received 1.64K bytes 8.21K bytes/sec total size is 4.57G speedup is 222553.34
Problem the Second - Excluded Directories
My next issue was that I wanted to exclude the Library directory from the files being synced. This would eliminate all the giant Docker files that are stored in Library (and Library mostly isn’t needed since it is Application level stuff; not really user data).
Here was my first attempt:
➜ rsync -avh --exclude '/Users/sjohnson/Library/' -e ssh ~ 192.168.1.25:/Users/sjohnson/backups/seas Password: building file list ... rsync: opendir "/Users/sjohnson/Library/Application Support/CallHistoryTransactions" failed: Operation not permitted (1)
So that clearly failed. Now I’ve long been aware that Apple has old versions of core open source tools like rsync, so:
➜ rsync --version rsync version 2.6.9 protocol version 29 Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others. <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details.
Clearly the 2006 copyright date is a problem. This calls for brew!
brew install rsync
And, naturally, brew does shell magic trickery that messed up for me. Happily tho rsync is just an executable that I can fully path to (all brew executables always end up in /usr/local/bin):
/usr/local/bin/rsync --version rsync version 3.1.3 protocol version 31 Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details.
and that still failed but here was the magic (and Virginia, yes, the error was all damn mine, not the version of rsync):
“Excluding a specific directory is same as excluding a file, just pass the relative path to the directory to the –exclude option” from Rsync Exclusion
So that made my rsync command this:
rsync -avh --exclude 'Library/' -e ssh ~ 192.168.1.25:/Users/sjohnson/backups/seas
And that worked like a charm. Sigh. Always read the docs well if at all possible.