tech stuff.

Sometime around duplicity 0.6.15 (ish) I started running into OSError exceptions that I just didn’t have time to track down. I’ve finally made time, though, and it wasn’t too hard to track down the culprit. I didn’t realize it at the time, but this only affects non-privileged users running duplicity. tl;dr choose a different TMPDIR.

First, a statement of the problem:
% duplicity -vw --no-print-statistics --full-if-older-than 1M --include-globbing-filelist /home/user/.backups.list --exclude /home/user /home/user file:///home/user/.backups
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 1394, in
with_tempdir(main)
File "/usr/local/bin/duplicity", line 1387, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 1263, in main
sync_archive(decrypt)
File "/usr/local/bin/duplicity", line 1060, in sync_archive
copy_to_local(fn)
File "/usr/local/bin/duplicity", line 1009, in copy_to_local
tdp.move(globals.archive_dir.append(loc_name))
File "/usr/local/lib/python2.7/site-packages/duplicity/path.py", line 617, in move
self.copy(new_path)
File "/usr/local/lib/python2.7/site-packages/duplicity/path.py", line 443, in copy
self.copy_attribs(other)
File "/usr/local/lib/python2.7/site-packages/duplicity/path.py", line 448, in copy_attribs
util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))
File "/usr/local/lib/python2.7/site-packages/duplicity/util.py", line 65, in maybe_ignore_errors
return fn()
File "/usr/local/lib/python2.7/site-packages/duplicity/path.py", line 448, in
util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))
OSError: [Errno 1] Operation not permitted: '/home/user/.cache/duplicity/07adf1b6a7475a5bd7cc218f415304f8/duplicity-full-signatures.20121010T000001Z.sigtar.gz'

Why would a chown() be failing?  It turns out it’s because of the BSD semantics for new file creation in which a new file inherits the GID of the directory in which it was created. e.g.

[user@host]:~% cd /tmp
[user@host]:/tmp% ls -ld
drwxrwxrwt 7 root wheel 512 Nov 13 01:18 .
[user@host]:/tmp% id
uid=1002(user) gid=1000(group) groups=1000(group)
[user@host]:/tmp% touch look_at_my_gid && ls -lh look_at_my_gid
-rw-r--r-- 1 user wheel 0B Nov 13 01:19 look_at_my_gid

Duplicity attempts to explicitly copy file attributes when importing a file from the temporary directory to duplicity’s cache directory. I’m a little unclear of the utility of this feature unless the same code path is used to copy files being backed up and restored. There’s no way to disable that behavior, but a decent work around is to use a tmp directory with a GID matching that of the user running duplicity. Since duplicity is using tempfile this can be accomplished by modifying the environment, but you could also use the –tempdir argument. I’ve added the following to all of my duplicity scripts:

export TMPDIR=/home/user/.cache/tmp

Bug Report