This should help identify what is going on in Johnny's answer, as well as answer the question of why this works on Linux but not Mac.
The problem lies in the fact that Mac OS X uses bsdtar
, whereas most Linux systems use gnutar
.
You can install gnutar
on a Mac with Homebrew, using brew install gnu-tar
, which will symlink gnutar
into /usr/local/bin
as gtar
.
If you install gnutar
, then you can reproduce the problem using the steps in Johnny's answer.
$ brew install gnu-tar
==> Downloading https://homebrew.bintray.com/bottles/gnu-tar-1.28.yosemite.bottle.2.tar.gz
######################################################################## 100.0%
==> Pouring gnu-tar-1.28.yosemite.bottle.2.tar.gz
==> Caveats
gnu-tar has been installed as "gtar".
If you really need to use it as "tar", you can add a "gnubin" directory
to your PATH from your bashrc like:
PATH="/usr/local/opt/gnu-tar/libexec/gnubin:$PATH"
==> Summary
🍺 /usr/local/Cellar/gnu-tar/1.28: 13 files, 1.6M
$ mkdir test
$ touch test/a test/b
$ gtar -zcvf test.tar.gz test test/a # make the archive with gnutar
test/
test/a
test/b
test/a
$ gtar -ztvf test.tar.gz
drwxr-xr-x adamliter/staff 0 2015-07-28 22:41 test/
-rw-r--r-- adamliter/staff 0 2015-07-28 22:41 test/a
-rw-r--r-- adamliter/staff 0 2015-07-28 22:41 test/b
hrw-r--r-- adamliter/staff 0 2015-07-28 22:41 test/a link to test/a
$ rm -r test
$ tar -xvf test.tar.gz # try to unpack the archive with bsdtar
x test/
x test/a
x test/b
x test/a: Can't create 'test/a'
tar: Error exit delayed from previous errors.
$ echo $?
1
So obviously gnutar
archives things differently in a way that causes bsdtar
to choke on duplicates. The fact that gtar -ztvf test.tar.gz
indciates that the second instance of test/a
is archived as a link to test/a
is relevant. As Johnny points out in the comments, gnutar
will store duplicates as hard links rather than the actual file, which can be disabled with --hard-dereference
.
That is, you could do the following:
$ mkdir test
$ touch test/a test/b
$ gtar -zcvf test.tar.gz test test/a --hard-dereference
test/
test/a
test/b
test/a
$ gtar -ztvf test.tar.gz test
drwxr-xr-x adamliter/staff 0 2015-07-28 23:49 test/
-rw-r--r-- adamliter/staff 0 2015-07-28 23:49 test/a
-rw-r--r-- adamliter/staff 0 2015-07-28 23:49 test/b
-rw-r--r-- adamliter/staff 0 2015-07-28 23:49 test/a # note that this is no longer a link
$ rm -r test
$ tar -xvf test.tar.gz # unpack with bsdtar
x test/
x test/a
x test/b
x test/a
$ echo $?
0
$ ls test/
a b
However, in this case, you obviously don't control the creation of the tarball, so --hard-dereference
is not an option. Luckily, based on the OP's answer, it seems that this problem has been fixed by upstream.
Nonetheless, if anybody else runs into this problem in the future and is in need of a quick fix or has an unresponsive upstream maintainer, there is a workaround.
Once you identify what the duplicate file is, you can use the --fast-read
option of bsdtar
(note that this option is only part of bsdtar
, not gnutar
):
-q (--fast-read)
(x and t mode only) Extract or list only the first archive entry that matches each pattern or filename operand. Exit as soon as each specified pat-
tern or filename has been matched. By default, the archive is always read to the very end, since there can be multiple entries with the same name
and, by convention, later entries overwrite earlier entries. This option is provided as a performance optimization.
So, in the toy example that I've created following the toy example in Johnny's answer, the duplicate file is test/a
. Thus, you could avoid this problem by doing the following:
# this set of commands picks up from the first set of commands
# i.e., the following assumes a tarball that was *not* made with
# the --hard-dereference option, although this will work just as well
# with one that was
$ tar -xvqf test.tar.gz test/a # unarchive the first instance of test/a
x test/a
$ tar -xvf test.tar.gz --exclude test/a # unarchive everything except test/a
x test/
x test/b
$ echo $?
0
$ ls test/
a b
Note, moreover, that gnutar
is perfectly happy to unpack an archive with duplicates that was created by itself, even when the --hard-dereference
option was not used:
$ rm -r test
$ gtar -xvf test.tar.gz
test/
test/a
test/b
test/a
$ echo $?
0
$ ls test/
a b
So this answers your question of why an error is thrown on Mac but not Linux. (Most) Linux distros ship with gnutar
, and since the tarball was presumably packaged with gnutar
, there will be no error when unpacking with gnutar
, but there will be an error when unpacking with bsdtar
.
For further reading and reference, one might want to look at What are the differences between bsdtar and GNU tar? on Unix.SE.
tar
command that comes with OS X.gunzip -c scip-3.2.0.tgz | tar xopf -
from the command line, as you would use it for your script?gunzip
works just fine, but when I try to extract the uncompressed tarball, that's when the error gets thrown.