1

While trying to remove duplicates of MRI readings I am tarring / compressing (tgz) the top-level directory with a mix of executables, pdf, text, dll, and data in proprietary format (but sometimes with different "last modified" dates). Any tgz files with same byte-size, I consider duplicate MRI scans. I am comparing 10+ MRI datasets that have been stored in a variety of compression formats in local and cloud drives for 10+ years. I'm sure some are duplicates.

tar cfa mri01.tgz MRI01

tar cfa mri02.tgz MRI02

For datasets that are, before tgz compression, typically about 615-mb with about 135 files, I sometimes see no size difference, a size difference of only 150-bytes or so, and significant size differences in the tgz files. I don't know what to think.

Could a 150-byte size difference between the tgz of such large datasets be caused just by file meta-data, like "last modified date"? Or, does such a tiny size difference indicate these are different MRI scans? Is there a better way to detect duplicates of this type of data?

Improve this question
0

1 Answer 1

Reset to default
1

does such a tiny size difference indicate these are different MRI scans? Is there a better way to detect duplicates of this type of data?

A tiny size difference could mean they are two completely different files or the exact same content (duplicate) with a minor metadata difference. A lot of medical imaging software will create images of fixed size (say 500MB, for example). Depending on the length of the scan, you may have 10 files in one case or 50 in another. But, each being 500MB, they'd all the be the same size and all completely different.

The size of the file shouldn't be use to determine whether or not files differ. It might be a good indicator that something is different.

Instead use diff

diff foo1.bar foo2.bar
Binary files foo1.bar and boo2.bar differ

There's also the cmp command:

cmp foo1.bar foo2.bar
foo1.bar and foo2.bar differ: char 1, line 1
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .