Modern filesystems, and even storage systems, might have built-in deduplication, but common filesystems still do not. So checking for redundant data and do deduplication when possible might save disk space.

Once up on a a time, there was a system, were we had this 6TB spool of binary files on an production ext4 filesystem, and the volume was running out of disk space. The owner of the data thought it likely that there were duplicates in the vast ammount of files, and wanted to check this up. We checked using fdupes, and yes, there were a lot of duplicates.

Running over the filetree with hardlink, we actually saved 30% of disk space. And could suspend the change of storage solution for some months.

For a testing, let’s make a a tree of directories with variable depth subdirectories and some diverse data:

#!/bin/bash
cd "$(mktemp -d)"
mkdir foo; pushd foo
for n in $(seq 1 100); do
  depth=$((RANDOM%10))
  for i in $(seq 1 $depth); do
    dir=dir$((RANDOM%10))
    mkdir -p $dir;
    pushd $dir;
  done;
  echo $((RANDOM%100)) > file$((RANDOM%100));
  for i in $(seq 1 $depth); do popd; done;
done;
find; echo; ls

Install hardlink. Note that there are different implementations of hardlink for Red Hat and Debian based distributions. While The Red Hat variant of hardlink is faster, the Debian variant has more fine-grained options for ignoring attributes like ownership, file mode and timestamp, and even filter filenames with regular expressions.

The following was tested using the Red Hat variant. On Debian and derivates, replace -c with -pot, or read the hardlink man page.

$ sudo yum install hardlink

Run hardlink on the current directory. It will run for a while. On a large filesystem, it might run for a very long while. Finally, it will show you a list of duplicates.

$ hardlink -c -vv -n .

Note: DO LOOK OVER THE OUTPUT BEFORE DELETING ANY PRODUCTION DATA. You have been warned. If you break something, you keep the parts.

Make a copy just to compare. Then run hardlink without the -n switch

$ cp -a ./ ../bar
$ hardlink -c -v .

Check that the copies are equal in content, though not in disk space

$ popd
$ diff -Naur foo bar && echo They are equal
$ du -s foo bar

To sum up: While a bit cumbersome and time-consuming, it is possible to use quite simple file tools to do deduplication, even on existing filled-up filesystems.

Test and consider thoroughly before using hardlink in production. Changes in the tree while hardlink is running might cause unpredictable results.