This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite
Modern file systems, and even storage systems, might have built-in deduplication, but common file systems still do not. So checking for redundant data and do deduplication when possible might save disk space.
Once up on a a time, there was a system, were we had this 6TB spool of
binary files on an production ext4 file-system, and the volume was
running out of disk space. The owner of the data thought it likely
that there were duplicates in the vast amount of files, and wanted to
check this up. We checked using fdupes
, and yes, there were a lot of
duplicates.
Running over the file-tree with hardlink, we actually saved 30% of disk space. And could suspend the change of storage solution for some months.
For a testing, let’s make a a tree of directories with variable depth sub-directories and some diverse data:
#!/bin/bash
cd "$(mktemp -d)"
mkdir foo; pushd foo
for n in $(seq 1 100); do
depth=$((RANDOM%10))
for i in $(seq 1 $depth); do
dir=dir$((RANDOM%10))
mkdir -p $dir;
pushd $dir;
done;
echo $((RANDOM%100)) > file$((RANDOM%100));
for i in $(seq 1 $depth); do popd; done;
done;
find; echo; ls
Install hardlink. Note that there are different implementations of hardlink for Red Hat and Debian based distributions. While The Red Hat variant of hardlink is faster, the Debian variant has more fine-grained options for ignoring attributes like ownership, file mode and timestamp, and even filter filenames with regular expressions.
The following was tested using the Red Hat variant. On Debian and derivatives, replace -c with -pot, or read the hardlink man page.
sudo yum install hardlink
Run hardlink on the current directory. It will run for a while. On a large file-system, it might run for a very long while. Finally, it will show you a list of duplicates.
hardlink -c -vv -n .
Note: DO LOOK OVER THE OUTPUT BEFORE DELETING ANY PRODUCTION DATA. You have been warned. If you break something, you keep the parts.
Make a copy just to compare. Then run hardlink without the -n switch
cp -a ./ ../bar
hardlink -c -v .
Check that the copies are equal in content, though not in disk space
popd
diff -Naur foo bar && echo They are equal
du -s foo bar
To sum up: While a bit cumbersome and time-consuming, it is possible to use quite simple file tools to do deduplication, even on existing filled-up file systems.
Test and consider thoroughly before using hardlink in production. Changes in the tree while hardlink is running might cause unpredictable results.
The irony of insecure security software
It can probably be understood from my previous blog post that if it was up to me, I’d avoid products like CrowdStrike - but every now and then I still have to install something like that. It’s not the idea of “security software” per se that I’m against, it’s the actual implementation of many of those products. This post lists up some properties that should be fulfilled for me to happy to install such a product.