This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

I come back to a specific problem every once in a while: Changing a program’s file descriptor while the same program is running.

We do stupid things

From time to time, we do stupid things, like running a very important shell command, and adding debug output to see that it works, then realizing that it will take hours to finish, and spitting gigabytes of debug to an xterm through SSH does not help. The dream is of course to just redirect that output to stdout in a screen process. Or just put it in /dev/null.

Other people do stupid things

An other typical example could be a finding an ill managed system with some daemon without proper log-file handling. Restarting that process right now is just out of the question, copy-truncating that 16GB log-file will take too much time, and by the way, the disk is almost full.

The dark side of gdb

Is it possible to just move the file descriptor for a running process? Yes it is. Welcome to the dark side of gdb.

With the power of gdb at your hand, you can hook into the inner parts of any running program, and change, well, virtually anything. Sounds insanely dangerous for systems in production, right? This hack is not that ugly. It does just what the doctor ordered: It changes a process’ fds for you while it’s running.

The hack: Google helps always

This script is based on a variant written by Robert McKay, and is one of the many versions that Google may find for you.

Basic usage:

fdswap /var/log/mydaemon/output.log /dev/null 1234
#!/bin/bash
#
# fdswap
#
if [ "$2" = "" ]; then
  echo "
    Usage: $0 /path/to/oldfile /path/to/newfile [pids]
    Example: $0 /var/log/daemon.log /var/log/newvolume/daemon.log 1234
    Example: $0 /dev/pts/53 /dev/null 2345"; exit 0
fi

if gdb --version > /dev/null 2>&1; then true
else echo "Unable to find gdb."; exit 1
fi

src="$1"; dst="$2"; shift; shift
pids=$*

for pid in ${pids:=$( /sbin/fuser $src | cut -d ':' -f 2 )};
do
    echo "src=$src, dst=$dst"
    echo "$src has $pid using it"
    (
        echo "attach $pid"
        echo 'call open("'$dst'", 66, 0666)'
        for ufd in $(LANG=C ls -l /proc/$pid/fd | \
        grep "$src"\$ | awk ' { print $9; } ');
       do echo 'call dup2($1,'"$ufd"')'; done
        echo 'call close($1)'
        echo 'detach'; echo 'quit'
        sleep 5
    ) | gdb -q -x -
done

Test your stuff before destroying everything

Do test this before smashing your production environment to pieces. The following uses non-password access to root, so make sure you can do

sudo whoami

and get root as an answer, without having to enter a password

Make small shell script like this:

#!/bin/bash
echo "This is the pid: $$"
echo "This is stdout:"; sudo ls -l /proc/$$/fd/0;
n=0; while true; do ((n++)); echo $n; sleep 1; done

Run that script in a separate window. Note the PID, and which VTY is connected to that process’ stdout

$ bash foo.sh &
This is the pid: 28073
This is stdout:
lrwx------. 1 ingvar ingvar 64 Nov 18 11:53 /proc/28073/fd/0 -> /dev/pts/9
1
2
3
4

So the script runs with PID 28073, and is pumping numbers to /dev/pts/9. Now start screen in another window, and find its VTY for stdout

$ screen
[screen]$ sudo ls -l /proc/$$/fd/0
lrwx------. 1 ingvar ingvar 64 Nov 18 11:54 /proc/28094/fd/0 -> /dev/pts/6

So, the screen process uses /dev/pts/6. We want to move the output from our foo.sh script to inside the screen

Run the fdswap, let all that gdb output scare you, but watch that number pumping magically stop. You need admin rights (sudo) unless you own all processes yourself.

sudo fdswap /dev/pts/9 /dev/pts/6 28073

Look inside the screen again. The numbers are now being pumped there.

Ingvar Hagelund

Team Lead, Application Management for Media at Redpill Linpro

Ingvar has been a system administrator at Redpill Linpro for more than 20 years. He is also a long time contributor to the Fedora and EPEL projects.

Containerized Development Environment

Do you spend days or weeks setting up your development environment just the way you like it when you get a new computer? Is your home directory a mess of dotfiles and metadata that you’re reluctant to clean up just in case they do something useful? Do you avoid trying new versions of software because of the effort to roll back software and settings if the new version doesn’t work?

Take control over your local development environment with containerization and Dev-Env-as-Code!

... [continue reading]

Ansible-runner

Published on February 27, 2024

Portable Java shell scripts with Java 21

Published on February 21, 2024