In bash, we often use redirects (that is < and > ) to get output from a command to a file, or input from a file to a command. But sometimes, commands takes two or more files as input. Then our ordinary scheme does not work anymore.
Let’s say you want to diff(1) the output of two commands. For example, compare the contents of two directories. ... [continue reading]
Amazon S3 has been around for a while, and it has become increasingly popular to use S3 or S3-like solutions as an object store. In many cases S3 replaces NFS as the chosen type of file system.
And with good reason. Separating application instance from application state is almost always a good decision. And by changing from an architecture that requires low-level access to the host running the application to using a REST interface, we can now deploy the application ... [continue reading]
There will come a time when you will find yourself asking “What the heck is that process doing?”. To uncover the mysteries behind the behaviour of a process, we have a tool called strace.
The program strace is very handy when you want to debug the execution of a program. It catches and states all the system calls performed called by a process. It will also catch and state any inter-process signals received by this process.
So you are happily working in your shell issuing commands with merry abandon. At some point you typing in a long command, but find yourself in the wrong directory. Do you hit CTRL-c, then change directory for then to type in the entire command again?
Enter kill and yank
First some basics, CTRL-a will move the cursor to the start of the command line while CTRL-e will move the cursor ... [continue reading]
Now and again, there is a need to override certain DNS lookups. For individual systems you could make changes in the hosts file, but this does not scale well and is easily forgotten. Establishing the override through DNS, one could either use split-view DNS or the old-school strategy of creating a stub zone for that exact hostname (or domain, if you were lucky), but this does not scale very well either and is, let’s admit it, rather ugly.
That said, ... [continue reading]
I tend to dislike scattering my file-systems with non-packaged script modules, be it Python, Perl, or Ruby. So I wrap rpm packages for my systems. But packing is such a bother, you might say. That might be true, but luckily, we have good tools to help us.
To build this website, we used Jekyll, an excellent small ruby HTTP server for static content. Jekyll is available as a standard Ruby gem, but not as rpms ... [continue reading]
Migrating disk volumes to new storage solutions can be a bit of a chore. Cloning file systems are usually done after taking your system offline in order to preserve consistency. But every so often you simply can’t afford to have your data or system unavailable for that period of time. So what to do?
We had invested in new SAN storage solution and had to move the disk volumes for our virtual machines from the old SAN ... [continue reading]
pv is a nice little tool that will take stdin and make a nice little progress bar and displaying time elapsed, percentage completed (if size is known) current throughput rate and the total data transferred and with an ETA.
So if you have a tool that can output it’s result on standard output or read from standard input, one can get a progress bar!
It’s nice to know that “data is flowing”.
... [continue reading]
I come back to a specific problem every once in a while: Changing a program’s file descriptor while the same program is running.
From time to time, we do stupid things, like running a very important shell command, and adding debug output to see that it works, then realizing that it will take hours to finish, and spitting gigabytes of debug to an xterm through SSH does not help. The dream is of course to ... [continue reading]
Rsync is a versatile tool for copying files or whole directory trees locally or over the network to another computer. rsync -a will copy files and ensure most of attributes, ownership and permissions are carried over. Rsync over network is trivial, with SSH as the default network protocol (Rsync can also run in daemon mode).
Often I need to do a rsync -a between hosts on directories I’m not the ... [continue reading]
