Have you ever tried to take a snapshot from a mysql server only to find an hour later that the box ran out of disk space when trying to create the tar ball? An easy solution is to use ssh to pipe the tar data directly to another server without it even touching the disk. For those unfamiliar with unix shells or pipes a pipe allows you to tie the output of one program to the input of another. This is most commonly used to manipulate that stream of data. For example if you want to find a specific file in a directory. In these examples cartman is server A (where the commands are ran from). Stan is server B.
cartman> ls | grep mysql-5
The | (pipe) tells the shell to direct the output of ls to the input of grep which looks for the pattern ‘mysql-5′. This same functionality can be used with the tar command. This next command would compress a file and uncompress it on itself. Of course this is pointless. Until we add in ssh
cartman> tar -cf – mysql-5 | tar -xf -
With ssh we can pipe data to another server without it being written to the local disk. If the same users and (uids) exist on both systems tar will even preserve the owner and permissions of the file which is exactly what we want for a mysql snapshot.
cartman> tar -cvf – mysql-5 | ssh stan ‘ tar -xf -’
You can of course add in calls to cd and sudo to preserve user permissions and make sure the snapshot ends up in the correct dir.
cartman> cd /usr/local/mysql; tar -cvf – data | ssh stan ‘cd /usr/local/mysql; sudo tar -xf -’
Just to confuse you this can also be ran the other way. From stan to cartman. Think of this as reaching over to the other server and pulling a file through.
stan> ssh cartman ‘cd store/mysql/; tar -cf – mysql-4.1.15.tar.gz’ | tar -xf -
This can also be used to speed up scp command that involve lots of small files. Tar will produce a stream of output which can be sent across the network in much larger blocks than the native scp protocol which will send a small packet for each file if the file is small enough. The linux command line contains hundreds of different tools that can be combined in millions (billions? trillions?) of different ways to easily accomplish tasks. I have been using linux for years and am still learning new shell tricks almost daily. If you’re an old school shell guy please add a comment with your favorite shell tricks.