If you are a webdeveloper using linux, you probably know this exhausting procedure:
You change a file, save it, and then you have to upload the changed file to the matching folder on your test web server in your favourite ftp program, i.e. FileZilla. You do this hundred times a day, thousand times a week.
On Windows, there is a program called WinSCP, a FTP client, that gets rid of this time consuming practice by offering an autosynchronization function for a specific folder after saving/modifying a file in this folder.
To immediately scale down your expectations: there is no linux GUI equivalent of WinSCP that has its built-in auto sync feature. But we have the shell. And we have built-in tools like rsync. And we have people like Gregg Hernandez, who create nifty little tools like watcher.
In this article I’ll show you how you can combine these two tools to get a powerful auto synchronization setup in less than 5 minutes.
Requirement: ssh connection without password
In order to upload files and directories from your local machine to a remote machine, you must be able to connect to that remote machine without being asked for a password during the connection attempt. So you have to use a public ssh key to grant proper identification of your machine. Here’s a quick HowTo:
ssh-keygen -t rsa
# hit 3 times enter to use default values
# copy your fresh key to the remote machine:
ssh-copy-id -i ~/.ssh/id\_rsa.pub user@remote.machine
# that's the last time you have to enter your password
That’s it. Now connect to the remote machine by entering
ssh user@remote.machine
to verify that it works. If the connection is made without requesting your password, you’re ready for the next step.
Monitoring files and folders for changes with watcher.py
watcher.py is a daemon written in python, that monitors selectable files and/or whole folders for changes. If there is a change in a monitored directory, watcher can trigger custom shell commands defined for that event.
- you can monitor multiple folders
- you can trigger multiple commands at once (i.e. upload a file to multiple servers)
- you can finetune the type of events watcher should listen to, i.e.
- writing a file
- file accessing
- file creation
Install watcher.py
The installation of watcher is simple. You can either download a zip-file of the program and extract it, or you can clone the git repository. As the ladder is my favourite because of it’s ease of updating, I’ll explain it in short.
First you need git. On debian based distributions (ubuntu, linux mint,…) open a terminal and type
sudo apt-get install git
You also need some python stuff. Enter
sudo apt-get install python python-pyinotify python-yaml
After git and python are installed, you can clone the watcher git repository by typing
# optional: make a folder called "tools" in your home directory
mkdir ~/tools/;cd ~/tools
# clone the repository
git clone https://github.com/greggoryhz/Watcher.git
Now you are ready to run watcher on your machine.
First run
When you run watcher.py for the first time, it will
- generate a .watcher dir in your home dir
- generate a jobs.yml in this ~/.watcher dir
In order to start watcher, you have to type in your shell:
~/tools/Watcher/watcher.py start
Note: these commands are pretty long, but we’ll get rid of them later.
Now you can stop watcher by entering:
~/tools/Watcher/watcher.py stop
After watcher stopped, you can create your first monitoring job:
Tell watcher which folder to observe
On startup watcher looks in the ~/.watcher/jobs.yml for monitoring jobs.
To create your first job, you have to open this file with an editor like vim or gedit.
gedit ~/.watcher/jobs.yml
There you should see an initial job dummy like following:
job1:
# a generic label for a job. Currently not used make it whatever you want
label: Watch /home/yourUser/path/to/the/watched/folder for added or removed files
# folder or file to watch. Probably should be abs path.
watch: /home/yourUser/path/to/the/watched/folder
# folder or files to exclude
exclude: \[\]
events: \['create', 'move\_from', 'move\_to', 'delete', 'modify'\]
options: \[\]
# if true, watcher will monitor folders recursively for changes
recursive: true
command: echo $datetime $filename $tflags # $src\_path
Note: I have removed most of the comments to preserve readability. The file is way better documented than most code that I’ve seen.
The most important parameter for this guide is the last one, ‘command’. As you can see, you can specify one or more shell commands to be triggered, if an event from the ‘events’ parameter in the watched folder happens.
Because we want to upload stuff with our little auto sync setup, we use a program called rsync.
The command to execute a synchronization between your local machine and a remote machine is:
command: rsync -rtv –exclude=“.*” /home/yourUser/path/to/the/watched/folder user@remotemachine:/home/aUser/path/to/remote/folder
Replace the last line (starting with ‘command: …’) with this line and you’re ready to auto sync 🙂
Getting rid of the long start and stop commands and enable logging
Typing “~/tools/Watcher/watcher.py start” everytime can be tediously, what about just typing “wstart” ? Let’s add a bash alias to accomplish that:
# open your ~/.bashrc with an editor like gedit by typing
gedit ~/.bashrc
# go to the last line and add following lines:
alias wstart='~/tools/Watcher/watcher.py start; tail -f ~/.watcher/watcher.log'
alias wstop='~/tools/Watcher/watcher.py stop'
What happened?
First we added an alias for starting watcher.py from anywhere in our terminal by simply typing ‘wstart’. Additionally we extended this alias with a log displaying feature, so you’re able to see what watcher uploads. Handy!
The last line defines an alias for stopping the watcher daemon by simply typing ‘wstop’ in the terminal.
Note: in order to use these aliases you have to write ‘source ~/.bashrc’ in your active terminal once or reopen it.
Phew. This was my first post in English, so please forgive me my lack of stylistic sense.
If you enjoyed this post, found any errors or have any questions, feel free to leave your feedback in the comments.
And, of course, the obligatory call to subscribe to my blog – either by email or by rss feed.