Errol: XMPP Automatic file sender

Errol is a file sender that rely on inotify. It can be used to watch a directory and automatically transfers the new files (or modified ones) with XMPP.

The origins

Errol find its origin in tasks I am doing for a small association, "Les compagnons du CEP", a joint buying organization who sells wines. I manage their ERP, the excellent Odoo. One of these tasks is the generation of their price list from an Excel spreadsheet (yeah, I know). I designed this process with a LaTeX generator written in python because I am fluent with it since 10 years. As I did not want to install a LaTeX distribution on the production server, the logical decision was to delocalize this task on another machine. The user uploads his excel file on a webpage, the file is saved in a "watched" directory and its transfer is triggered on the second machine with XMPP. The generation of the latex document and its compilation is performed with LaTeX and the resulting PDF is sent back to the server. The PDF is therefore available for download.

Why Errol?

In the fictional universe of Harry Potter, Errol is the Weasley family's owl. It is quite old and awkward. One could says the same about XMPP but Errol is quite useful, XMPP is too :-). Errol is a great grey owl. (see pictures)

photo credit: blurred.ca https://www.flickr.com/photos/blurredca/10527590684/ Photo credit: blurred.ca, Great Grey Owl

Howto

Prerequisites

Errol needs the following requirements:

  • A system supporting inotify (Linux).
  • an XMPP (jabber) account supporting the following XEPs: Stream Management, Publish-Subscribe, Multi-User Chat
  • A PubSub service where the nodes can be set as open. The node name is defined in the configuration file. I personally use sat_pubsub. A PubSub component developed for the project Salut à Toi.
  • A Multi-User Chat because not all XMPP accounts support PubSub. For now, some information are still shared through MUC messages. This behavior could change in the future.
  • The latest (dev) version of Slixmpp.

You can use your own XMPP server or choose a XMPP service among the following list.

Create the PubSub node

This step is optional if you already have a write access on the pubsub node. The following example use jp, the Salut à  Toi command-line interface but slixmpp or sleekxmpp can be used. 

$ jp pubsub node create -f publish_model open be.agayon.errol:0 -s pubsub.agayon.be -c

The node name be.agayon.errol:0 is recommended in order to identify the functionality.

As an example, there are the node options on the service pubsub.agayon.be:

$ jp pubsub node info be.agayon.errol:0 -s pubsub.agayon.be
persist_items: True
deliver_payloads: True
serial_ids: False
publish_model: open
access_model: open
send_last_published_item: on_sub

If your server supports Personal Eventing Protocol(PEP) or if you do not want to use the generic PubSub service of your server, you can use your jid.

$ jp pubsub node create -f publish_model open be.agayon.errol:0 -s info@agayon.be -c

Tests

You can test your setup with the examples scripts of slixmpp.

Example:

./s5b_file_sender.py -j jid@example.org -p pass -r john@example.org -f /path/to/file.txt

See the scripts for more information.

Getting started with Errol

Errol needs the following dependencies:

Installing

You can easily install errol with pip:

$ pip install errol

Note: errol can be installed in a virtualenv.

You can also clone the git repository:

$ git clone https://gitlab.com/jnanar/errol.git
$ cd errol
$ python3 setup.py install

On Archlinux:

A PKGBUILD will be available soon.

Configuration

You need to provide information about the XMPP account.

$ cat config.example.ini

[XMPP]
pubsub=pubsub.example.org
node=be.agayon.errol:0
room=chat@chat.example.org
jid=jid@example.org/errol
password=pass
ressource_receiver=-receiver
ressource_sender=-
nick_sender=example_sender
nick_receiver=example_receiver
receiver=jid@example.org/errol-receiver
  • jid : the jabber account
  • password: the xmpp password
  • pubsub: the PubSub server (publish activity)
  • room: the MUC (chatroom) where errol display information.

The files will be sent by jid@example.org/errol-0 and received by jid@example.org/errol-receiver . The nicks are the usernames used on the MUC.

Flying

Photo credit: Wisconsin Department of Natural Resources https://www.flickr.com/photos/widnr/ Photo credit: Wisconsin Department of Natural Resources, Great Grey Owl at Mauston

Once installed, Errol can be launched in a terminal.

$ errol --help
usage: errol [-h] [-e EVENTS] [-f FILE] [-d] -p PATH -c COMMAND

Automatic XMPP file sender and directory watcher

optional arguments:
  -h, --help            show this help message and exit
  -e EVENTS, --events EVENTS
                        Number of events to watch (delete, create modify) in
                        the directory. Once reached, the program stops.
  -f FILE, --file FILE  Config file containing XMPP parameters
  -d, --debug           set logging to DEBUG
  -p PATH, --path PATH  The path watched.
  -c COMMAND, --command COMMAND
                        The executed command: xmpp or watcher

In Hogwarts

If you want to watch the directory /tmp/sender, the following command can be used:

$ errol -f config.example.ini -p /tmp/sender -c watcher

All modified or new files created in the watched location will be sent by XMPP.

In Azkaban

If you want to receive the files, you have to launch Errol with the following command line.

$ errol -f config.example.ini -p /tmp/receiver -c xmpp

All the received files will be stored in the directory defined with the option '-p'.

License

This project is licensed under the GPLv3 - see the LICENSE.txt file for details

Why not X or Y?

Photo credit: Bernard Spragg. NZ https://www.flickr.com/photos/volvob12b/37310719232/ Photo credit: Bernard Spragg. NZ, Great Grey Owl (Strix nebulosa)

There are plenty solutions for this kind of needs. Some of them are more mature. I choose XMPP for several reasons:

  • agayon.be already provides a up-to-date XMPP server with all the needed XEPs enabled.
  • I do not want to open additional port on the client that performs the LaTeX compilation.
  • I wanted to learn to work with XMPP for machine to machine communications and use PubSub notifications (because why not?).

Among the alternatives, I could have build the service on top of:

  • sockets
  • HTTP file transfer
  • REST API
  • SSH and remote commands.
  • rsync
  • ...

Acknowledgments

Photo credit: lasta29, Great grey owl, Osaka Tennoji Zoo https://www.flickr.com/photos/115391424@N05/36873334554/ Photo credit: lasta29, Great grey owl, Osaka Tennoji Zoo

  • The french XMPP community (availaible on sat@chat.jabberfr.org, jabberfr@chat.jabberfr.org, ...)
  • The Slixmpp maintainers Florent Le Coz, Mathieu Pasquet for their nice library.
  • Emmanuel Gil Peyrot (Link mauve) for its reactivity.
  • Goffi from the Salut à Toi project for his explanations and his disponibility.

Links

links

social