Journal of a Bothttps://blog.agayon.be/2024-02-04T12:00:00+01:00AgayonTodo2024-02-04T12:00:00+01:002024-02-04T12:00:00+01:00Arnaudtag:blog.agayon.be,2024-02-04:/agayontodo.html
<p>For years, I have used <a href="https://www.mytinytodo.net/">myTinyTodo</a> to manage my to-do lists. It is super light, fast and easy to use.
At some point, I struggled to update it to use my up-to-date version of PHP. Last year, I wanted to explore other languages than Python. I decided to start a small project to replace myTinyTodo. As I wanted to explore <a href="https://go.dev/">Go</a> and <a href="https://react.dev/">React</a>, I created two projects to build my to-do list system.</p>
<p>The project fulfills my needs for now. The IU is not perfect, and it lacks a lot of functionalities of MyTinyTodo but I don't really need them for now.</p>
<h1>Development</h1>
<p>I named the project AgayonTodo. The repositories can be found here:</p>
<ul>
<li><a href="https://gitlab.com/jnanar/agayontodo_js">Javascript frontend</a></li>
<li><a href="https://gitlab.com/jnanar/agayontodo">Go backend</a></li>
</ul>
<p>The Javascript frontend relies on React and calls the API to display the data. I used the <a href="https://github.com/facebook/create-react-app/"> create-react-app</a> tool to build the whole thing and followed the main tutorial. Even if I had to download the whole internet of dependencies, it is pretty efficient.
I used the <a href="https://bulma.io/">Bulma</a> CSS Framework because I wanted to try something else than Bootstrap. It can be used without Javascript, which could be helpful in some of my other projects.</p>
<p>The Go backend service relies on <a href="https://gorm.io/index.html">gorm</a>, an ORM in Go. The service only provides a small <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD API</a> and some small tools because I don't need much at the moment. I did not want to do all the SQL requests by myself, especially if I want to add <a href="https://en.wikipedia.org/wiki/Access-control_list">access control</a> in the future. I thought it would be better to learn how to use that popular library.</p>
<p>For years, I have used <a href="https://www.mytinytodo.net/">myTinyTodo</a> to manage my to-do lists. It is super light, fast and easy to use.
At some point, I struggled to update it to use my up-to-date version of PHP. Last year, I wanted to explore other languages than Python. I decided to start a small project to replace myTinyTodo. As I wanted to explore <a href="https://go.dev/">Go</a> and <a href="https://react.dev/">React</a>, I created two projects to build my to-do list system.</p>
<p>The project fulfills my needs for now. The IU is not perfect, and it lacks a lot of functionalities of MyTinyTodo but I don't really need them for now.</p>
<h1>Development</h1>
<p>I named the project AgayonTodo. The repositories can be found here:</p>
<ul>
<li><a href="https://gitlab.com/jnanar/agayontodo_js">Javascript frontend</a></li>
<li><a href="https://gitlab.com/jnanar/agayontodo">Go backend</a></li>
</ul>
<p>The Javascript frontend relies on React and calls the API to display the data. I used the <a href="https://github.com/facebook/create-react-app/"> create-react-app</a> tool to build the whole thing and followed the main tutorial. Even if I had to download the whole internet of dependencies, it is pretty efficient.
I used the <a href="https://bulma.io/">Bulma</a> CSS Framework because I wanted to try something else than Bootstrap. It can be used without Javascript, which could be helpful in some of my other projects.</p>
<p>The Go backend service relies on <a href="https://gorm.io/index.html">gorm</a>, an ORM in Go. The service only provides a small <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD API</a> and some small tools because I don't need much at the moment. I did not want to do all the SQL requests by myself, especially if I want to add <a href="https://en.wikipedia.org/wiki/Access-control_list">access control</a> in the future. I thought it would be better to learn how to use that popular library.</p>
<h1>Production</h1>
<p>The frontend is built and copied in a nginx virtual host and the backend static executable is run with a systemd unit. The system is basically launched like that. Running the frontend and the backend in the same virtual host prevents dealing with CORS issues.</p>
<div class="highlight"><pre><span></span><code><span class="n">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">listen</span><span class="w"> </span><span class="o">*</span><span class="p">:</span><span class="mi">443</span><span class="w"> </span><span class="n">ssl</span><span class="p">;</span>
<span class="w"> </span><span class="n">listen</span><span class="w"> </span><span class="p">[::]:</span><span class="mi">443</span><span class="w"> </span><span class="n">ssl</span><span class="w"> </span><span class="n">http2</span><span class="p">;</span>
<span class="w"> </span><span class="n">server_name</span><span class="w"> </span><span class="n">example</span><span class="o">.</span><span class="n">org</span><span class="p">;</span>
<span class="w"> </span><span class="n">ssl_certificate</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">example</span><span class="o">.</span><span class="n">pem</span><span class="p">;</span>
<span class="w"> </span><span class="n">ssl_certificate_key</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">example</span><span class="o">.</span><span class="n">key</span><span class="p">;</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">robots</span><span class="o">.</span><span class="n">txt</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">add_header</span><span class="w"> </span><span class="n">Content</span><span class="o">-</span><span class="n">Type</span><span class="w"> </span><span class="n">text</span><span class="o">/</span><span class="n">plain</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">200</span><span class="w"> </span><span class="s2">"User-agent: *</span><span class="se">\n</span><span class="s2">Disallow: /</span><span class="se">\n</span><span class="s2">"</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">error_log</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">error</span><span class="o">.</span><span class="n">log</span><span class="p">;</span>
<span class="w"> </span><span class="n">include</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">global</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">.</span><span class="n">conf</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="o">/</span><span class="n">srv</span><span class="o">/</span><span class="n">http</span><span class="o">/</span><span class="n">example</span><span class="o">.</span><span class="n">org</span><span class="p">;</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">auth_basic</span><span class="w"> </span><span class="s2">"Authorized users only"</span><span class="p">;</span>
<span class="w"> </span><span class="n">auth_basic_user_file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">security</span><span class="o">.</span><span class="n">txt</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">/</span><span class="n">api</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">auth_basic</span><span class="w"> </span><span class="s2">"Authorized users only"</span><span class="p">;</span>
<span class="w"> </span><span class="n">auth_basic_user_file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">security</span><span class="o">.</span><span class="n">txt</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_pass</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">8080</span><span class="o">/</span><span class="n">api</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h1>Captures</h1>
<p><img alt="Capture of AgayonTodo" src="https://blog.agayon.be/images/agayontodo/capture.png.webp" style="width: 1468px; height: auto; max-width: 100%;"/>
<img alt="Capture 3 of AgayonTodo" src="https://blog.agayon.be/images/agayontodo/capture3.png.webp" style="width: 846px; height: auto; max-width: 100%;"/></p>
<h1>Links</h1>
<ul>
<li><a href="https://www.mytinytodo.net/">myTinyTodo</a></li>
</ul>2023 Summary2023-12-26T19:00:00+01:002023-12-26T19:00:00+01:00Arnaudtag:blog.agayon.be,2023-12-26:/news_2023.html
<p>First I would like to wish all the readers (if there is any), a merry Christmas and a nice happy end of year.</p>
<p>The time flew since the last article. As I was more busy with my other hobby, improv theater, I had less time to hack stuff and build new things. Nevertheless, I will summarize here what I did in the past months. </p>
<h1>My First Thinkpad T</h1>
<p>First, I bought a refurbished <a href="https://www.lenovo.com/ee/et/laptops/thinkpad/t-series/ThinkPad-T480s/p/22TP2TT480S">Levovo Thinkpad T480s</a>. It is the best machine I've had. For around €300 it is really a great purchase from my point of view. I had already purchased a second-hand computer for some relatives at <a href="https://media-monster.be/en">Media Monster</a>, and I wanted to buy one myself to be more autonomous.</p>
<p>I am not a gamer and I don't have big needs. I just upgraded the RAM to 24Go in total, which is sufficient for my needs. After checking that everything was alright with Windows 11 installed by default, I formatted everything and <a href="https://support.lenovo.com/us/en/downloads/ds502226-bios-update-utility-bootable-cd-for-windows-10-64-bit-linux-thinkpad-t480s">updated the UFI Bios</a>. Then I installed
<a href="https://archlinux.org/">Arch Linux</a> with ext4 partition in an encrypted LVM. </p>
<h1>Sway</h1>
<p>For 15 years, I have been using <a href="https://www.enlightenment.org/">Enlightenment</a> as my primary DM. I loved it, especially the ability to change from one virtual desktop to another by moving the mouse outside the screen. With the arrival of this new computer, knowing that I would have a 14' screen, I wanted to be more efficient and to use the keyboard as much as possible. Therefore, I decided to try a modern tiling window manager. In the past, I already tried the tiling mode of Enlightenment, but I did not liked it in the long run. This time I installed <a href="https://swaywm.org/">Sway</a>, a tiling WM compatible with <a href="https://i3wm.org/">i3</a>. I love it. Even if at beginning, it was a little bit difficult to remember all the shortcuts. I feel really efficient, and I like to keep my hands on the super comfortable keyboard on the Thinkpad.
I also installed <a href="https://codeberg.org/dnkl/foot">Foot</a> as my primary <a href="https://en.wikipedia.org/wiki/Terminal_emulator">terminal emulator</a>.</p>
<h1>Neovim</h1>
<p>I also took the opportunity to start to use <a href="https://neovim.io/">Neovim</a> as my primary text editor. I wanted to try it for a really long time but never took the time for it. I recently discovered the <a href="https://neovim.io/doc/user/pi_tutor.html#%3ATutor">:Tutor mode of Neovim (vimtutor)</a> and adopted it as my primary editor. For now, I use it quite basically instead of <a href="https://www.nano-editor.org/">Nano</a> but I feel it will really improve my productivity and speed in the future.</p>
<h1>Cleanup of this blog</h1>
<p>This year, I also took the time to improve the performance of this blog. With time, and the growth of article, I observed that the blog was slow to load. The main issue was the use of non-optimized pictures or videos. I decided to use the <a href="https://developer.chrome.com/docs/lighthouse/overview">Lighthouse</a> tool natively available in Chrome/Chromium and follow the diagnostic help to improve the blog. At first, I was afraid it was not possible, and I would have to move the blog to <a href="https://gohugo.io/">Hugo</a>, another static site generator. Even if Hugo is more modern, I was not happy with that idea because it could mean that the RSS feed would have been republished. As this blog pushes updates to <a href="https://planet.jabber.org/">the jabber Planet</a>, I was afraid it would spam all users with old articles. After reading the diagnostics of Lighthouse, I was able to highly improve the performances of the blog. I can keep <a href="https://getpelican.com/">Pelican</a>, my current static blog generator. I even updated the blueidea theme according to the latest changes in <a href="https://github.com/getpelican/pelican/commits/master/pelican/themes/notmyidea">notmyidea</a>, the default theme of Pelican. Two pictures are better than one big sentence.</p>
<h2>Before</h2>
<p><img alt="ligthouse before" src="https://blog.agayon.be/images/lighthouse_before.webp" style="width: 720px; height: auto; max-width: 100%;"/></p>
<h2>After</h2>
<p><img alt="lighthouse after" src="https://blog.agayon.be/images/lighthouse_after.webp" style="width: 755px; height: auto; max-width: 100%;"/></p>
<h2>Strategy</h2>
<p>In order to improve the score, the following changes were made:</p>
<ul>
<li>Convert most of the images into webp, reduce the size and resolution of the biggest ones</li>
<li>Update the iframe settings of some embedded youtube video</li>
<li>Decrease the number of article per page</li>
<li>Improve some links to avoid "read more", "go" etc. generic descriptions</li>
<li>Activate <a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP2</a> on the blog virtualhost</li>
</ul>
Écouter des Podcasts MP3 en voiture2023-04-13T15:00:00+02:002023-04-13T15:00:00+02:00Arnaudtag:blog.agayon.be,2023-04-13:/rss_podcasts.html
<p>Pendant mes trajets en voiture, j'aime écouter des podcasts. Ma voiture supporte la connection Bluetooth avec un téléphone mais pour diverses raisons, je préfère utiliser une bonne vieille clé USB.
Je récupère les épisodes à l'aide de l'application <a href="https://apps.gnome.org/app/org.gnome.Podcasts/">GNOME Podcast</a> qui récupère les flux RSS/Atom.</p>
<p>Les scripts ci-dessous me permettent d'écouter <a href="https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451">La semaine des 5 heures</a> pendant des heures, étant donné qu'ils sont passés à une formule quotidienne. J'écoute également d'excellents podcasts de France Inter: <a href="https://www.radiofrance.fr/franceinter/podcasts/blockbusters-le-podcast">Blockbusters</a> et <a href="https://www.radiofrance.fr/franceculture/podcasts/la-science-cqfd">La Science, CQFD</a> sur France Culture.</p>
<h1>RTBF AUVIO</h1>
<p>Il y a quelques mois, le site <a href="https://auvio.rtbf.be/">Auvio</a> de la RTBF a changé et les liens des podcasts n'étaient plus autant mis en avant. Je suis nénmoins tombé sur ce <a href="https://libreantenne.radioactu.com/topic/40041-rtbf-flux-rss-des-podcast-introuvables/">forum</a> qui m'a aidé à trouver la solution:</p>
<blockquote>
<p>Le nom du programme en toutes lettres est remplacé par des chiffres, identiques au code d'Auvio, la plateforme de Replay/Podcast de la RTBF.</p>
</blockquote>
<p>Ainsi, la semaine de 5h, l'émission cinéma dont l'identifiant <code>1451</code> voit ses addresses évoluer comme ceci: </p>
<ul>
<li><strong>AUVIO</strong> https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451 </li>
<li><strong>Podcast</strong> http://rss.rtbf.be/media/rss/audio/1451.xml</li>
</ul>
<h1>Scripts de conversion et renommage</h1>
<p>Par ailleurs, ma voiture ne sait pas lire les fichiers <code>m2a</code> récupérés par l'application. Je les convertis à l'aide de <a href="https://ffmpeg.org/">FFMPEG</a> afin de les lire. J'utilise les scripts ci-dessous pour lire le tout.</p>
<p>Le premier converti les fichiers en mp3 et le second les renommes en fonction de la date de l'épisode.</p>
<p>Pendant mes trajets en voiture, j'aime écouter des podcasts. Ma voiture supporte la connection Bluetooth avec un téléphone mais pour diverses raisons, je préfère utiliser une bonne vieille clé USB.
Je récupère les épisodes à l'aide de l'application <a href="https://apps.gnome.org/app/org.gnome.Podcasts/">GNOME Podcast</a> qui récupère les flux RSS/Atom.</p>
<p>Les scripts ci-dessous me permettent d'écouter <a href="https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451">La semaine des 5 heures</a> pendant des heures, étant donné qu'ils sont passés à une formule quotidienne. J'écoute également d'excellents podcasts de France Inter: <a href="https://www.radiofrance.fr/franceinter/podcasts/blockbusters-le-podcast">Blockbusters</a> et <a href="https://www.radiofrance.fr/franceculture/podcasts/la-science-cqfd">La Science, CQFD</a> sur France Culture.</p>
<h1>RTBF AUVIO</h1>
<p>Il y a quelques mois, le site <a href="https://auvio.rtbf.be/">Auvio</a> de la RTBF a changé et les liens des podcasts n'étaient plus autant mis en avant. Je suis nénmoins tombé sur ce <a href="https://libreantenne.radioactu.com/topic/40041-rtbf-flux-rss-des-podcast-introuvables/">forum</a> qui m'a aidé à trouver la solution:</p>
<blockquote>
<p>Le nom du programme en toutes lettres est remplacé par des chiffres, identiques au code d'Auvio, la plateforme de Replay/Podcast de la RTBF.</p>
</blockquote>
<p>Ainsi, la semaine de 5h, l'émission cinéma dont l'identifiant <code>1451</code> voit ses addresses évoluer comme ceci: </p>
<ul>
<li><strong>AUVIO</strong> https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451 </li>
<li><strong>Podcast</strong> http://rss.rtbf.be/media/rss/audio/1451.xml</li>
</ul>
<h1>Scripts de conversion et renommage</h1>
<p>Par ailleurs, ma voiture ne sait pas lire les fichiers <code>m2a</code> récupérés par l'application. Je les convertis à l'aide de <a href="https://ffmpeg.org/">FFMPEG</a> afin de les lire. J'utilise les scripts ci-dessous pour lire le tout.</p>
<p>Le premier converti les fichiers en mp3 et le second les renommes en fonction de la date de l'épisode.</p>
<h2>Conversion en mp3</h2>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">EXT</span><span class="o">=</span><span class="s2">".m2a"</span>
<span class="nv">FILES</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>/*<span class="s2">"</span><span class="nv">$EXT</span><span class="s2">"</span>
<span class="nv">DIRECTORY</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
/usr/bin/python3<span class="w"> </span>podcast_rename.py<span class="w"> </span><span class="s2">"</span><span class="nv">$DIRECTORY</span><span class="s2">"</span>
<span class="k">for</span><span class="w"> </span>filename<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span>find<span class="w"> </span><span class="s2">"</span><span class="nv">$DIRECTORY</span><span class="s2">"</span><span class="w"> </span>-name<span class="w"> </span><span class="s2">"*</span><span class="nv">$EXT</span><span class="s2">"</span><span class="w"> </span>-type<span class="w"> </span>f<span class="sb">`</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span>
<span class="w"> </span>ffmpeg<span class="w"> </span>-i<span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span><span class="w"> </span>-acodec<span class="w"> </span>mp3<span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span>.mp3
<span class="w"> </span>rm<span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span>
<span class="k">done</span>
<span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<h2>Renommage des fichiers mp3</h2>
<p>Les fichiers téléchargés suivent une séquence propre au programme GNOME Podcast mais il est parfois intéressant de savoir à quelle date correspond ce fichier. C'est particulièrement utile pour suivre les sorties cinéma.</p>
<p>Pour cela, j'analyse les metadata des fichiers m2a et j'extrait la date à partir du titre ou du tag <code>date</code>. </p>
<p>Le programme suivant </p>
<ul>
<li>recherche les fichiers <code>m2a</code> dans un dossier donné</li>
<li>renomme le fichier en se basant sur la date présente dans les métadata (titre ou champs date).</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env python</span>
<span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">import</span> <span class="nn">ffmpeg</span>
<span class="kn">import</span> <span class="nn">glob</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">_get_date</span><span class="p">(</span><span class="n">input_str</span><span class="p">,</span> <span class="n">datetime_format</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Return the datetime object from the string containing the date of the media file</span>
<span class="sd"> :param input_str: string to analyze</span>
<span class="sd"> :param: regex_date: regex format of the date</span>
<span class="sd"> :return: datetime object</span>
<span class="sd"> """</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'(\d+(-|\/)\d+(-|\/)\d+)'</span><span class="p">,</span> <span class="n">input_str</span><span class="p">)</span>
<span class="n">date_str</span> <span class="o">=</span> <span class="n">match</span> <span class="ow">and</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="n">date_str</span> <span class="ow">and</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">date_str</span><span class="p">,</span> <span class="n">datetime_format</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">_rename_file</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">date_obj</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Rename media file based on datetime object</span>
<span class="sd"> :param media_file: input filename</span>
<span class="sd"> :param date_obj: datetime of the media file</span>
<span class="sd"> :return:</span>
<span class="sd"> """</span>
<span class="n">vals</span> <span class="o">=</span> <span class="n">media_file</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">if</span> <span class="n">vals</span> <span class="ow">and</span> <span class="n">date_obj</span><span class="p">:</span>
<span class="n">dst_name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">vals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">date_obj</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s1">'</span><span class="si">%d</span><span class="s1">-%m-%Y'</span><span class="p">)</span><span class="si">}</span><span class="s2">.m2a"</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Trying to rename </span><span class="si">{</span><span class="n">media_file</span><span class="si">}</span><span class="s2"> --> </span><span class="si">{</span><span class="n">dst_name</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">rename</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">dst_name</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">rename_on_title_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Rename a m2a file based on the date present in the title. Example:</span>
<span class="sd"> La Semaine des 5 Heures - TITLE OF THE DAY - 03/04/2023</span>
<span class="sd"> :param media_file: fullpath of the m2a filename</span>
<span class="sd"> :param metadata: metadata dictionary of the file</span>
<span class="sd"> :return: True if the file could be renamed. Otherwise False or None</span>
<span class="sd"> """</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'format'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tags'</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">tags</span> <span class="ow">and</span> <span class="n">tags</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'title'</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">title</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">date_obj</span> <span class="o">=</span> <span class="n">_get_date</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="s2">"</span><span class="si">%d</span><span class="s2">/%m/%Y"</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">_rename_file</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">date_obj</span><span class="p">)</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">rename_on_record_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Rename a m2a file based on the date present in the record date metadata.</span>
<span class="sd"> Example: 2023-03-23</span>
<span class="sd"> :param media_file: fullpath of the m2a filename</span>
<span class="sd"> :param metadata: metadata dictionary of the file</span>
<span class="sd"> :return: True if the file could be renamed. Otherwise False</span>
<span class="sd"> """</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'format'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tags'</span><span class="p">)</span>
<span class="n">date_str</span> <span class="o">=</span> <span class="n">tags</span> <span class="ow">and</span> <span class="n">tags</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'date'</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">date_str</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">date_obj</span> <span class="o">=</span> <span class="n">_get_date</span><span class="p">(</span><span class="n">date_str</span><span class="p">,</span> <span class="s2">"%Y-%m-</span><span class="si">%d</span><span class="s2">"</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">_rename_file</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">date_obj</span><span class="p">)</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">launch</span><span class="p">(</span><span class="n">dir_name</span><span class="p">):</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">dir_name</span><span class="p">,</span> <span class="s1">'*.m2a'</span><span class="p">)</span>
<span class="n">files</span> <span class="o">=</span> <span class="n">glob</span><span class="o">.</span><span class="n">glob</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">for</span> <span class="n">media_file</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
<span class="n">metadata</span> <span class="o">=</span> <span class="n">ffmpeg</span><span class="o">.</span><span class="n">probe</span><span class="p">(</span><span class="n">media_file</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">rename_on_title_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">res</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">rename_on_record_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">res</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The file could not be renamed </span><span class="si">{</span><span class="n">media_file</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">UserWarning</span><span class="p">(</span><span class="s2">"You need to provide a directory name"</span><span class="p">)</span>
<span class="n">directory</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">launch</span><span class="p">(</span><span class="n">directory</span><span class="p">)</span>
</code></pre></div>Updates: chatty server and HTTPAuthentificationOverXMPP2022-09-11T15:00:00+02:002022-09-11T15:00:00+02:00Arnaudtag:blog.agayon.be,2022-09-11:/xmpp_auth_update.html
<p>It's been a long time since I updated this blog. It will be a short update post about two projects.</p>
<h1>chatty_server</h1>
<p>The first is chatty_server, a small XMPP bot I use to interact with my server. It allows me to get information about the CPU load, traffic, weather etc.
It also has a small feature to get reminder messages. There was a bug that allowed anyone to spam reminders. Anybody can add the bot to their rooster and could create random reminders that I would get.
I got none, so the bot must remain quite unheard-of. </p>
<h1>HTTPAuthentificationOverXMPP</h1>
<p>The second project is HTTPAuthentificationOverXMPP, a component that I use to allow 2 Factor authentication with XMPP.
The <a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">original project</a> had not been updated for a long time and I wanted to try to modify it to rely on another XMPP go library.
I have never coded in <a href="https://go.dev/">the Golang language</a> and it seemed like a nice introduction. I relied on <a href="https://github.com/FluuxIO/go-xmpp">go-xmpp</a> where I added the <a href="https://github.com/jarobase/go-xmpp/commit/1346ff2a1327414bca0cf4d8b6dba5bea27aa3fc">support for XEP 0070</a>. It was really interesting. The new component is running for several months and I am quite happy with it even if I don't have any serious project relying on it.</p>
<p>The https://demo.agayon.be/ website is still up if you want to test it.<br/>
I hope being able to provide more update about my projects in the future :-).</p>
<h1 id="links">Links</h1>
<ul>
<li><a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a></li>
<li><a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">Authentication component by Chteufleur</a></li>
<li><a href="https://github.com/jarobase/go-xmpp/tree/xep-0070-httpauth">Fork of go-xmpp</a></li>
<li><a href="https://gitlab.com/jnanar/HTTPAuthentificationOverXMPP/-/tree/update_FluuxIO">Fork of HTTPAuthentificationOverXMPP</a></li>
<li><a href="https://gitlab.com/r1dScripts/chatty_server">Chatty Server</a></li>
</ul>
Agayon at the Hospital 🏥2021-11-29T15:00:00+01:002021-11-29T15:00:00+01:00Arnaudtag:blog.agayon.be,2021-11-29:/agayon_hospital.html
<p>Today, I was lucky to be invited in the school of the Hospital <a href="https://fr.wikipedia.org/wiki/Clinique_CHC_MontL%C3%A9gia"><em>Mont Légia</em></a> in Liège. It was an afternoon session following their science course to present the Agayon to children and teenagers (~6 to 16 years old).
They follow their schooling while being hospitalized.
I was contacted by a friend of a friend who works there as a physics teacher. He likes to organize interactive sessions with the children to ease their day. It mainly teaches to the older ones but the youngest were quite happy to discover it too. </p>
<h1>Equipment and setup</h1>
<p>I brought the Agayon, R1D1, my other small robot, the wireless PS4 controller,some material and additional batteries.</p>
<p><img alt="Equipment" src="https://blog.agayon.be/images/r1d3/hospital/matos.jpg.webp" style="width: 800px; height: auto; max-width: 100%;"/></p>
<p>The day before, I made sure I could connect to the robot through <a href="https://en.wikipedia.org/wiki/Secure_Shell">SSH</a> using the WiFi of my phone as a tethering hotspot. I would not be able to have another internet connection.
Fortunately, the webcam streaming and web connection works well with my <a href="https://en.wikipedia.org/wiki/Fairphone_3">Fairphone 3</a>.</p>
<p><img alt="ssh" src="https://blog.agayon.be/images/r1d3/hospital/ssh.JPG.webp" style="width: 1000px; height: auto; max-width: 100%;"/></p>
<h1>The animation</h1>
<p>I started with a small <a href="https://agayon.be/presentation/agayon/#/Titre">presentation</a> available online (in french) and question/answer with the children. </p>
<p><img alt="Class" src="https://blog.agayon.be/images/r1d3/hospital/class.JPG.webp" style="width: 1000px; height: auto; max-width: 100%;"/>
<img alt="Window" src="https://blog.agayon.be/images/r1d3/hospital/window.JPG.webp" style="width: 1000px; height: auto; max-width: 100%;"/></p>
<p>After that, the children played with the Agayon using the remote or the autonomous mode. They also used some awesome accessories.
<img alt="Doll" src="https://blog.agayon.be/images/r1d3/hospital/doll.JPG.webp" style="width: 1000px; height: auto; max-width: 100%;"/></p>
<p>I was super happy with the result, the interactions and to see that I made the day of some of them 😁.</p>
<h1>Link</h1>
<ul>
<li><a href="https://agayon.be/presentation/agayon/#/Titre">Presentation</a> </li>
</ul>
Using Prosody with a HTTP Reverse Proxy2021-03-21T15:00:00+01:002021-03-21T15:00:00+01:00Arnaudtag:blog.agayon.be,2021-03-21:/prosody_http.html
<p>It's been a while since I first installed prosody on Agayon.be. I use it to experiments with my bots, to keep contact with the XMPP community and discover new cool stuffs to do.
Recently I struggled a bit because I wanted to hide the prosody small HTTP server behind my Proxy. For various reasons, I still use Apache 2.4 and I could not get it to work with prosody. I mostly use the HTTP server for bosh authentication with <a href="https://conversejs.org/">Converse.js</a> and with the <a href="https://modules.prosody.im/mod_http_upload.html">http_upload</a> module.
When the 5281 port was accessible and Prosody handled the requests directly on the internet it worked well. But when I followed the documentation to use a proxy, it stopped working.
All my PUT requests got a 404 error. I tested my setup with <a href="https://slixmpp.readthedocs.io/en/latest/">Slixmpp</a> and the http_upload example.</p>
<p>Here is my configuration before the fix:</p>
<h2>Prosody</h2>
<h3>Main config</h3>
<div class="highlight"><pre><span></span><code><span class="k">[...]</span>
<span class="na">https_ports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">{ 5281 }</span>
<span class="na">https_interfaces</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">{ "127.0.0.1", "::1" }</span>
<span class="na">trusted_proxies</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">{ "127.0.0.1", "::1"}</span>
<span class="k">[...]</span>
</code></pre></div>
<h3>VirtualHost</h3>
<div class="highlight"><pre><span></span><code><span class="k">[...]</span>
<span class="na">Component "upload.example.com" "http_upload"</span>
<span class="w"> </span><span class="na">http_max_content_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">10485760</span>
<span class="w"> </span><span class="na">http_external_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"https://upload.example.com/"</span>
<span class="k">[...]</span>
</code></pre></div>
<h2>Apache VirtualHost</h2>
<div class="highlight"><pre><span></span><code><span class="o">[</span>...<span class="o">]</span>
ProxyPass<span class="w"> </span>/<span class="w"> </span>http://localhost:5280/
ProxyPassReverse<span class="w"> </span>/<span class="w"> </span>http://localhost:5280/
<span class="o">[</span>...<span class="o">]</span>
</code></pre></div>
<h2>Logs</h2>
<h3>Client</h3>
<div class="highlight"><pre><span></span><code><span class="k">[...]</span>
<span class="na">Client</span><span class="o">:</span>
<span class="na">DEBUG SEND</span><span class="o">:</span><span class="w"> </span><span class="s"><iq id="23efd54cf4b2487386852e800f2ea411" to="upload.example.com" type="get"><request xmlns="urn:xmpp:http:upload:0" filename="robot.png" size="118037" content-type="image/png" /></iq></span>
<span class="na">DEBUG RECV</span><span class="o">:</span><span class="w"> </span><span class="s"><iq type="result" id="23efd54cf4b2487386852e800f2ea411" from="upload.example.com" to="test@example.com/test"><slot xmlns="urn:xmpp:http:upload:0"><get url="https://upload.example.com/upload/au5rOiUMomJbDI3q/robot.png" /><put url="https://upload.example.com/upload/au5rOiUMomJbDI3q/robot.png" /></slot></iq></span>
<span class="na">ERROR Could not upload file</span><span class="o">:</span><span class="w"> </span><span class="s">404 (<!DOCTYPE html></span>
<span class="na"><html></span>
<span class="na"><head></span>
<span class="na"><meta charset</span><span class="o">=</span><span class="s">"utf-8"></span>
<span class="na"><title>404 Not Found</title></span>
<span class="na">example.com</span>
<span class="k">[...]</span>
</code></pre></div>
<h3>Server</h3>
<div class="highlight"><pre><span></span><code><span class="k">[...]</span>
<span class="na">Mar 21 10</span><span class="o">:</span><span class="s">02:42 c2s5586c8e88960 debug Received[c2s]: <iq id='23efd54cf4b2487386852e800f2ea411' type='get' to='upload.example.com'></span>
<span class="na">Mar 21 10</span><span class="o">:</span><span class="s">02:42 c2s5586c8e88960 debug Given upload slot "au5rOiUMomJbDI3q/robot.png"</span>
<span class="na">Mar 21 10</span><span class="o">:</span><span class="s">02:42 c2s5586c8e88960 debug Sending[c2s]: <iq type='result' id='23efd54cf4b2487386852e800f2ea411' from='upload.example.com' to='memo@agayon.be/test'></span>
<span class="na">Mar 21 10</span><span class="o">:</span><span class="s">02:42 socket debug server.lua: accepted new client connection from ::1:49436 to 5280</span>
<span class="na">Mar 21 10</span><span class="o">:</span><span class="s">02:42 http.server debug Firing event: PUT /upload/au5rOiUMomJbDI3q/robot.png</span>
<span class="na">Mar 21 10</span><span class="o">:</span><span class="s">02:42 http.server debug Firing event: PUT localhost/upload/au5rOiUMomJbDI3q/robot.png</span>
</code></pre></div>
<p>What is important to see is the second event:</p>
<div class="highlight"><pre><span></span><code><span class="n">PUT</span><span class="w"> </span><span class="n">localhost</span><span class="o">/</span><span class="n">upload</span><span class="o">/</span><span class="n">au5rOiUMomJbDI3q</span><span class="o">/</span><span class="n">robot</span><span class="o">.</span><span class="n">png</span>
</code></pre></div>
<p>on a working configuration it is </p>
<div class="highlight"><pre><span></span><code><span class="n">PUT</span><span class="w"> </span><span class="n">upload</span><span class="o">.</span><span class="n">example</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">upload</span><span class="o">/</span><span class="n">au5rOiUMomJbDI3q</span><span class="o">/</span><span class="n">robot</span><span class="o">.</span><span class="n">png</span>
</code></pre></div>
<h1>The fix</h1>
<p>After a small discussion with Link Mauve from JabberFr, he suggested me to use setup the proxy to tell him to force its headers:</p>
<h2>Apache</h2>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="w"> </span><span class="n">RequestHeader</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="n">Host</span><span class="w"> </span><span class="s2">"upload.example.com"</span>
<span class="w"> </span><span class="n">ProxyPreserveHost</span><span class="w"> </span><span class="n">On</span>
<span class="w"> </span><span class="n">ProxyPass</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">5280</span><span class="o">/</span>
<span class="w"> </span><span class="n">ProxyPassReverse</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">5280</span><span class="o">/</span>
<span class="w"> </span><span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<h2>Nginx</h2>
<div class="highlight"><pre><span></span><code><span class="k">[...]</span>
<span class="na">proxy_set_header Host "upload.example.com";</span>
<span class="k">[...]</span>
</code></pre></div>
<p>Two small lines and now it is working as expected :-).</p>
<h1 id="links">Links</h1>
<ul>
<li><a href="https://modules.prosody.im/mod_http_upload.html">http_upload module</a></li>
<li><a href="https://prosody.im/doc/http">Posody http configuration</a></li>
</ul>
What have I done ?2020-12-23T12:00:00+01:002020-12-23T12:00:00+01:00Arnaudtag:blog.agayon.be,2020-12-23:/what_have_I_done.html
<h1>Intro</h1>
<p>During these weird time of corona virus crisis, like many, I am working from home and cancelled all social activities. Apart from the distanciation, the virus, the lack of activities, it has been a quite fruitful period as I work <em>a lot</em> on the robot. This articles sums up the latest developments and achievements.</p>
<h1>New functionnalities</h1>
<p>My robot, the <em>Agayon</em> has now kite a lot more features. Among which: </p>
<ul>
<li>Video streaming nearly in real time. The stream can be viewed in a web browser. It works quite well and has been used across the internet. Some friends or family members have visited me with the help of the robot.
The only drawback is that I have to limit the stream to 10 frames per second.
The setup is based on <a href="https://github.com/jacksonliam/mjpg-streamer">mjpg-streamer</a> and the custom layout is <a href="https://gitlab.com/r1d3/www_agayon">available on his dedicated Gitlab repository</a>. </li>
<li>The Agayon can now be remotely controlled with<ul>
<li>A PS4 controller (Bluetooth)</li>
<li>REST API to control the robot.</li>
<li>A web interface that uses the API (see screenshot below)</li>
</ul>
</li>
<li>Lidar mapping: It can be triggered by the PS4 controller, the web interface or XMPP. I use a <a href="https://www.robotshop.com/eu/en/rplidar-a1m8-360-degree-laser-scanner-development-kit.html">RPLidar A1M8</a>.
Data is saved in a file that can be analyzed afterward. No real time data processing for now.</li>
<li>XMPP: migration from the deprecated <a href="https://github.com/fritzy/SleekXMPP">Sleekxmpp</a> library to the more modern one <a href="https://lab.louiz.org/poezio/slixmpp">Slixmpp</a>.</li>
</ul>
<h1>Changelogs</h1>
<h2>Python (Raspberry Pi)</h2>
<p>The code is available <a href="https://gitlab.com/r1d3/rpi">on my gitlab account</a>.</p>
<ul>
<li>Small turn angles for remote control. The robot was sometime too much brutal during rotation. This version adds the support for 'gentle turns'. During the new gentle turns, only one wheel is moving. The 'normal turn' remains and it rotates both wheels in opposite direction.</li>
<li>Start and Stop Webcam streaming from socket. It relies on the <code>restart_stream.sh</code> and <code>stop_stream.sh</code> <a href="https://gitlab.com/r1d3/misc/-/tree/master">scripts</a>. It is used from the web interface and XMPP.</li>
<li>Mapping: simple scan and logging of all mapping data. It Saves a picture (polar graph) for each snapshot.</li>
<li>Arduino communication: serial data to get the ultrasonic sensors measurements.</li>
<li>The rover can be remotely controlled with a PS4 controller. Events are caught and instructions are sent through serial communication to Arduino.</li>
<li>Some refactoring and cleaning.</li>
<li>A lot of bug fix.</li>
</ul>
<h2>Arduino</h2>
<p>The code is available <a href="https://gitlab.com/r1d3/arduino">on the gitlab repository</a>.</p>
<ul>
<li>I created a small Serial manager to handle orders from the RPI. It can be used to remotely control the motors, get ultrasonic measurements, change speed, change mode (incremental or directly to a mode number), capacity to turn a little (small angles but <a href="https://orbi.uliege.be/handle/2268/190378">not that small </a>) </li>
<li>Send formatted serial data that can be easily parsed by the RPI process (odometry).</li>
<li>A lot of bug fix</li>
</ul>
<h2>Misc</h2>
<p>The web remote control is based on a small webservice depending on with the <a href="https://palletsprojects.com/p/flask/">Flask</a> framework. It is served with <a href="https://uwsgi-docs.readthedocs.io/en/latest/">uWSGI</a> with the help of <a href="https://www.nginx.com/">Nginx</a>. The sources are <a href="https://gitlab.com/r1d3/rest_api">on the Gitab related repository</a>. It is used to control the robot with a web interface but any client able to use a web API can use it. Maybe an Android client will follow?</p>
<p>When the video streaming service is unavailable, a <a href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#5xx_Server_Error">503 error</a> is displayed with a custom page.</p>
<p><img alt="503 error" src="https://blog.agayon.be/images/r1d3/web/503.png.webp" style="width: 428px; height: auto; max-width: 100%;"/></p>
<p>I use it to launch the Webcam streaming directly from the navigator.</p>
<p>All the configuration, HTML pages, scripts and config files are available in a <a href="https://gitlab.com/r1d3/misc">separate repository</a>.</p>
<p>Once <code>mjpg_streamer</code> is launched, it serves a <a href="https://gitlab.com/r1d3/www_agayon">small page</a> to interact with the API. </p>
<p>This picture was taken at the beginning of a ~4m narrow hallway.</p>
<p><img alt="Hallway" src="https://blog.agayon.be/images/r1d3/web/capture_stream.png.webp" style="width: 1000px; height: auto; max-width: 100%;"/></p>
<h1>Pictures</h1>
<p>Here are some pictures token in my apartment. As long as no Lidar measurment has been made, a cartoonish picture of the robot is displayed.
As the robot is going forward, the mapping is updated by clicking on the Lidar camera button. For unknown reason, the left/right mapping is inverted in the picture.</p>
<h2>Start</h2>
<p><img alt="Start Lidar" src="https://blog.agayon.be/images/r1d3/what_have_i_done/start_0.png.webp" style="width: 900px; height: auto; max-width: 100%;"/>
<img alt="Start Lidar" src="https://blog.agayon.be/images/r1d3/what_have_i_done/start_1.png.webp" style="width: 900px; height: auto; max-width: 100%;"/></p>
<h2>Middle</h2>
<p><img alt="Middle Lidar" src="https://blog.agayon.be/images/r1d3/what_have_i_done/middle.png.webp" style="width: 900px; height: auto; max-width: 100%;"/></p>
<h2>End</h2>
<p><img alt="End Lidar" src="https://blog.agayon.be/images/r1d3/what_have_i_done/end.png.webp" style="width: 900px; height: auto; max-width: 100%;"/></p>
<h1>Future</h1>
<p>There are still a lot of room for improvements. Here are some ideas to occupy me in the future:</p>
<ul>
<li>Automatically save a camera picture when the lidar mapping is triggered.</li>
<li>Detect the kernel messages about battery. When the voltage is too low, the information is logged by the kernel and can be found with systemd. The idea would be to shut down the robot when necessary to avoid SD card memory corruption.</li>
<li>Apply some nice OpenCV filter to obtain a transformed video stream line in the movie Terminator. :-)</li>
<li>Use one button to trigger the video recording. Use a blinking LED to let known that it is "On Air".</li>
<li>...</li>
</ul>
<h2>Recording movies with the camera</h2>
<p>Recording a movie with OpenCV can be done in less than 25 python lines. It works well with my old <em>Logitech C170</em> but for some reasons it did not work out of the box with my <em>Microsoft LifeCam Studio</em>.
Dear visitor, if you have such a camera, here the magic trick to produce your new feature movie ! First, make sure to have this model with the <code>lsudb</code> command: </p>
<div class="highlight"><pre><span></span><code>Bus<span class="w"> </span><span class="m">001</span><span class="w"> </span>Device<span class="w"> </span><span class="m">007</span>:<span class="w"> </span>ID<span class="w"> </span>045e:0811<span class="w"> </span>Microsoft<span class="w"> </span>Corp.<span class="w"> </span>Microsoft®<span class="w"> </span>LifeCam<span class="w"> </span>Studio<span class="o">(</span>TM<span class="o">)</span>
</code></pre></div>
<p>The camera produce naturally a MJPG stream. I would sum it up a stream of JPEG pictures. By default, the VideoWriter class will produce empty video files. Unfortunately, by default there are no debug message to help you.
Don't forget to use the following environment variables during your hacking sessions: </p>
<div class="highlight"><pre><span></span><code><span class="nv">OPENCV_VIDEOIO_DEBUG</span><span class="o">=</span><span class="m">1</span><span class="w"> </span>
<span class="nv">OPENCV_LOG_LEVEL</span><span class="o">=</span>verbose<span class="w"> </span>
</code></pre></div>
<p>To make it work, you need to tune the camera settings before obtaining the desired result. The complete program is displayed here. Follow the comments to see where the magic happens.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">cv2</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="n">cap</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">VideoCapture</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">cap</span><span class="o">.</span><span class="n">isOpened</span><span class="p">():</span>
<span class="k">raise</span> <span class="ne">IOError</span><span class="p">(</span><span class="s2">"Cannot open webcam"</span><span class="p">)</span>
<span class="c1"># Magic number corresponding to a MJPG Stream</span>
<span class="n">codec</span> <span class="o">=</span> <span class="mh">0x47504A4D</span>
<span class="c1"># You actually says that your camera produces such a stream</span>
<span class="n">cap</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">cv2</span><span class="o">.</span><span class="n">CAP_PROP_FOURCC</span><span class="p">,</span> <span class="n">codec</span><span class="p">)</span>
<span class="c1"># We fix the resolution, the framerate and use exactly the same in the VideoWriter arguments </span>
<span class="n">cap</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">cv2</span><span class="o">.</span><span class="n">CAP_PROP_FRAME_WIDTH</span><span class="p">,</span> <span class="mi">640</span><span class="p">)</span>
<span class="n">cap</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">cv2</span><span class="o">.</span><span class="n">CAP_PROP_FRAME_HEIGHT</span><span class="p">,</span> <span class="mi">480</span><span class="p">)</span>
<span class="c1"># The framerate is not limitating here</span>
<span class="n">cap</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">cv2</span><span class="o">.</span><span class="n">CAP_PROP_FPS</span><span class="p">,</span> <span class="mf">20.0</span><span class="p">)</span>
<span class="c1"># next we define the filename, the writer options and start the infinite loop.</span>
<span class="n">videoname</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">'/tmp/'</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"output.avi"</span><span class="p">)</span>
<span class="n">fourcc</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">VideoWriter_fourcc</span><span class="p">(</span><span class="o">*</span><span class="s1">'MJPG'</span><span class="p">)</span>
<span class="n">video_writer</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">VideoWriter</span><span class="p">(</span><span class="n">videoname</span><span class="p">,</span> <span class="n">fourcc</span><span class="p">,</span> <span class="mf">20.0</span><span class="p">,</span> <span class="p">(</span><span class="mi">640</span><span class="p">,</span> <span class="mi">480</span><span class="p">))</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">ret</span><span class="p">,</span> <span class="n">frame</span> <span class="o">=</span> <span class="n">cap</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">video_writer</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span>
<span class="c1"># When the Q key is pressed, the loop is stopped</span>
<span class="k">if</span> <span class="n">cv2</span><span class="o">.</span><span class="n">waitKey</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">&</span> <span class="mh">0xFF</span> <span class="o">==</span> <span class="mi">27</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">cap</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="n">video_writer</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="n">cv2</span><span class="o">.</span><span class="n">destroyAllWindows</span><span class="p">()</span>
</code></pre></div>
<h2>Webserver configuration</h2>
<p>You can find here the webserver and uWSGI configurations used to make it work.</p>
<h3>Nginx</h3>
<div class="highlight"><pre><span></span><code><span class="n">upstream</span> <span class="n">mjpeg</span> <span class="p">{</span>
<span class="n">server</span> <span class="mf">127.0.0.1</span><span class="p">:</span><span class="mi">8090</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">server</span> <span class="p">{</span>
<span class="n">listen</span> <span class="mi">443</span> <span class="n">ssl</span><span class="p">;</span>
<span class="n">server_name</span> <span class="n">namek</span><span class="o">.</span><span class="n">agayon</span><span class="o">.</span><span class="n">netlib</span><span class="o">.</span><span class="n">re</span><span class="p">;</span>
<span class="n">location</span> <span class="o">/</span> <span class="p">{</span>
<span class="n">proxy_redirect</span> <span class="n">off</span><span class="p">;</span>
<span class="n">proxy_pass</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">mjpeg</span><span class="p">;</span>
<span class="n">index</span> <span class="n">index</span><span class="o">.</span><span class="n">html</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">location</span> <span class="o">/</span><span class="n">api</span> <span class="p">{</span>
<span class="n">include</span> <span class="n">uwsgi_params</span><span class="p">;</span>
<span class="n">uwsgi_pass</span> <span class="mf">127.0.0.1</span><span class="p">:</span><span class="mi">3031</span><span class="p">;</span>
<span class="n">proxy_read_timeout</span> <span class="mi">300</span><span class="n">s</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">location</span> <span class="o">/</span><span class="n">static</span><span class="o">/</span> <span class="p">{</span>
<span class="n">alias</span> <span class="o">/</span><span class="n">srv</span><span class="o">/</span><span class="n">http</span><span class="o">/</span><span class="n">ngnix</span><span class="o">/</span><span class="n">r1d3</span><span class="o">/</span><span class="n">public_html</span><span class="o">/</span><span class="n">static</span><span class="o">/</span><span class="p">;</span>
<span class="n">try_files</span> <span class="err">$</span><span class="n">uri</span> <span class="err">$</span><span class="n">uri</span><span class="o">/</span> <span class="o">/</span><span class="n">static</span><span class="o">/</span><span class="n">lost_bot</span><span class="o">.</span><span class="n">jpg</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">ssl_certificate</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">agayon</span><span class="o">.</span><span class="n">netlib</span><span class="o">.</span><span class="n">re_fullchain</span><span class="o">.</span><span class="n">pem</span><span class="p">;</span>
<span class="n">ssl_certificate_key</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">agayon</span><span class="o">.</span><span class="n">netlib</span><span class="o">.</span><span class="n">re</span><span class="o">.</span><span class="n">key</span><span class="p">;</span>
<span class="n">ssl_session_timeout</span> <span class="mi">1</span><span class="n">d</span><span class="p">;</span>
<span class="n">access_log</span> <span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">r1d3</span><span class="o">.</span><span class="n">access</span><span class="o">.</span><span class="n">log</span><span class="p">;</span>
<span class="n">error_log</span> <span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">r1d3</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="n">log</span><span class="p">;</span>
<span class="n">error_page</span> <span class="mi">500</span> <span class="mi">502</span> <span class="mi">503</span> <span class="mi">504</span> <span class="o">/</span><span class="mf">500.</span><span class="n">html</span><span class="p">;</span>
<span class="n">location</span> <span class="o">=</span> <span class="o">/</span><span class="mf">500.</span><span class="n">html</span> <span class="p">{</span>
<span class="n">root</span> <span class="o">/</span><span class="n">srv</span><span class="o">/</span><span class="n">http</span><span class="o">/</span><span class="n">ngnix</span><span class="o">/</span><span class="n">r1d3</span><span class="o">/</span><span class="n">errors</span><span class="p">;</span>
<span class="n">allow</span> <span class="nb">all</span><span class="p">;</span>
<span class="n">internal</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">proxy_read_timeout</span> <span class="mi">720</span><span class="n">s</span><span class="p">;</span>
<span class="n">proxy_connect_timeout</span> <span class="mi">720</span><span class="n">s</span><span class="p">;</span>
<span class="n">proxy_send_timeout</span> <span class="mi">720</span><span class="n">s</span><span class="p">;</span>
<span class="n">proxy_set_header</span> <span class="n">X</span><span class="o">-</span><span class="n">Forwarded</span><span class="o">-</span><span class="n">Host</span> <span class="err">$</span><span class="n">host</span><span class="p">;</span>
<span class="n">proxy_set_header</span> <span class="n">X</span><span class="o">-</span><span class="n">Forwarded</span><span class="o">-</span><span class="n">For</span> <span class="err">$</span><span class="n">proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="n">proxy_set_header</span> <span class="n">X</span><span class="o">-</span><span class="n">Forwarded</span><span class="o">-</span><span class="n">Proto</span> <span class="err">$</span><span class="n">scheme</span><span class="p">;</span>
<span class="n">proxy_set_header</span> <span class="n">X</span><span class="o">-</span><span class="n">Real</span><span class="o">-</span><span class="n">IP</span> <span class="err">$</span><span class="n">remote_addr</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<h3>uWSGI</h3>
<div class="highlight"><pre><span></span><code><span class="k">[uwsgi]</span>
<span class="na">chdir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/srv/http/ngnix/api/rest_api/</span>
<span class="na">processes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">4</span>
<span class="na">threads</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">2</span>
<span class="na">plugin</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">python</span>
<span class="na">virtualenv</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/srv/http/ngnix/api/rest_api/venv</span>
<span class="na">module</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">api_agayon.agayon_app:app</span>
<span class="na">callable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">app</span>
<span class="na">wsgi-file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">wsgi.py</span>
<span class="na">master</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">true</span>
<span class="na">socket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">127.0.0.1:3031</span>
<span class="c1">;route-uri = ^/api/(.*) rewrite:/$1</span>
<span class="na">vacuum</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">true</span>
<span class="na">die-on-term</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">true</span>
<span class="na">kill-on-idle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">true</span>
<span class="na">wsgi-file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">wsgi.py</span>
<span class="na">stats</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">127.0.0.1:9191</span>
<span class="na">buffer-size</span><span class="o">=</span><span class="s">32768</span>
</code></pre></div>
<h1>Links</h1>
<ul>
<li><a href="https://gitlab.com/r1d3/rpi">Python source code</a></li>
<li><a href="https://gitlab.com/r1d3/arduino">Arduino source code</a></li>
<li><a href="https://gitlab.com/r1d3/www_agayon">HTML sources served by mjpg_streamer</a></li>
<li><a href="https://gitlab.com/r1d3/rest_api">REST API</a></li>
<li><a href="https://gitlab.com/r1d3/misc">Misc scripts and data</a></li>
<li><a href="https://github.com/jacksonliam/mjpg-streamer">mjpg streamer project page</a></li>
</ul>
Update Errol: the XMPP Automatic file sender2020-05-10T17:00:00+02:002020-05-10T17:00:00+02:00Arnaudtag:blog.agayon.be,2020-05-10:/errol_update1.html
<p>Errol is a file sender that can be used to watch a directory and automatically transfers the new files (or modified ones) with XMPP. </p>
<p>You can find the <a href="https://blog.agayon.be/errol.html">previous description here</a>.</p>
<p>I recently updated it to V2.0.1. This articles describes the changes since last article.</p>
<p>Errol is a file sender that can be used to watch a directory and automatically transfers the new files (or modified ones) with XMPP. </p>
<p>You can find the <a href="https://blog.agayon.be/errol.html">previous description here</a>.</p>
<p>I recently updated it to V2.0.1. This articles describes the changes since last article.</p>
<h1>Changelog</h1>
<ul>
<li>Replaced all the <code>yield from</code> syntax to <code>async/await</code> syntax for asyncio calls</li>
<li>Added some options to enable or not the muc/pubsub features. If your XMPP account does not have a proper pubsub service or if you don't want to advertise your file transfer, these functionalities are not mandatory anymore.</li>
<li>Moving from <a href="https://github.com/rbarrois/aionotify">aionotify</a> to <a href="https://pypi.org/project/watchdog/">watchdog</a>. aionotify was not actively developed since 2 years. Even if is not asynchronous, watchdog can be used with <a href="https://github.com/biesnecker/hachiko">hachiko</a>.</li>
</ul>
<p><strong>Errol should now be usable on Linux, Windows, Mac OS X and FreeBSD</strong> because watchdog supports all these operating systems. Note: only Linux has been tested so far.</p>
<p>I also performed small improvements, refactoring and bug fixes.</p>
<h1>Getting started</h1>
<p>The list of dependencies has been updated but remains quite small.</p>
<ul>
<li><a href="https://lab.louiz.org/poezio/slixmpp">slixmpp</a></li>
<li><a href="https://docs.python.org/3/library/asyncio.html">asyncio </a></li>
<li><a href="https://docs.python.org/3/library/configparser.html">configparser</a></li>
<li><a href="https://pypi.org/project/watchdog/">watchdog</a></li>
<li><a href="https://github.com/biesnecker/hachiko">hachiko</a></li>
</ul>
<h2>Installing</h2>
<p>You can easily install errol with pip.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>errol
</code></pre></div>
<p>On Archlinux, a PKGBUILD is available in <a href="https://aur.archlinux.org/packages/python-errol/">AUR</a>.</p>
<h2>Configuration</h2>
<p>The complete list of options is available in the template config file.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">cat</span><span class="w"> </span><span class="n">config</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">ini</span>
<span class="o">[</span><span class="n">XMPP</span><span class="o">]</span>
<span class="n">pubsub_enable</span><span class="o">=</span><span class="k">true</span>
<span class="n">muc_enable</span><span class="o">=</span><span class="k">true</span>
<span class="n">pubsub</span><span class="o">=</span><span class="n">pubsub</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">org</span>
<span class="n">node</span><span class="o">=</span><span class="n">be</span><span class="p">.</span><span class="n">agayon</span><span class="p">.</span><span class="nl">errol</span><span class="p">:</span><span class="mi">0</span>
<span class="n">room</span><span class="o">=</span><span class="n">chat</span><span class="nv">@chat</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">org</span>
<span class="n">jid</span><span class="o">=</span><span class="n">jid</span><span class="nv">@example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">errol</span>
<span class="n">password</span><span class="o">=</span><span class="n">pass</span>
<span class="n">ressource_receiver</span><span class="o">=-</span><span class="n">receiver</span>
<span class="n">ressource_sender</span><span class="o">=-</span>
<span class="n">nick_sender</span><span class="o">=</span><span class="n">example_sender</span>
<span class="n">nick_receiver</span><span class="o">=</span><span class="n">example_receiver</span>
<span class="n">receiver</span><span class="o">=</span><span class="n">jid</span><span class="nv">@example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">errol</span><span class="o">-</span><span class="n">receiver</span>
<span class="n">presence_file</span><span class="o">=/</span><span class="n">tmp</span><span class="o">/</span><span class="n">errol_presence</span><span class="p">.</span><span class="n">txt</span>
</code></pre></div>
<ul>
<li>jid : the jabber account</li>
<li>password: the xmpp password</li>
<li>pubsub: the pubsub server (publish activity)</li>
<li>room: the MUC (chatroom) where errol display information.</li>
<li>presence_file: a writable file used to keep track of presences. I use it in a Django Application. [ref]When receiver is online, the file contains '1' and '0' otherwise. It is not super clean but I did not wanted to bring XMPP features in a Django app.[/ref]</li>
</ul>
<p>The files will be sent by jid@example.org/errol-0 and received by jid@example.org/errol-receiver
. The nicks are the usernames used on the MUC.</p>
<h1>Use it</h1>
<p>Errol should now be usable with only a simple XMPP account and a directory to watch. If you are interested by the Pubsub feature, don't hesitate to read the <a href="https://blog.agayon.be/errol.html">previous article</a>. It contains a section to setup a pubsub node, configure it and access it with several tools.</p>
<h1 id="links">Links</h1>
<ul>
<li><a href="https://gitlab.com/jnanar/errol">Git repository</a></li>
<li><a href="https://pypi.python.org/pypi/errol">pypi package</a></li>
<li><a href="https://blog.agayon.be/errol.html">Previous article</a></li>
</ul>A remotely controlled mapping device2020-01-05T12:00:00+01:002020-01-05T12:00:00+01:00Arnaudtag:blog.agayon.be,2020-01-05:/remote_mapping.html
<p>During the Christmas holidays, I took the time to work on the Agayon. I hope that the mechanical parts are almost finished and I will be able to focus on the code in the following weeks/months.</p>
<h1>Do as you’re told</h1>
<p>A few years ago, I have been gifted with a PS4 controller to play on my Retropie Setup. These are quality controllers but Sony does not like makers. My model cannot be used with vanilla bluetooth drivers on my Pie. And for unknown reasons, the alternative driver <a href="https://github.com/retropie/retropie-setup/wiki/PS4-Controller">ds4drv</a> does not works for me. I purchased a 5m USB mini cable to not disappoint my lovely niece and nephew. They can plan with <a href="https://supertuxkart.net/Main_Page">Supertuxkart</a> for hours. But it's not the subject of this article. I plan to use the Agayon not only in my apartment but also to <strong>go on tour with it</strong> ! (let's dream a bit). Unfortunately, it is already quite heavy and it could be cool to drive it from my door to the car. I could dream again and imagine a system where it recognize me after a little bit of training like a dog or a <a href="https://www.youtube.com/watch?v=5-yevDAY-7U">case</a>. It seems possible but why not use the controller at first. It should be easier and quick to implement. Moreover, the kids and friends seems to love the idea. So let's do it ! I will keep you updated when it is reliable and easy to use.</p>
<p><img alt="Back to the future remote" src="https://blog.agayon.be/images/back_to_the_future.jpg.webp" style="width: 1920px; height: auto; max-width: 100%;"/>
Photo credit: <a href="https://en.wikipedia.org/wiki/Back_to_the_Future">Back to the Future</a> (1985)</p>
<h1>Re-verify our range to target... one ping only</h1>
<p>I have finally mounted the Lidar on the Agayon. The Raspberry Pi 2 was too slow to handle the data but the Pi 4 does well the job.</p>
<p><img alt="Mounted lidar" src="https://blog.agayon.be/images/r1d3/mounted_lidar2.jpg.webp" style="width: 600px; height: auto; max-width: 100%;"/></p>
<!-- <video width="640" height="480" controls="controls" src="/images/r1d3/moving_lidar.mp4">Mounted lidar</video>-->
<h2>My living room</h2>
<p>The following animation has been made with the <a href="https://github.com/SkoltechRobotics/rplidar/blob/master/examples/animate.py">animate.py</a> script from the RPLIDAR repository.
<img alt="lidar animation polar coordinates" src="https://blog.agayon.be/images/r1d3/lidar_animation.gif" style="width: 640px; height: auto; max-width: 100%;"/></p>
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; " allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/jr0JaXfKj68?si=2G76s07Z9e4kI9Dt" title="YouTube video player" width="560"></iframe>
<h1>Streaming</h1>
<p>Last year I discovered the <a href="https://github.com/benbusby/raztot">Raznot</a> project and it inspired me. According to the readme, <em>the RazTot is an easy DIY project which allows you to remotely control a roving security camera securely from your browser</em>. After some tests, I decided to not use this project because the flask server and interface would not so nicely integrate with my R1D3 base code. I only use Janus <a href="https://janus.conf.meetecho.com/">Janus</a>, a general purpose WebRTC server to stream from the robot in a generic web page for now. Unfortunately, my <a href="https://www.microsoft.com/accessories/en-us/products/webcams/lifecam-studio/q2f-00013">Microsoft LifeCam Studio</a> does not produce a stream compatible with Janus. I need to transcode the MPEG-4 video stream to H264 with ffmpeg to see it in a browser. I hope to be able to release the code in the following weeks.</p>
<p>More to come, stay tuned !</p>
New Converse plugin: Verify HTTP requests with XMPP2019-09-15T12:30:00+02:002019-09-15T12:30:00+02:00Arnaudtag:blog.agayon.be,2019-09-15:/converse_xep_0070.html
<p><a href="https://conversejs.org/">Converse</a> is a JavaScript XMPP client that can be run in a web browser. Unfortunately, it does not support the XMPP extension (XEP) that allows to verify HTTP Requests via XMPP (XMPP): <a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a>. So I decided to code a <a href="https://gitlab.com/jnanar/converse-http-auth">small plugin</a> to provide this functionality.</p>
<p>This article follows the serie about this XEP:</p>
<ul>
<li>One about <a href="https://blog.agayon.be/xmpp_auth_django.html">Passordless Authentication using XMPP on a Django website</a>.</li>
<li>A <a href="https://blog.agayon.be/xmpp_auth_django_demo.html">presentation about a demo website</a> providing this functionnality.</li>
</ul>
<p><img alt="screenshot" src="https://blog.agayon.be/images/http_auth.png" style="width: 1080px; height: auto; max-width: 100%;"/></p>
<p><a href="https://conversejs.org/">Converse</a> is a JavaScript XMPP client that can be run in a web browser. Unfortunately, it does not support the XMPP extension (XEP) that allows to verify HTTP Requests via XMPP (XMPP): <a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a>. So I decided to code a <a href="https://gitlab.com/jnanar/converse-http-auth">small plugin</a> to provide this functionality.</p>
<p>This article follows the serie about this XEP:</p>
<ul>
<li>One about <a href="https://blog.agayon.be/xmpp_auth_django.html">Passordless Authentication using XMPP on a Django website</a>.</li>
<li>A <a href="https://blog.agayon.be/xmpp_auth_django_demo.html">presentation about a demo website</a> providing this functionnality.</li>
</ul>
<p><img alt="screenshot" src="https://blog.agayon.be/images/http_auth.png" style="width: 1080px; height: auto; max-width: 100%;"/></p>
<h2>How to use it</h2>
<p>See the <a href="https://m.conversejs.org/docs/html/plugin_development.html">official documentation</a> on how to install this plugin.</p>
<p>If you want to hide the authentication request of your provider, you can set the following option in your <code>converse.initialize</code> function:</p>
<p><code>hidden: ['auth.example.com']</code></p>
<p>It will also work with any jid. </p>
<p>You can test the implementation with my demo website: <a href="https://demo.agayon.be/">https://demo.agayon.be/</a></p>
<h1>Links</h1>
<ul>
<li><a href="https://gitlab.com/jnanar/converse-http-auth">Code on Gitlab</a></li>
<li><a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a></li>
<li><a href="https://blog.agayon.be/xmpp_auth_django.html">Django authentification</a></li>
<li><a href="https://blog.agayon.be/xmpp_auth_django_demo.html">Demo</a></li>
<li><a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">Authentication component by Chteufleur</a></li>
<li><a href="https://linuxfr.org/news/authentifiez-vous-sans-mot-de-passe-grace-a-xmpp">Description of the mechanism</a> (french)</li>
</ul>The rise of the machine2019-05-11T20:00:00+02:002019-05-11T20:00:00+02:00Arnaudtag:blog.agayon.be,2019-05-11:/rise.html
<h1>Hurrah !</h1>
<p>After months of work on the Agayon, I can present some significant improvements ! This article is a little bit longer than the previous ones but it worth the read! </p>
<h1>Software updates !</h1>
<p>During the past few weeks, the code base of the Agayon has been updated. I forked my own project, r1d2 to update it. The new repository is named <a href="https://gitlab.com/r1d3/rpi">r1d3</a>. I hesitated a long time before forking it. As the hardware base of the Agayon completely changed, I preferred to change the code name to maintain coherence between hardware and software.</p>
<p>The update aims to provide</p>
<ul>
<li>Python 3 support only</li>
<li>OpenCV 4 support for<ul>
<li>Face recognition</li>
<li>Sign tracking</li>
<li>Face/hand detection and tracking</li>
</ul>
</li>
<li>Better XMPP ad hoc support</li>
<li><a href="https://en.wikipedia.org/wiki/I%C2%B2C">I2C</a> support</li>
<li>Hardware switches support. </li>
</ul>
<h2>XMPP Ad hoc commands</h2>
<p>As I made tests with SLeekXMPP to control the bot, I observed some problems with Gajim. The <a href="https://xmpp.org/extensions/xep-0050.html">Ad-Hoc</a> extension allows one to send commands to an XMPP bot. R1D3 displays the following menus and submenus (in french):</p>
<p><img alt="menu1" src="https://blog.agayon.be/images/r1d3/rise/menu1.png.webp" style="width: 584px; height: auto; max-width: 100%;"/>
<img alt="menu2" src="https://blog.agayon.be/images/r1d3/rise/menu2.png.webp" style="width: 583px; height: auto; max-width: 100%;"/>
<img alt="menu3" src="https://blog.agayon.be/images/r1d3/rise/menu3.png.webp" style="width: 584px; height: auto; max-width: 100%;"/></p>
<p>When I try to use the "execute" button, SleekXMPP start a new session and Gajim complains that the session identifier has changed. I reported the problem to <a href="https://github.com/fritzy/SleekXMPP/issues/503">SleekXMPP</a> and its fork <a href="https://lab.louiz.org/poezio/slixmpp/issues/3432">SliXMPP</a>. The XMPP community is great and Maxime Buquet responded quickly. To quote him, there are two problems (see the bug report for the whole explanations):</p>
<blockquote>
<ul>
<li>Slixmpp shouldn't assume execute is the start of a command</li>
<li>I don't see a place in the XEP that says that next or execute can be equivalent to complete. What to do?</li>
</ul>
</blockquote>
<p><a href="https://mail.jabber.org/pipermail/standards/2019-April/036071.html">He sent an email on the "Standards" mailing list</a> and some responses followed. It seems difficult to fix the protocol at the moment without breaking compatibility. Maxime proposed a <a href="https://lab.louiz.org/poezio/slixmpp/merge_requests/13">patch to fix Slixmpp</a> and it should work on SleekXMPP. For now, I just don't use the "Execute" button as "Forward" does the job. The depreciation of the "Execute action" is actually <a href="https://mail.jabber.org/pipermail/standards/2018-September/035359.html">discussed</a>.</p>
<h1>New hardware !</h1>
<p>The Agayon has now 8 LEDs and 6 switches. They are placed on a control panel.</p>
<p>The LEDs aim to provide status information</p>
<ul>
<li>5V power (orange)</li>
<li>Pi powered up (green)</li>
<li>I2C on the Arduino (green)</li>
<li>I2C + serial on the Pi (green)</li>
<li>Serial communication (green, Arduino)</li>
<li>Video capture (red)</li>
<li>Internet connection (blue)</li>
<li>LIDAR mapping (red)</li>
</ul>
<p>The switches aim to provide</p>
<ul>
<li>Power on (12V) (<a href="https://www.ebay.com/itm/20PC-Led-Dot-Light-12V-Car-Auto-Boat-Round-Rocker-ON-OFF-Toggle-SPST-Switch-LIOK/254014300149?hash=item3b246e9ff5:g:4e8AAOSwKM1cCOos">Ebay</a>)</li>
<li>Start R1D3 (<a href="https://shop.mchobby.be/fr/bouton/255-rocker-vert-3232100002555.html">MCHobby</a>)</li>
<li>On/Off Demo mode (Arduino) (<a href="https://www.ebay.fr/itm/Universal-Single-Double-Pole-Toggle-Flick-Switch-15A-250VAC-SPST-SPDT-DPST-DPDT/143056852354?hash=item214eda5582:m:mJVG-Tr4ILmGlRENl3BaHzg">Ebay</a>)</li>
<li>On/Off Power down (Pi) (<a href="https://www.ebay.fr/itm/Universal-Single-Double-Pole-Toggle-Flick-Switch-15A-250VAC-SPST-SPDT-DPST-DPDT/143056852354?hash=item214eda5582:m:mJVG-Tr4ILmGlRENl3BaHzg">Ebay</a></li>
<li>Emergency stop (cut Arduino power) (<a href="https://www.ebay.fr/itm/N-O-N-C-Emergency-Stop-Switch-Push-Button-Mushroom-4-Screw-Terminals/183733271517?hash=item2ac75b3bdd:g:19MAAOSwRKtckkgD">Ebay</a>)</li>
<li>Movie recording (Pi)</li>
</ul>
<p>In addition, the following hardware are also mounted to provide information and input/output. I2C addresses are displayed (0Xxx)</p>
<ul>
<li><a href="https://shop.mchobby.be/en/breakout/212-8x8-bi-color-led-matrix-i2c-30mm-3232100002128-adafruit.html">I2C bicolor matrix 8x8</a>: 0x70</li>
<li><a href="https://shop.mchobby.be/en/afficheur-lcd-tft-oled/637-shield-lcd-rgb-pour-pi-affichage-positif-keypad-3232100006379-adafruit.html">LCD with 5 buttons</a>: 0x20</li>
<li><a href="https://www.pololu.com/product/2468">MinIMU-9 v3</a> I2C <a href="https://en.wikipedia.org/wiki/Inertial_measurement_unit">Inertial measurement unit</a> (<a href="https://getyarn.io/yarn-clip/b7654952-54ec-4cbf-b57a-afbc2663b7fe">IMU</a>): L3GD20H (gyroscope): 0x6B, LSM303D (3-axis magnetometer and 3-axis accelerometer) 0x1D</li>
<li>6 <a href="https://www.adafruit.com/product/3942">HC-SR04 Ultrasonic sonar distance sensor</a> for surrounding obstacle detection</li>
<li><a href="https://fr.farnell.com/sensirion/sht71/capteur-humidite-temp-v4/dp/1590513">SHT71 relative humidity and temperature sensor</a></li>
</ul>
<p>The <a href="http://www.mouser.com/ds/2/682/Sensirion_Humidity_SHT7x_Datasheet_V5-469726.pdf">documentation of the SHT71</a> explains why the sensor has no I2C adress.</p>
<blockquote>
<p>The serial interface of the SHT7x is optimized for sensor readout and effective power consumption. The sensor cannot be addressed by I2C protocol, however, the sensor can be connected to an I2C bus without interference with other devices connected to the bus. Microcontroller must switch between protocols.</p>
</blockquote>
<h1>One ground to rule them all</h1>
<p>I have been advised to use an epoxy base coated with a copper layer. The aim is to connect it to the negative pole of the battery. It is really useful because it decrease the wiring. The perfboards are fixed on metallics spacer bars to avoid shortcuts. </p>
<p><img alt="plate1" src="https://blog.agayon.be/images/r1d3/rise/plate1.JPG.webp" style="width: 270px; height: auto; max-width: 100%;"/></p>
<h1>I2C Scans</h1>
<p><a href="https://en.wikipedia.org/wiki/I%C2%B2C">I²C</a> is a bus communication that allows multiple device to communicate with each other.</p>
<p>I2C devices are recognized by the Arduino (5V) and the Raspberry PI (3.3V) with the help of a <a href="https://shop.mchobby.be/en/breakout/131-4-channels-logic-level-converter-bi-directionnal-i2c-compatible-3232100001312-adafruit.html">level shifter</a>.</p>
<p>I've used the <a href="https://playground.arduino.cc/Main/I2cScanner">I2C scanner provided by the Arduino documentation</a>.</p>
<h2>Arduino</h2>
<div class="highlight"><pre><span></span><code><span class="nx">Scanning</span><span class="o">...</span>
<span class="nx">I2C</span><span class="w"> </span><span class="nx">device</span><span class="w"> </span><span class="nx">found</span><span class="w"> </span><span class="nx">at</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="mh">0x1D</span><span class="w"> </span><span class="p">!</span>
<span class="nx">I2C</span><span class="w"> </span><span class="nx">device</span><span class="w"> </span><span class="nx">found</span><span class="w"> </span><span class="nx">at</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="mh">0x20</span><span class="w"> </span><span class="p">!</span>
<span class="nx">I2C</span><span class="w"> </span><span class="nx">device</span><span class="w"> </span><span class="nx">found</span><span class="w"> </span><span class="nx">at</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="mh">0x6B</span><span class="w"> </span><span class="p">!</span>
<span class="nx">I2C</span><span class="w"> </span><span class="nx">device</span><span class="w"> </span><span class="nx">found</span><span class="w"> </span><span class="nx">at</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="mh">0x70</span><span class="w"> </span><span class="p">!</span>
<span class="nx">done</span>
</code></pre></div>
<h2>Raspberry PI</h2>
<div class="highlight"><pre><span></span><code><span class="c">user$ i2cdetect </span><span class="nb">-</span><span class="c">y 1</span>
<span class="c"> 0 1 2 3 4 5 6 7 8 9 a b c d e f</span>
<span class="c">00: </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">10: </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> 1d </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">20: 20 </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">30: </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">40: </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">50: </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">60: </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> 6b </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
<span class="c">70: 70 </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span><span class="c"> </span><span class="nb">--</span>
</code></pre></div>
<h1>Gallery</h1>
<h2>"Scaffolding" with hot glue</h2>
<p>During the past few months, my best friend has been my hot glue gun. I was skeptical at first but really much effective and fun. I used it to insulate some connectors. In <a href="https://en.wikipedia.org/wiki/Li%C3%A8ge">Liège</a>, we would say "mettre une noquette de colle" which translates to "put a knob of glue".</p>
<p><img alt="glue1" src="https://blog.agayon.be/images/r1d3/rise/glue1.JPG.webp" style="width: 270px; height: auto; max-width: 100%;"/>
<img alt="glue3" src="https://blog.agayon.be/images/r1d3/rise/glue3.JPG.webp" style="width: 270px; height: auto; max-width: 100%;"/></p>
<p><img alt="capot2" src="https://blog.agayon.be/images/r1d3/rise/capot2.JPG.webp" style="width: 270px; height: auto; max-width: 100%;"/>
<img alt="capot3" src="https://blog.agayon.be/images/r1d3/rise/capot3.JPG.webp" style="width: 270px; height: auto; max-width: 100%;"/></p>
<h2>Evolution of the frame</h2>
<p><img alt="full_base" src="https://blog.agayon.be/images/r1d3/rise/full_base.JPG.webp" style="width: 270px; height: auto; max-width: 100%;"/>
<img alt="connection_base_capot" src="https://blog.agayon.be/images/r1d3/rise/connection_base_capot.JPG.webp" style="width: 640px; height: auto; max-width: 100%;"/></p>
<p><img alt="complete_1" src="https://blog.agayon.be/images/r1d3/rise/complete_1.jpg.webp" style="width: 800px; height: auto; max-width: 100%;"/></p>
<p><img alt="complete_3" src="https://blog.agayon.be/images/r1d3/rise/complete_3.jpg.webp" style="width: 800px; height: auto; max-width: 100%;"/>
<img alt="complete_4" src="https://blog.agayon.be/images/r1d3/rise/complete_4.jpg.webp" style="width: 1000px; height: auto; max-width: 100%;"/></p>
<ul>
<li>A : Battery</li>
<li>B : Level shifter between Arduino (5V) and Raspberry Pi (3.3V)</li>
<li>C : Arduino Mega</li>
<li>D : Power lines and I2C (12V, 5 V, 3.3V, SDA 5V, SCL 5V, SDA 3.3V, SCL 3.3V)</li>
<li>E : Raspberry Pi (in his case)</li>
<li>F : Buttons and their pull down (3.3V or 5V depending on the GPIO)</li>
<li>G : LEDs</li>
</ul>
<h2>New (old) Oscilloscope</h2>
<p>One of my colleague has been cleaning his lab, and he asked me if I was interested to have an old 20 MHz oscilloscope. I gladly accepted. It is a 34 years old Circuitmate 9020 (bought in 1985).
I will use it for I2C debugging and visualization.
<img alt="oscillo" src="https://blog.agayon.be/images/r1d3/rise/oscillo.JPG" style="width: 640px; height: auto; max-width: 100%;"/></p>
<h1>Conclusion</h1>
<p>The hardware is almost done. I am happy to have a nice reliable base. I hope to be able to drive it with my smartphone soon. I will continue the programming to add the mapping functionality and a nice demo mode.</p>
<p>Stay tuned !</p>
<p><img alt="smile" src="https://blog.agayon.be/images/r1d3/rise/smile.JPG" style="width: 640px; height: auto; max-width: 100%;"/></p>
Back on tracks !2019-01-06T20:00:00+01:002019-01-06T20:00:00+01:00Arnaudtag:blog.agayon.be,2019-01-06:/back_on_tracks.html
<p>The Agayon is back on tracks ! I had a lot to deal with in 2018 but I finally found time to move forward on the Agayon.</p>
<p>The Agayon is back on tracks ! I had a lot to deal with in 2018 but I finally found time to move forward on the Agayon.</p>
<p>The chassis was not satisfactory and I decided to start over on a wine crate painted in black. This design is much stronger and the base is robust. I was helped by a colleague who was able to build a nice wheel fastening system.</p>
<p><img alt="Axe" src="https://blog.agayon.be/images/r1d3/axe.jpg" style="width: 992px; height: auto; max-width: 100%;"/>
<img alt="Frame" src="https://blog.agayon.be/images/r1d3/frame.jpg" style="width: 795px; height: auto; max-width: 100%;"/></p>
<p>In the <a href="https://blog.agayon.be/work_in_progress_3.html">last episode</a>, I presented my homemade Lidar system. It worked great inside my house but unfortunately, the optical rotary encoder was saturated with sun light. It was impossible to operate. We discussed the problem with some friend and concluded that the optical encoder could be removed if the angle is deduced from the stepper motor position. Time passed and roboshop presented a <a href="https://www.robotshop.com/eu/en/rplidar-a1m8-360-degree-laser-scanner-development-kit.html">new RPlidar system</a>. It is cheaper than my <a href="https://www.robotshop.com/eu/en/lidar-lite-3-laser-rangefinder.html">LIDAR-Lite 3</a>. I decided to give it a go. It will be easier to get accurate measurements with this one. Its sample frequency goes from 4000Hz to 8000Hz. My custom system mad a measurement every 500 ms. As the LIDAR-Lite 3 is equiped with a servo motor, I will try to map in 3D. Both lidar will be useful to check the accuracy of the measurement and debug my code.</p>
<p><img alt="rplidar" src="https://blog.agayon.be/images/r1d3/rplidar.jpg" style="width: 645px; height: auto; max-width: 100%;"/></p>
<p>There is a <a href="https://github.com/SkoltechRobotics/rplidar">nice python module</a> for the RPlidar and it works out of the box with the <a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> library. The following picture displays my living room.</p>
<p><img alt="SLAM" src="https://blog.agayon.be/images/r1d3/map.png" style="width: 620px; height: auto; max-width: 100%;"/></p>
<p>My homemade mapping system:</p>
<video controls="controls" height="480" src="/images/lidar.mp4" width="640">Homemade mapping system.</video>
<h1>Future</h1>
<ul>
<li>In the next weeks, I would like to connect the motors, motor driver and Arduino.</li>
<li>The code needs to be written to get information about the rotary encoder of the <a href="https://www.pololu.com/product/2826">Polulu motors</a>.</li>
<li>The Raspberry PI will be connected to the system and it will acquire the data with a serial connection. As the interrupt pins are perturbed by the serial connection, data will be sent when the robot is stopped.</li>
</ul>Fail2Ban analysis2018-12-05T19:00:00+01:002018-12-05T21:00:00+01:00Arnaudtag:blog.agayon.be,2018-12-05:/fail2ban.html
<p>Running a server on the internet is quite an adventure but it's not always easy to keep track of log files, security and potential threads. This article describes some actions carried out on this server to mitigate <a href="https://en.wikipedia.org/wiki/Script_kiddie">script kiddies</a>.</p>
<p>Running a server on the internet is quite an adventure but it's not always easy to keep track of log files, security and potential threads. This article describes some actions carried out on this server to mitigate <a href="https://en.wikipedia.org/wiki/Script_kiddie">script kiddies</a>.</p>
<p>Among the good practices, I have </p>
<ul>
<li>a subscription to the Debian security mailing list, </li>
<li>automatic security updates (enabled by default with the Scaleway Debian images)</li>
<li>A monitoring tool (<a href="http://munin-monitoring.org/">Munin</a>) which provides useful graphs to watch the activity of the server.</li>
<li><a href="https://www.fail2ban.org/wiki/index.php/Main_Page">Fail2Ban</a></li>
<li>Backups</li>
</ul>
<p>The following paragraphs describe how I analyze the country IP banned by Fail2Ban.</p>
<p>As explained on their website, Fail2Ban scans log files and bans IPs that show the malicious signs: too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, mail, ssh, etc). </p>
<p>Recently, I had the need to check if Belgian IP were blacklisted. Most of my users are Belgian and one of my Fail2Ban rules was too strict. I decided to log the IP in a file to perform a geolocalisation analysis to detect and prevent false positives.</p>
<p>The <a href="https://github.com/bcambl/fail2ban-blacklist">fail2ban-blacklist</a> script was used to log blacklisted IP into a CSV file. The analysis is performed on another computer.</p>
<h1>Scripts</h1>
<p>The following script is called <code>generate_report.py</code>. It read a <code>CSV</code> file that has several information about bans: the date, time and IP. The country IP are discovered with the whois information thanks to a script. Finally, a barplot is generated to visualize the amount of hits per country. The whole process is launched with the <code>report.sh</code> script.</p>
<h2>generate_report.py</h2>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">matplotlib</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span>
<span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">,</span>
<span class="nb">format</span><span class="o">=</span><span class="s2">"[</span><span class="si">%(asctime)s</span><span class="s2">] </span><span class="si">%(levelname)s</span><span class="s2"> </span><span class="si">%(message)s</span><span class="s2">"</span><span class="p">,</span>
<span class="n">datefmt</span><span class="o">=</span><span class="s2">"%H:%M:%S"</span><span class="p">,</span>
<span class="n">stream</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_country</span><span class="p">(</span><span class="n">ip</span><span class="p">):</span>
<span class="n">whois</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s1">'whois'</span><span class="p">,</span> <span class="n">ip</span><span class="p">],</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
<span class="n">whois</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="c1"># Prevent problems if output is not utf8</span>
<span class="n">str_whois</span> <span class="o">=</span> <span class="n">whois</span><span class="o">.</span><span class="n">communicate</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
<span class="n">find_country</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'country:(.*)'</span><span class="p">,</span> <span class="n">str_whois</span><span class="p">)</span>
<span class="k">if</span> <span class="n">find_country</span><span class="p">:</span>
<span class="k">return</span> <span class="n">find_country</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">"NONE"</span>
<span class="k">def</span> <span class="nf">plot_graph</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="p">)</span>
<span class="n">df</span><span class="o">.</span><span class="n">country</span><span class="o">.</span><span class="n">value_counts</span><span class="p">()</span><span class="o">.</span><span class="n">plot</span><span class="o">.</span><span class="n">bar</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">'Country'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">'Counts'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s1">'Counts of blacklisted countries'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="s2">"fail2ban_report.png"</span><span class="p">,</span> <span class="n">dpi</span><span class="o">=</span><span class="mi">150</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s1">'w'</span><span class="p">,</span> <span class="n">edgecolor</span><span class="o">=</span><span class="s1">'w'</span><span class="p">,</span>
<span class="n">orientation</span><span class="o">=</span><span class="s1">'portrait'</span><span class="p">,</span> <span class="n">papertype</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">transparent</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">bbox_inches</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">pad_inches</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span>
<span class="n">frameon</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">report</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'blacklist.pkl'</span>
<span class="n">df_backup</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">df_csv</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">df</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">df_concat</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
<span class="n">df_backup</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_pickle</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="sa">r</span><span class="s1">'blacklist.csv'</span>
<span class="n">df_csv</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"UTF-8"</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span> <span class="n">engine</span><span class="o">=</span><span class="s1">'python'</span><span class="p">,</span> <span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">df_backup</span><span class="p">)</span> <span class="o"><</span> <span class="nb">len</span><span class="p">(</span><span class="n">df_csv</span><span class="p">):</span>
<span class="n">df_tmp</span> <span class="o">=</span> <span class="n">df_backup</span><span class="p">[[</span><span class="s1">'DATE'</span><span class="p">,</span> <span class="s1">'TIME'</span><span class="p">,</span> <span class="s1">'IP_ADDRESS'</span><span class="p">]]</span>
<span class="n">df_concat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">concat</span><span class="p">([</span><span class="n">df_csv</span><span class="p">,</span> <span class="n">df_tmp</span><span class="p">])</span><span class="o">.</span><span class="n">drop_duplicates</span><span class="p">(</span><span class="n">keep</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">df_concat</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">""</span>
<span class="c1"># df = df_backup.append(df_concat,sort=False)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Append </span><span class="si">{}</span><span class="s2"> lines"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">df_concat</span><span class="p">)))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No difference between backup and CSV"</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df_backup</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">friend_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1"># df_friends = pd.DataFrame(columns=list(df_backup.columns.values))</span>
<span class="n">df_friends</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_pickle</span><span class="p">(</span><span class="s2">"blacklist_friends.pkl"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">df_concat</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">for</span> <span class="n">idx</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">df_concat</span><span class="o">.</span><span class="n">iterrows</span><span class="p">():</span>
<span class="c1"># Do not process the dataframe multiples times.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">row</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Process IP : </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s1">'IP_ADDRESS'</span><span class="p">])))</span>
<span class="n">country</span> <span class="o">=</span> <span class="n">get_country</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s1">'IP_ADDRESS'</span><span class="p">]))</span>
<span class="n">df_concat</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">idx</span><span class="p">,</span> <span class="s1">'country'</span><span class="p">]</span> <span class="o">=</span> <span class="n">country</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">country</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]</span>
<span class="c1"># country == 'Be' do not work</span>
<span class="n">country</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">country</span><span class="p">)</span>
<span class="k">if</span> <span class="n">country</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'be'</span><span class="p">,</span> <span class="s1">'other_friendly_country'</span><span class="p">]:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="n">row</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]</span> <span class="o">=</span> <span class="n">country</span>
<span class="n">friend_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df_backup</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">df_concat</span><span class="p">,</span> <span class="n">sort</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Append friends"</span><span class="p">)</span>
<span class="n">df_friends</span> <span class="o">=</span> <span class="n">df_friends</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">friend_list</span><span class="p">)</span>
<span class="n">df_friends</span><span class="o">.</span><span class="n">to_pickle</span><span class="p">(</span><span class="s2">"./blacklist_friends.pkl"</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">IndexError</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No friend to append"</span><span class="p">)</span>
<span class="n">df</span><span class="o">.</span><span class="n">to_pickle</span><span class="p">(</span><span class="s2">"./blacklist.pkl"</span><span class="p">)</span>
<span class="n">plot_graph</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">report</span><span class="p">()</span>
</code></pre></div>
<h2>report.sh</h2>
<p>This script copy the Fail2Ban CSV file from the server (whois requests are forbidden on my VPS), generate the data and display the bar plot with the help of <code>typop</code>, a built-in function of <a href="https://www.enlightenment.org/about-terminology">terminology</a>, a great terminal emulator for Linux/BSD/UNIX systems.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/sh</span>
scp<span class="w"> </span>user@agayon.be:/etc/fail2ban/report.py<span class="w"> </span>.
python<span class="w"> </span>generate_report.py
typop<span class="w"> </span>fail2ban_report.png
<span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<h2>Graphs</h2>
<p>This graph was generated on 5 of december 2018. Some countries are more represented but the threat is global.</p>
<p><img alt="Fail2Ban report graph" src="https://blog.agayon.be/images/fail2ban_report.png" style="width: 2400px; height: auto; max-width: 100%;"/></p>
<p>The following graphs is generated by Munin. It display the number of ban per jail.</p>
<p><img alt="Fail2Ban graph week" src="https://blog.agayon.be/images/fail2ban-week.png" style="width: 897px; height: auto; max-width: 100%;"/></p>
<h1>Links</h1>
<ul>
<li><a href="https://www.fail2ban.org/wiki/index.php/Main_Page">Fail2Ban</a></li>
<li><a href="https://github.com/bcambl/fail2ban-blacklist">bcambl/fail2ban-blacklist</a></li>
<li><a href="http://munin-monitoring.org/">Munin</a></li>
</ul>Upgrading to Odoo 122018-11-08T11:00:00+01:002019-02-23T12:00:00+01:00Arnaudtag:blog.agayon.be,2018-11-08:/odoo_12.html
<p><a href="https://www.odoo.com/fr_FR/blog/notre-blog-5/post/odoo-12-a-mature-business-management-software-515">Odoo 12.0</a> is out since october. I am currently investigating the differences with previous versions to update the instance of the association <a href="http://www.compagnonsducep.be/">Les Compagnons du CEP</a>. A lot of changes have been made in a few years but the workflow stays about the same. This article describes my workflow, the backup policy, how a module was used and fixed to restore a missing feature. Finally, the changes in my custom product import function are presented. </p>
<p><a href="https://www.odoo.com/fr_FR/blog/notre-blog-5/post/odoo-12-a-mature-business-management-software-515">Odoo 12.0</a> is out since october. I am currently investigating the differences with previous versions to update the instance of the association <a href="http://www.compagnonsducep.be/">Les Compagnons du CEP</a>. A lot of changes have been made in a few years but the workflow stays about the same. This article describes my workflow, the backup policy, how a module was used and fixed to restore a missing feature. Finally, the changes in my custom product import function are presented. </p>
<h1>Production and debug setup</h1>
<p>Agayon.be instance of Odoo runs inside a Docker container on a small VPS. Postgresql is installed as a core package. This setup is great in production but it is difficult to debug some python code with this configuration. The first step is to run Odoo with an IDE (I personally use the excellent <a href="https://www.jetbrains.com/pycharm/">Pycharm</a>). </p>
<h2>Running and debugging Odoo</h2>
<p>The first step is the creation of the virtualenv. Once the <a href="https://wiki.archlinux.org/index.php/Odoo#Configuring_PostgreSQL_to_run_with_Odoo">Postgresql instance is ready</a>, you can build the environment.</p>
<div class="highlight"><pre><span></span><code>git clone https://www.github.com/odoo/odoo --depth 1 --branch 12.0
$ virtualenv myenv
$ source myenv/bin/activate
(myenv)$ pip install -r odoo/requirements.txt
(myenv)$ mkdir custom-addons
(myenv)$ chown odoo: odoo11-custom-addons
</code></pre></div>
<p>Edit odoo.conf and then run odoo:</p>
<div class="highlight"><pre><span></span><code>odoo/odoo-bin -c odoo.conf
</code></pre></div>
<h1>Merge purchase order</h1>
<p><strong>UPDATE: 23/02/2019: The following paragraph is not needed anymore. Odoo 12 restored the automatic merge of purchase orders.</strong> </p>
<p>Unfortunately, a major feature has disappeared in version 12.0: automatic merge of purchase order. I don't know why since it seems unrealistic to send each quotation separately to your vendor. Fortunately a <a href="https://github.com/odooaktiv/MergePurchaseOrder/">free module</a> can be used to perform the merge but it has a critical bug. Some quotation lines are merged even <a href="https://github.com/odooaktiv/MergePurchaseOrder/issues/2">if they concerns different products</a>. </p>
<h2>Fix</h2>
<p>After <a href="https://github.com/jarobase/MergePurchaseOrder">forking</a> it, I decided to start by refactoring it. The module is quite small but a lot of code is duplicated. As I try to avoid <a href="https://en.wikipedia.org/wiki/Spaghetti_code">spaghetti code</a>, it needed to be <a href="https://github.com/jarobase/MergePurchaseOrder/commit/66f70a19255a66ee659fac429315276dc7991080">refactored</a>.I think the new code may be improved but no line is duplicated. Finally, the bug has been <a href="https://github.com/jarobase/MergePurchaseOrder/commit/fe653be0f693eb319852944f52e6330bfc33c4cf">fixed</a>.</p>
<h1>Backup management</h1>
<p>Since a few version, the <code>filestore</code> is mandatory in Odoo. If it is incoherent with the database, some really <a href="https://www.odoo.com/fr_FR/forum/aide-1/question/solved-how-to-recover-deleted-attachment-files-from-filestore-folder-local-share-odoo-filestore-128453">annoying errors are raised</a> and the solution is quite tedious. My backup procedure has been updated to avoid losing any data. It is based on the article from <a href="https://zeroheure.info/how-to-restore-an-odoo-backup-quick-help/">zeroheure</a>. The backups are saved with the <a href="https://www.odoo.com/apps/modules/12.0/auto_backup/">auto_backup</a> module. Restoring the data is not possible with the web interface because the process reaches the memory limit but it can be performed with the following shell script.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">BACKUPLOCATION</span><span class="o">=</span><span class="s2">"/path/to/backup.zip"</span>
<span class="nv">DBNAME</span><span class="o">=</span><span class="s2">"db_name"</span>
<span class="nv">FILESTORE_DIR</span><span class="o">=</span><span class="s2">"/path/to/filestore"</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$FILESTORE_DIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$DBNAME</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$BACKUPLOCATION</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"verify your variables"</span>
<span class="k">else</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$BACKUPLOCATION</span>
<span class="w"> </span>rm<span class="w"> </span>filestore<span class="w"> </span>
<span class="w"> </span>unzip<span class="w"> </span>-q<span class="w"> </span><span class="nv">$DBNAME</span>.zip
<span class="w"> </span>cp<span class="w"> </span>-r<span class="w"> </span>filestore<span class="w"> </span><span class="nv">$DBNAME</span>
<span class="w"> </span>sudo<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="nv">$FILESTORE_DIR</span>/<span class="nv">$DBNAME</span>
<span class="w"> </span>sudo<span class="w"> </span>mv<span class="w"> </span><span class="nv">$DBNAME</span><span class="w"> </span><span class="nv">$FILESTORE_DIR</span>
<span class="w"> </span>sudo<span class="w"> </span>chown<span class="w"> </span>-R<span class="w"> </span>odoo:odoo<span class="w"> </span><span class="nv">$FILESTORE_DIR</span>/<span class="nv">$DBNAME</span>
<span class="w"> </span>dropdb<span class="w"> </span>-U<span class="w"> </span>odoo<span class="w"> </span><span class="nv">$DBNAME</span>
<span class="w"> </span>createdb<span class="w"> </span>-U<span class="w"> </span>odoo<span class="w"> </span><span class="nv">$DBNAME</span>
<span class="w"> </span>psql<span class="w"> </span><span class="nv">$DBNAME</span><span class="w"> </span>--quiet<span class="w"> </span><<span class="w"> </span>dump.sql
<span class="k">fi</span>
</code></pre></div>
<h1>Wine import with Django website</h1>
<p>Version 12.0 needs some minor changes in the code displayed in the <a href="https://blog.agayon.be/pandas_odoo.html">previous article</a>.
These modifications includes:</p>
<ul>
<li>removing state property in the product template.</li>
<li>Adding the invoice_policy and purchase_method in the product template.[ref]See <a href="https://www.odoo.com/fr_FR/forum/aide-1/question/access-sale-config-settings-value-from-sale-order-117728">this post</a> and this <a href="https://github.com/camptocamp/odoo-dj/issues/101">issue</a> [/ref]</li>
<li>Add a reference to the standard price in the product_supplierinfo dictionary. This value is used in the orders when purchasing wines to suppliers.</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">):</span>
<span class="c1"># 1) search if default code is used?)</span>
<span class="c1"># 2) search if suppliers is in the database</span>
<span class="c1"># 3) search if the name is already used</span>
<span class="c1"># Retrieve the dataframes. This example comes from a jupyter nootebook.</span>
<span class="c1"># df_product and df_sellers are already defined.</span>
<span class="c1"># In a real case, we should use class variables.</span>
<span class="n">n_code</span><span class="p">,</span> <span class="n">df_code</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">default_code</span><span class="p">,</span>
<span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_supplier</span><span class="p">,</span> <span class="n">df_suppliers</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_suppliers</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'function'</span><span class="p">,</span>
<span class="n">search_item</span><span class="o">=</span><span class="n">supplier_code</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_name</span><span class="p">,</span> <span class="n">df_name</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'name'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">wine_name</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># ids_product = list(df_code['id'])</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">df_code</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_supplier</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'success'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Another product uses the same name with another code</span>
<span class="k">return</span> <span class="s1">'e_code_used_same_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># the code is already used</span>
<span class="k">return</span> <span class="s1">'e_code_used'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_supplier</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Cannot find the supplier</span>
<span class="k">return</span> <span class="s1">'e_missing_seller'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># A product with the same name and another code exists.</span>
<span class="k">return</span> <span class="s1">'e_code_used_different_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">def</span> <span class="nf">import2odoo</span><span class="p">():</span>
<span class="n">route_warehouse0_mto</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">route_warehouse0_manufacture</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># iterate over all rows, read the cells and assign the wine parameters to variables</span>
<span class="c1"># each row correspond to one wine</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">seller_name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">do_import</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="n">standard_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span>
<span class="n">list_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span>
<span class="n">seller_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span>
<span class="n">res_search</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span> <span class="o">=</span> <span class="n">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">seller_code</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res_search</span> <span class="o">==</span> <span class="s1">'success'</span> <span class="ow">and</span> <span class="n">do_import</span> <span class="o">==</span> <span class="s2">"1"</span><span class="p">:</span>
<span class="n">product_template</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'active'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'standard_price'</span><span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="s1">'list_price'</span><span class="p">:</span> <span class="n">list_price</span><span class="p">,</span>
<span class="s1">'description'</span><span class="p">:</span> <span class="n">comment</span> <span class="p">,</span>
<span class="s1">'default_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span>
<span class="s1">'purchase_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'sale_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_po_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'product'</span><span class="p">,</span>
<span class="s1">'cost_method'</span><span class="p">:</span> <span class="s1">'standard'</span><span class="p">,</span>
<span class="s1">'invoice_policy'</span> <span class="p">:</span> <span class="s1">'order'</span> <span class="p">,</span> <span class="c1"># ordered quantities</span>
<span class="s1">'purchase_method'</span> <span class="p">:</span> <span class="s1">'receive'</span><span class="p">,</span> <span class="c1"># control received quantities (or ordered ones, test to delivery)</span>
<span class="s1">'route_ids'</span><span class="p">:</span> <span class="p">[(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="n">route_warehouse0_mto</span><span class="p">,</span> <span class="n">route_warehouse0_manufacture</span><span class="p">])]</span>
<span class="p">}</span>
<span class="c1"># For each wine, a template must be created</span>
<span class="n">template_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.template'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span>
<span class="n">product_template</span><span class="p">)</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'product_code'</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="c1"># code for supplier</span>
<span class="s1">'product_name'</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">15</span><span class="p">],</span> <span class="c1"># name for supplier</span>
<span class="s1">'min_qty'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'delay'</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="s1">'price'</span> <span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span>
<span class="s1">'product.supplierinfo'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span> <span class="n">product_supplierinfo</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Wine </span><span class="si">{}</span><span class="s2"> : </span><span class="si">{}</span><span class="s2"> has been added"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.</span>
</code></pre></div>
<h1>Conclusions</h1>
<p>After using Odoo in production for 4 years, I still love it. My users are happy to use it daily and the improvements over the year are impressive. Like any other alive project, some changes may break the workflow but Odoo and it's community make it relatively simple to adapt with plugins. Odoo 12 is strong and I look forward to use it for the next 3 years :-).</p>
<h1>Links</h1>
<ul>
<li><a href="https://www.odoo.com/">Odoo</a></li>
<li><a href="https://zeroheure.info/how-to-restore-an-odoo-backup-quick-help/">Backup management</a></li>
<li><a href="https://zeroheure.info/how-to-restore-an-odoo-backup-quick-help/">zeroheure backup quick help</a></li>
</ul>3615 MyLife2018-07-19T10:20:00+02:002018-07-19T10:21:00+02:00Arnaudtag:blog.agayon.be,2018-07-19:/algoo.html
<p>This year, the holidays and city trip was the occasion to meet Damien Accorsi, founder of <a href="https://www.algoo.fr/">Algoo SAS</a> and his team.</p>
<p>This summer, I had the opportunity to meet Damien Accorsi in Moirans near Grenoble. He is the founder of <a href="https://www.algoo.fr/">Algoo SAS</a>, a company that provides software development services and Tracim. <a href="https://github.com/tracim/tracim">Tracim</a> is <em>a collaborative software designed to allow people to share and work on various data and document types</em>. </p>
<p>Everything started from a post on <a href="https://linuxfr.org/forums/general-hors-sujets/posts/vacances-region-de-grenoble">LinuxFR</a>. I stayed near Grenoble for a week in the beginning of July and therefore, I asked the community about nice activities to do in the region. Damien answered quite quickly and made some useful suggestions. We never talked before. He also suggested meeting in his startup in Moirans. I happily accepted and we have met in his quarter. We talked about his activities. If you speak French, I suggest his instructive <a href="https://linuxfr.org/users/lebouquetin">posts in LinuxFR</a>. We talked about our projects, we laugh and had a really good time. His employees are really nice and fun. When I left, I promised myself to make more IRL meetings in the future.</p>
<p>Yet It was not the first time I visited a software developer. Two years ago, I have met Goffi from the <a href="https://salut-a-toi.org/">Salut-à-Toi</a> (SàT) project in Prague (Czech Republic). We had nice conversations and exchange about XMPP, the link between communication tools and politics, the struggle of developers to take part in open source project during free time, building a community with limited resources, etc. From this exchange started a nice collaboration on his tool. To this day, I write the PKGBUILD (packages) of SàT for <a href="https://aur.archlinux.org/packages/?K=jnanar&SeB=m">Archlinux</a>. </p>
<p>I hope to meet other people during holidays and events like FOSDEM. It is nice to put a face on a nickname. Maybe next time it will be an inventor or an artist.</p>
<p>In the meantime, if you go to Grenoble, according to Damien and myself, you should try:</p>
<ul>
<li><a href="http://www.vertige38.com/spip.php?rubrique3">bungee jumping</a></li>
</ul>
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;" allowfullscreen="" fetchpriority="high" frameborder="0" height="315" src="https://www.youtube.com/embed/B8vpDMUac5Y?si=NgS1rlIpx7386eEW" title="YouTube video player" width="560"></iframe>
<ul>
<li><a href="https://www.hikideas.com/walk-himalayan-footbridges-of-monteynard/">hiking on the Himalayan footbridges of Monteynard</a>
<img alt="Monteynard" src="https://blog.agayon.be/images/lac_monteynard.JPG" style="width: 560px; height: auto; max-width: 100%;"/></li>
<li>Chill at the "Grand lac de Laffrey".
<img alt="Laffrey" src="https://blog.agayon.be/images/lac_laffrey.JPG" style="width: 560px; height: auto; max-width: 100%;"/></li>
<li>Watch Belgium beats Brazil at Football. \o/</li>
</ul>
Demo Website: Authentication with XMPP2018-03-18T19:00:00+01:002018-03-18T19:00:00+01:00Arnaudtag:blog.agayon.be,2018-03-18:/xmpp_auth_django_demo.html
<p>This article follow up the article about <a href="https://blog.agayon.be/xmpp_auth_django.html">Authentication without password using XMPP on a Django website</a> previously presented here. This article introduced the XMPP extension (XEP-0070) which allows one to connect on a website with his XMPP account, without additional password. </p>
<p>Unfortunately, the XEP-0070 is not widely used but this article aims to present you my little contribution to change this situation.</p>
<p>This article follow up the article about <a href="https://blog.agayon.be/xmpp_auth_django.html">Authentication without password using XMPP on a Django website</a> previously presented here. This article introduced the XMPP extension (XEP-0070) which allows one to connect on a website with his XMPP account, without additional password. </p>
<p>Unfortunately, the XEP-0070 is not widely used but this article aims to present you my little contribution to change this situation.</p>
<h1>Introduction</h1>
<p>Those who know me know that I tends to have multiple projects in parallels. I have one in the back of my head since several months but it is too early to talk about it yet. I want this project to be usable with XMPP. Some resources will be accessible on a web server but I don't want to force users to manage additional passwords. This is the perfect use case of XEP-0070.[ref]<a href="https://www.goffi.org/">Goffi</a> from the <a href="https://salut-a-toi.org/">Salut à Toi </a> project is curently working on a different solution to adress these kind of challenges. [/ref] </p>
<p>In order to test the accessibility and the number of user-friendly clients that support this extension, I published a small demo website: </p>
<h1><a href="https://demo.agayon.be/">https://demo.agayon.be/</a></h1>
<p>This website is built with the famous <a href="https://www.djangoproject.com/">Django framework</a>. Moreover, the revelant code is available in the <a href="https://blog.agayon.be/xmpp_auth_django.html">previous post</a> and on the <a href="https://gitlab.com/earth_explorer/demo">Gitlab page</a>.</p>
<p>You can try the demo by clicking on the "Sign In" button. The only information needed is your XMPP account (JID). You will then receive a request on your XMPP client. If it does support the XEP, you will be invited to click in a dialog box in order to accept the connection.</p>
<p>It your client does not support it, there is a fallback mechanism where you receive a message from "auth.agayon.be". Unfortunately, The method may be tedious on some mobile clients. It is quite sad because it is the best use case of the XEP.</p>
<h1>Examples</h1>
<h2>Movim (new)</h2>
<p>Shortly after I submitted a <a href="https://github.com/movim/movim/issues/600">bug report</a> about this feature, <a href="https://nl.movim.eu/?blog/edhelas@movim.eu">edhelas</a> from the Movim project implemented it. His reactivity is remarkable. The feature is available on the <a href="https://movim.eu/#get_it">official pods</a>.</p>
<p><img alt="Movim" src="https://blog.agayon.be/images/XEP-0070-movim.png" style="width: 406px; height: auto; max-width: 100%;"/></p>
<h2>Gajim</h2>
<p><img alt="Gajim XEP-0070" src="https://blog.agayon.be/images/XEP-0070-Gajim.png" style="width: 514px; height: auto; max-width: 100%;"/></p>
<h2>Salut à toi (Primitivus)</h2>
<p><img alt="Primitivus XEP-0070" src="https://blog.agayon.be/images/XEP-0070-primitivus.png.webp" style="width: 658px; height: auto; max-width: 100%;"/></p>
<h2>Conversations</h2>
<p>Unfortunately, <a href="https://conversations.im/">Conversations</a>, one of the best mobile clients, <a href="https://github.com/siacs/Conversations/issues/1972">does not support the feature yet</a>.</p>
<p><img alt="Conversations" src="https://blog.agayon.be/images/XEP-0070-conversations.png" style="width: 384px; height: auto; max-width: 100%;"/></p>
<h1>Sources</h1>
<p>The sources are available on my <a href="https://gitlab.com/earth_explorer/demo">gitlab.com account</a>.</p>
<p>The service is still in beta. Do not hesitate to contact me if you experience some bugs with the demo.</p>
<h1>Conclusions</h1>
<p>You no longer have excuses to avoid to authenticate users on your platform with XMPP. </p>
<ul>
<li>Nice and user-friendly clients implement it.</li>
<li>The Chteufleur's component is stable and easy to use.</li>
<li>It is quite easy to add the functionality on the website running with django, <a href="https://wordpress.org/plugins/xmpp-auth/">Wordpress</a> and it should not be an insurmountable challenge with <a href="http://rubyonrails.org/">ruby on rails</a>.</li>
</ul>
<p>In the future, I will probably encourage new users to use Movim with my projects as it does not require installation and it supports the XEP.</p>
<p>I hope to be able to give you updates soon about the agayon project.</p>
<p>In the meantime, bug reports have been submitted:</p>
<ul>
<li><a href="https://github.com/siacs/Conversations/issues/1972">Conversations</a></li>
<li><a href="https://github.com/jcbrand/converse.js/issues/1020">Converse.js</a></li>
<li><a href="https://github.com/movim/movim/issues/600">Movim</a> (Already fixed as previously mentioned)</li>
</ul>
<p>Stay tuned !</p>
<h1>Links</h1>
<ul>
<li><a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a></li>
<li><a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">Authentication component by Chteufleur</a></li>
<li><a href="https://linuxfr.org/news/authentifiez-vous-sans-mot-de-passe-grace-a-xmpp">Description of the mechanism</a> (french)</li>
<li><a href="https://docs.djangoproject.com/en/1.10/topics/auth/default/">Django authentification</a></li>
<li><a href="https://blog.agayon.be/xmpp_auth_django.html">Previous article</a></li>
<li><a href="https://gitlab.com/earth_explorer/demo">Code on Gitlab</a></li>
</ul>Mixing Pandas with Odoo2018-03-07T21:00:00+01:002018-03-13T19:00:00+01:00Arnaudtag:blog.agayon.be,2018-03-07:/pandas_odoo.html
<p>This article describes the use of XML-RPC API provided by <a href="https://www.odoo.com/">Odoo</a>, a well-known ERP system. Upgrading to version 11.0 is the occasion to update my python scripts to reduce considerably the number of requests. The improvements were done with the help of <a href="https://pandas.pydata.org/">pandas</a>, the famous data structures and data analysis library.</p>
<p><a href="https://www.flickr.com/photos/ken_from_md/4697952954/"><img alt="Photo credit: Panda_3956, Ken_from_MD" src="https://blog.agayon.be/images/panda0.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a> <br/>
Photo credit: Panda_3956, <a href="https://www.flickr.com/photos/ken_from_md/">Ken_from_MD</a> </p>
<p>This article describes the use of XML-RPC API provided by <a href="https://www.odoo.com/">Odoo</a>, a well-known ERP system. Upgrading to version 11.0 is the occasion to update my python scripts to reduce considerably the number of requests. The improvements were done with the help of <a href="https://pandas.pydata.org/">pandas</a>, the famous data structures and data analysis library.</p>
<p><a href="https://www.flickr.com/photos/ken_from_md/4697952954/"><img alt="Photo credit: Panda_3956, Ken_from_MD" src="https://blog.agayon.be/images/panda0.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a> <br/>
Photo credit: Panda_3956, <a href="https://www.flickr.com/photos/ken_from_md/">Ken_from_MD</a> </p>
<h1>Introduction</h1>
<p>In my spare time, I help a small association, <a href="http://www.compagnonsducep.be/">Les Compagnons du CEP</a>, a joint buying organization who buys French wines for its members directly from producers. Since 2014, I set up an <a href="https://www.odoo.com/">Odoo</a> instance to manage the quotations, purchase order and the members. Odoo fulfills all their needs and we are happy to use it daily.</p>
<p>In addition to the user-friendly Web interface, the following management tasks are performed within a custom <a href="https://www.djangoproject.com/">Django</a> website.</p>
<ul>
<li>Calculate the price of wines based on the seller price, taxes, transport cost, VAT, <a href="https://www.fostplus.be/en">Fost+</a>. </li>
<li>Import wines to database. The list of sellable wines is updated two times a year in order to adapt to the seasonal dishes[ref]The covered regions are: Champagne, Alsace, Loire, Bourgogne, Beaujolais - Maconnais, Rhône, Provence, Languedoc [/ref]. Of course, the vintages are changed each year depending of the wines. The association sells approximately 600 different wines (up to 10000 bottles a year). As a result, I wrote a massive import script. This program reads large Excel files containing the several parameters (name, vintage, seller name, description, etc) and it uses the XML-RPC API offered by Odoo to create the items in the PostgreSQL database. This task is the main subject if this article.</li>
<li>Generate the price list based on the newly added wines. A price list Excel file is uploaded by the user. The file is transferred on another machine with the help of <a href="https://blog.agayon.be/errol.html">Errol</a>. A <a href="https://www.latex-project.org/">LaTeX</a> document is generated with python from the Excel file and it is compiled to PDF with <a href="https://www.tug.org/texlive/">TeX Live</a>. Afterward, the PDF is automatically copied on the Django website with Errol.</li>
</ul>
<p>The Django website is successfully used since 4 years. Unfortunately, I observed slowdown in the process since the implementation of product update from the Excel file.</p>
<p>In order to prepare the upgrade to <a href="https://www.odoo.com/fr_FR/blog/notre-blog-5/post/introducing-odoo-11-455">Odoo 11.0</a>, I decided to update the XML-RPC calls in order to reduce their number and therefore accelerate the import process.</p>
<h1>Some code to get your teeth into</h1>
<p><a href=""><img alt="Photo credit: Panda, Sue Cantan https://www.flickr.com/photos/suecan/4349221370/" src="https://blog.agayon.be/images/panda3.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a><br/>
Photo credit: Panda, <a href="https://www.flickr.com/photos/suecan/">Sue Cantan</a></p>
<p>To chose the more suitable strategy, I decided to compare the current code base to a new scenario.</p>
<h2>Description of the current code</h2>
<p>Each row of the Excel file correspond to a wine. The name, seller code[ref]The seller code is saved in the field 'function'[/ref] and default code are searched in the Odoo database in order to avoid duplicates. If no collision is found with the older products, a product template is created and the supplier information are updated. Since 2014, version 8.0 is used in production with <a href="https://www.odoo.com/documentation/user/9.0/inventory/settings/products/variants.html">product variant</a> support. Unfortunately, the variant requires to perform an additional search to update the price of the product. In version 11.0, we will get rid of the product variant. </p>
<p>The python code can be summarized as follows.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">xmlrpc.client</span> <span class="k">as</span> <span class="nn">xmlrpclib</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="n">username</span> <span class="o">=</span> <span class="s1">'user'</span>
<span class="n">pwd</span> <span class="o">=</span> <span class="s1">'password'</span>
<span class="n">dbname</span><span class="o">=</span> <span class="s1">'database_name'</span>
<span class="n">sock_common</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span><span class="s1">'http://localhost:8069/xmlrpc/common'</span><span class="p">)</span>
<span class="n">uid</span> <span class="o">=</span> <span class="n">sock_common</span><span class="o">.</span><span class="n">login</span><span class="p">(</span><span class="n">database</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">pwd</span><span class="p">)</span>
<span class="n">sock</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span><span class="s1">'http://localhost:8069/xmlrpc/object'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">):</span>
<span class="n">supplier_code</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">supplier_code</span><span class="p">))</span>
<span class="n">args_p</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">default_code</span><span class="p">)]</span>
<span class="n">args_s</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'function'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">)]</span>
<span class="n">args_n</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">)]</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">n_supplier</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">n_name</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span> <span class="n">args_p</span><span class="p">)</span>
<span class="c1"># Rather than retrieve a possibly gigantic list of records and count them, search_count()</span>
<span class="c1"># can be used to retrieve only the number of records matching the query.</span>
<span class="c1"># It takes the same domain filter as search() and no other parameter.</span>
<span class="c1"># https://www.odoo.com/documentation/8.0/api_integration.html</span>
<span class="n">n_supplier</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'res.partner'</span><span class="p">,</span> <span class="s1">'search_count'</span><span class="p">,</span> <span class="n">args_s</span><span class="p">)</span>
<span class="n">n_name</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search_count'</span><span class="p">,</span> <span class="n">args_n</span><span class="p">)</span>
<span class="n">n_supplier</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_supplier</span><span class="p">)</span>
<span class="n">n_name</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_name</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="n">start_session</span><span class="p">()</span>
<span class="k">return</span> <span class="s2">"e_initialization"</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids_product</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_supplier</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'success'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids_product</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Le code est déjà utilisé et un produit du même nom existe.</span>
<span class="k">return</span> <span class="s1">'e_code_used_same_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids_product</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># Le code est déjà utilisé</span>
<span class="k">return</span> <span class="s1">'e_code_used'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="c1"># if len(ids_product) == 0:</span>
<span class="k">if</span> <span class="n">n_supplier</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># Le fournisseur n existe pas</span>
<span class="k">return</span> <span class="s1">'e_missing_seller'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># un produit du même nom mais pas le même code existe</span>
<span class="k">return</span> <span class="s1">'e_code_used_different_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">def</span> <span class="nf">import2odoo</span><span class="p">():</span>
<span class="n">route_warehouse0_mto</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">route_warehouse0_manufacture</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># iterate over all rows, read the cells and assign the wine parameters to variables</span>
<span class="c1"># each row correspond to one wine</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">seller_name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">do_import</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="n">standard_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span>
<span class="n">list_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span>
<span class="n">seller_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span>
<span class="n">res_search</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">ids_supplier</span><span class="p">,</span> <span class="n">ids_name</span> <span class="o">=</span> <span class="n">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">seller_code</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res_search</span> <span class="o">==</span> <span class="s1">'success'</span> <span class="ow">and</span> <span class="n">do_import</span> <span class="o">==</span> <span class="s2">"1"</span><span class="p">:</span>
<span class="c1"># Success, the wine may be added</span>
<span class="n">results_dict</span><span class="p">[</span><span class="s1">'added_wines'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">default_code</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">standard_price</span><span class="p">,</span> <span class="n">list_price</span><span class="p">,</span> <span class="n">comment</span><span class="p">])</span>
<span class="n">product_template</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'active'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'state'</span><span class="p">:</span> <span class="s1">'sellable'</span><span class="p">,</span>
<span class="s1">'standard_price'</span><span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="s1">'list_price'</span><span class="p">:</span> <span class="n">list_price</span><span class="p">,</span>
<span class="s1">'description'</span><span class="p">:</span> <span class="n">comment</span> <span class="p">,</span>
<span class="s1">'purchase_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'sale_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_po_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'product'</span><span class="p">,</span>
<span class="s1">'cost_method'</span><span class="p">:</span> <span class="s1">'standard'</span><span class="p">,</span>
<span class="s1">'route_ids'</span><span class="p">:</span> <span class="p">[(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="n">route_warehouse0_mto</span><span class="p">,</span> <span class="n">route_warehouse0_manufacture</span><span class="p">])]</span>
<span class="p">}</span>
<span class="c1"># For each wine, a template must be created</span>
<span class="n">template_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.template'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span>
<span class="n">product_template</span><span class="p">)</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'product_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span> <span class="c1"># code chez le fournisseur</span>
<span class="s1">'product_name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="c1"># name chez le producteur</span>
<span class="s1">'min_qty'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'delay'</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span>
<span class="s1">'product.supplierinfo'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span> <span class="n">product_supplierinfo</span><span class="p">)</span>
<span class="c1"># The product id must be obtained to set the default code</span>
<span class="n">product_product</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="s1">'default_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">name</span><span class="p">),</span> <span class="p">]</span>
<span class="n">product_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span>
<span class="n">args</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'write'</span><span class="p">,</span>
<span class="n">product_id</span><span class="p">,</span>
<span class="n">product_product</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.</span>
</code></pre></div>
<h3>Metrics</h3>
<p>If <code>N</code> is the number of wines to add, <code>x</code>, the number of milliseconds to research an item and <code>y</code> the number of milliseconds to write an item into the database, the previous code is made of:</p>
<ul>
<li>3 researches per wine : 3<code>N</code> * <code>x</code></li>
<li>templates creation: <code>N</code> <code>y</code> </li>
<li>supplierinfo creation: <code>N</code> <code>y</code></li>
<li>research products: <code>N</code> <code>x</code></li>
<li>product update: <code>N</code> <code>y</code></li>
</ul>
<p><strong>Total: <code>4N x + 3N y</code></strong></p>
<p>This code is not optimized. It runs slowly even if it achieves what we expect from it. Now, we can try to do enhance it.</p>
<h2>The new version</h2>
<p>As previously mentioned, the product variant suppression will remove one search but there is more room for improvements.</p>
<p>The API call for each row is the main bottleneck of the script. I decided to replace them by a search of all product at the beginning of the script. The list if wine is return in a list of dictionary. Fortunately, a list of dicts is the easiest object to convert to a Pandas Dataframe.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="k">def</span> <span class="nf">dataframes_generator</span><span class="p">():</span>
<span class="c1"># Use a search function with empty args to get all ids :</span>
<span class="c1"># https://www.odoo.com/fr_FR/forum/aide-1/question/is-it-possible-to-retrieve-2-fields-of-all-entry-within-a-model-thru-xml-rpc-6886</span>
<span class="c1"># 1st => search all ids 2nd => read the selected fields on the whole list of ids.</span>
<span class="c1"># Only 2 rpc-xml requests :o)</span>
<span class="c1">#1</span>
<span class="n">args_product</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="s1">'!='</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">)]</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span> <span class="n">args_product</span><span class="p">)</span>
<span class="c1"># 2</span>
<span class="n">fields_products</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">,</span> <span class="s1">'default_code'</span><span class="p">]</span>
<span class="n">recordset_products</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'read'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">fields_products</span><span class="p">)</span>
<span class="n">args_seller</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'!='</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">)]</span>
<span class="n">fields_seller</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">,</span> <span class="s1">'function'</span><span class="p">]</span>
<span class="n">ids_sellers</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'res.partner'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span> <span class="n">args_seller</span><span class="p">)</span>
<span class="n">recordset_sellers</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'res.partner'</span><span class="p">,</span> <span class="s1">'read'</span><span class="p">,</span> <span class="n">ids_sellers</span><span class="p">,</span>
<span class="n">fields_seller</span><span class="p">)</span>
<span class="n">df_product</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">recordset_products</span><span class="p">)</span>
<span class="n">df_sellers</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">recordset_sellers</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">True</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">ids_sellers</span>
</code></pre></div>
<p>For now, the ids list is generated by searching all product with a dummy code ('foo' in the example). My trials to get rid of the <code>args_product</code> variable failed.</p>
<p>As the dataframes are generated, it is easy to search the product and sellers among their respective dataframes:</p>
<div class="highlight"><pre><span></span><code><span class="n">code_to_find</span> <span class="o">=</span> <span class="s1">'A18'</span>
<span class="n">df_product</span><span class="p">[</span><span class="n">df_product</span><span class="p">[</span><span class="s1">'default_code'</span><span class="p">]</span><span class="o">.</span><span class="n">str</span><span class="o">.</span><span class="n">contains</span><span class="p">(</span><span class="n">code_to_find</span><span class="p">)]</span>
<span class="n">default_code</span> <span class="nb">id</span> <span class="n">name</span>
<span class="mi">0</span> <span class="n">A18</span><span class="o">/</span><span class="mi">999</span> <span class="mi">21</span> <span class="n">First</span> <span class="n">wine</span>
<span class="mi">1</span> <span class="n">A18</span><span class="o">/</span><span class="mi">999</span> <span class="mi">22</span> <span class="n">Second</span> <span class="n">wine</span>
</code></pre></div>
<p>A small function <code>search_string_df</code> can be written to facilitate the future searches in dataframes. It returns the dataframe and its lenght:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="c1"># if we search an int.</span>
<span class="k">if</span> <span class="n">search_int</span><span class="p">:</span>
<span class="n">df_res</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">df</span><span class="p">[</span><span class="n">col_name</span><span class="p">]</span> <span class="o">==</span> <span class="n">search_item</span><span class="p">]</span>
<span class="c1"># if we search a string. We must ignore NaN values.</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">df_res</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="n">df</span><span class="p">[</span><span class="n">col_name</span><span class="p">]</span><span class="o">.</span><span class="n">str</span><span class="o">.</span><span class="n">contains</span><span class="p">(</span><span class="n">search_item</span><span class="p">,</span> <span class="n">na</span><span class="o">=</span><span class="kc">False</span><span class="p">)]</span>
<span class="n">n_res</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">df_res</span><span class="p">)</span>
<span class="k">return</span> <span class="n">n_res</span><span class="p">,</span> <span class="n">df_res</span>
<span class="n">n</span><span class="p">,</span> <span class="n">df</span> <span class="o">=</span> <span class="n">search_string_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="n">search_str</span><span class="o">=</span><span class="s1">'A18'</span><span class="p">)</span>
</code></pre></div>
<h3>Final cut</h3>
<p>When the research function is modified, the whole code can also be adapted:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">):</span>
<span class="c1"># 1) search if default code is used?)</span>
<span class="c1"># 2) search if suppliers is in the database</span>
<span class="c1"># 3) search if the name is already used</span>
<span class="c1"># Retrieve the dataframes. This example comes from a jupyter nootebook.</span>
<span class="c1"># df_product and df_sellers are already defined.</span>
<span class="c1"># In a real case, we should use class variables.</span>
<span class="n">n_code</span><span class="p">,</span> <span class="n">df_code</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">default_code</span><span class="p">,</span>
<span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_supplier</span><span class="p">,</span> <span class="n">df_suppliers</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_suppliers</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'function'</span><span class="p">,</span>
<span class="n">search_item</span><span class="o">=</span><span class="n">supplier_code</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_name</span><span class="p">,</span> <span class="n">df_name</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'name'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">wine_name</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># ids_product = list(df_code['id'])</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">df_code</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_supplier</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'success'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Another product uses the same name with another code</span>
<span class="k">return</span> <span class="s1">'e_code_used_same_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># the code is already used</span>
<span class="k">return</span> <span class="s1">'e_code_used'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_supplier</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Cannot find the supplier</span>
<span class="k">return</span> <span class="s1">'e_missing_seller'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># A product with the same name and another code exists.</span>
<span class="k">return</span> <span class="s1">'e_code_used_different_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">def</span> <span class="nf">import2odoo</span><span class="p">():</span>
<span class="n">route_warehouse0_mto</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">route_warehouse0_manufacture</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># iterate over all rows, read the cells and assign the wine parameters to variables</span>
<span class="c1"># each row correspond to one wine</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">seller_name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">do_import</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="n">standard_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span>
<span class="n">list_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span>
<span class="n">seller_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span>
<span class="n">res_search</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span> <span class="o">=</span> <span class="n">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">seller_code</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res_search</span> <span class="o">==</span> <span class="s1">'success'</span> <span class="ow">and</span> <span class="n">do_import</span> <span class="o">==</span> <span class="s2">"1"</span><span class="p">:</span>
<span class="n">product_template</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'active'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'standard_price'</span><span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="s1">'list_price'</span><span class="p">:</span> <span class="n">list_price</span><span class="p">,</span>
<span class="s1">'description'</span><span class="p">:</span> <span class="n">comment</span> <span class="p">,</span>
<span class="s1">'default_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span>
<span class="s1">'purchase_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'sale_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_po_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'product'</span><span class="p">,</span>
<span class="s1">'cost_method'</span><span class="p">:</span> <span class="s1">'standard'</span><span class="p">,</span>
<span class="s1">'route_ids'</span><span class="p">:</span> <span class="p">[(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="n">route_warehouse0_mto</span><span class="p">,</span> <span class="n">route_warehouse0_manufacture</span><span class="p">])]</span>
<span class="p">}</span>
<span class="c1"># For each wine, a template must be created</span>
<span class="n">template_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.template'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span>
<span class="n">product_template</span><span class="p">)</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'product_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span> <span class="c1"># code chez le fournisseur</span>
<span class="s1">'product_name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="c1"># name chez le producteur</span>
<span class="s1">'min_qty'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'delay'</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span>
<span class="s1">'product.supplierinfo'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span> <span class="n">product_supplierinfo</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Wine </span><span class="si">{}</span><span class="s2"> : </span><span class="si">{}</span><span class="s2"> has been added"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.</span>
</code></pre></div>
<h3>Metrics</h3>
<p>If we follow the same conventions than before:</p>
<p>The new code is made of:</p>
<ul>
<li>3 researches: 3 * <code>x</code></li>
<li>templates creation: <code>N</code> <code>y</code> </li>
<li>supplierinfo creation: <code>N</code> <code>y</code></li>
</ul>
<p><strong>Total: <code>3 x + 2N y</code></strong></p>
<p>This result is a huge improvement. The data search in Dataframe is really fast and therefore, the impact of the bottleneck is decreased.</p>
<h1>Conclusions</h1>
<p>After several tests, the new implementation seems solid. It is not as fast as I would have expected but I am working on it. Some helpful <a href="https://fr.slideshare.net/openobject/performance2014-35689113">resources</a> are available on the web.</p>
<p>The import campaign of spring is already finished but the new algorithm will be tested with the aim to be ready for the wines coming in autumn. Nevertheless, reducing the number of request from 3N to 3 where N is the number of wines can only have beneficial effects. </p>
<p>The next step will be to watch and analyse the <a href="https://www.odoo.com/documentation/11.0/reference/cmdline.html#logging">databases requests</a>. The option <code>log-level=debug_rpc</code> will probably be crucial.</p>
<p>The results of these investigations will be shared here.</p>
<p>Stay tuned !</p>
<p><a href="https://www.flickr.com/photos/gzlu/7708872342/"><img alt="Photo credit: Panda in China, George Lu https://www.flickr.com/photos/gzlu/7708872342/" src="https://blog.agayon.be/images/panda1.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a><br/>
Photo credit: Panda in China, <a href="https://www.flickr.com/photos/gzlu/">George Lu</a> </p>
<h1>Links</h1>
<ul>
<li><a href="https://www.odoo.com/">Odoo</a></li>
<li><a href="https://pandas.pydata.org/">Pandas</a></li>
<li><a href="https://www.djangoproject.com/">Django</a></li>
<li><a href="https://fr.slideshare.net/openobject/performance2014-35689113">Improving the performance of Odoo deployments</a></li>
<li><a href="https://www.odoo.com/documentation/11.0/reference/cmdline.html">Command-line interface: odoo-bin</a></li>
</ul>Errol: XMPP Automatic file sender2018-01-01T14:00:00+01:002018-01-01T14:00:00+01:00Arnaudtag:blog.agayon.be,2018-01-01:/errol.html
<p>Errol is a file sender that rely on <a href="https://en.wikipedia.org/wiki/Inotify">inotify</a>. It can be used to watch a directory and automatically transfers the new files (or modified ones) with XMPP.</p>
<p>Errol is a file sender that rely on <a href="https://en.wikipedia.org/wiki/Inotify">inotify</a>. It can be used to watch a directory and automatically transfers the new files (or modified ones) with XMPP.</p>
<h1>The origins</h1>
<p>Errol find its origin in tasks I am doing for a small association, "<a href="http://www.compagnonsducep.be">Les compagnons du CEP</a>", a joint buying organization who sells wines. I manage their ERP, the excellent <a href="https://www.odoo.com/">Odoo</a>. One of these tasks is the generation of their price list from an Excel spreadsheet (yeah, I know). I designed this process with a <a href="https://www.latex-project.org/">LaTeX</a> generator written in python because I am fluent with it since 10 years. As I did not want to install a LaTeX distribution on the production server, the logical decision was to delocalize this task on another machine. The user uploads his excel file on a webpage, the file is saved in a "watched" directory and its transfer is triggered on the second machine with XMPP. The generation of the latex document and its compilation is performed with LaTeX and the resulting PDF is sent back to the server. The PDF is therefore available for download.</p>
<h2>Why Errol?</h2>
<p>In the fictional universe of Harry Potter, <a href="https://en.wikipedia.org/wiki/Magical_creatures_in_Harry_Potter#The_Weasleys'_creatures"><strong>Errol</strong></a> is the Weasley family's owl. It is quite old and awkward. One could says the same about XMPP but Errol is quite useful, XMPP is too :-). Errol is a great grey owl. (see pictures)</p>
<p><a href="https://www.flickr.com/photos/blurredca/10527590684/"><img alt="photo credit: blurred.ca https://www.flickr.com/photos/blurredca/10527590684/" src="https://blog.agayon.be/images/owl6.jpg" style="width: 752px; height: auto; max-width: 100%;"/></a>
Photo credit: <a href="https://www.flickr.com/photos/blurredca/">blurred.ca</a>, Great Grey Owl</p>
<hr/>
<h1>Howto</h1>
<h2>Prerequisites</h2>
<p>Errol needs the following requirements:</p>
<ul>
<li>A system supporting <a href="https://en.wikipedia.org/wiki/Inotify">inotify</a> (Linux).</li>
<li>an XMPP (jabber) account supporting the following XEPs: <a href="https://xmpp.org/extensions/xep-0198.html">Stream Management</a>, <a href="https://xmpp.org/extensions/xep-0060.html">Publish-Subscribe</a>, <a href="https://xmpp.org/extensions/xep-0045.html">Multi-User Chat</a></li>
<li>A PubSub service where the nodes can be set as open. The node name is defined in the configuration file. I personally use <a href="https://blog.agayon.be/sat_pubsub.html">sat_pubsub</a>. A PubSub
component developed for the project <a href="https://salut-a-toi.org/">Salut à Toi</a>. </li>
<li>A Multi-User Chat because not all XMPP accounts support PubSub. For now, some information are still shared through MUC messages. This behavior could change in the future.</li>
<li>The latest (dev) version of <a href="https://lab.louiz.org/poezio/slixmpp">Slixmpp</a>.</li>
</ul>
<p>You can use your own XMPP server or choose a XMPP service among the following <a href="https://conversations.im/compliance/">list</a>. </p>
<h3>Create the PubSub node</h3>
<p>This step is optional if you already have a write access on the pubsub node. The following example use <a href="https://blog.agayon.be/sat_jp.html">jp</a>, the Salut à Toi command-line interface but
<a href="https://lab.louiz.org/poezio/slixmpp">slixmpp</a> or
<a href="https://github.com/fritzy/SleekXMPP">sleekxmpp</a> can be used. </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>node<span class="w"> </span>create<span class="w"> </span>-f<span class="w"> </span>publish_model<span class="w"> </span>open<span class="w"> </span>be.agayon.errol:0<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-c
</code></pre></div>
<p>The node name be.agayon.errol:0 is recommended in order to identify the functionality.</p>
<p>As an example, there are the node options on the service pubsub.agayon.be:</p>
<div class="highlight"><pre><span></span><code><span class="o">$</span><span class="w"> </span><span class="n">jp</span><span class="w"> </span><span class="n">pubsub</span><span class="w"> </span><span class="n">node</span><span class="w"> </span><span class="n">info</span><span class="w"> </span><span class="n">be</span><span class="o">.</span><span class="n">agayon</span><span class="o">.</span><span class="n">errol</span><span class="p">:</span><span class="mi">0</span><span class="w"> </span><span class="o">-</span><span class="n">s</span><span class="w"> </span><span class="n">pubsub</span><span class="o">.</span><span class="n">agayon</span><span class="o">.</span><span class="n">be</span>
<span class="n">persist_items</span><span class="p">:</span><span class="w"> </span><span class="n">True</span>
<span class="n">deliver_payloads</span><span class="p">:</span><span class="w"> </span><span class="n">True</span>
<span class="n">serial_ids</span><span class="p">:</span><span class="w"> </span><span class="n">False</span>
<span class="n">publish_model</span><span class="p">:</span><span class="w"> </span><span class="n">open</span>
<span class="n">access_model</span><span class="p">:</span><span class="w"> </span><span class="n">open</span>
<span class="n">send_last_published_item</span><span class="p">:</span><span class="w"> </span><span class="n">on_sub</span>
</code></pre></div>
<p>If your server supports <a href="https://xmpp.org/extensions/xep-0163.html">Personal Eventing Protocol</a>(PEP) or if you do not want to use the generic PubSub service of your server, you can use your jid.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>node<span class="w"> </span>create<span class="w"> </span>-f<span class="w"> </span>publish_model<span class="w"> </span>open<span class="w"> </span>be.agayon.errol:0<span class="w"> </span>-s<span class="w"> </span>info@agayon.be<span class="w"> </span>-c
</code></pre></div>
<h3>Tests</h3>
<p>You can test your setup with the examples scripts of <a href="https://git.poez.io/slixmpp">slixmpp</a>.</p>
<ul>
<li><a href="https://git.poez.io/slixmpp/tree/examples/pubsub_client.py">pubsub_client.py</a></li>
<li><a href="https://git.poez.io/slixmpp/tree/examples/pubsub_events.py">pubsub_events.py</a></li>
<li><a href="https://git.poez.io/slixmpp/tree/examples/s5b_transfer/s5b_receiver.py">s5b_receiver.py</a></li>
<li><a href="https://git.poez.io/slixmpp/tree/examples/s5b_transfer/s5b_sender.py">s5b_sender.py</a></li>
</ul>
<p>Example:</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="o">/</span><span class="n">s5b_file_sender</span><span class="p">.</span><span class="n">py</span><span class="w"> </span><span class="o">-</span><span class="n">j</span><span class="w"> </span><span class="n">jid</span><span class="nv">@example</span><span class="p">.</span><span class="n">org</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="n">pass</span><span class="w"> </span><span class="o">-</span><span class="n">r</span><span class="w"> </span><span class="n">john</span><span class="nv">@example</span><span class="p">.</span><span class="n">org</span><span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="o">/</span><span class="k">path</span><span class="o">/</span><span class="k">to</span><span class="o">/</span><span class="k">file</span><span class="p">.</span><span class="n">txt</span><span class="w"> </span>
</code></pre></div>
<p>See the scripts for more information.</p>
<h2>Getting started with Errol</h2>
<p>Errol needs the following dependencies:</p>
<ul>
<li><a href="https://lab.louiz.org/poezio/slixmpp">slixmpp</a> (python 3 only), git revision after 2017-12-27</li>
<li><a href="https://docs.python.org/3/library/asyncio.html">asyncio </a></li>
<li><a href="https://docs.python.org/3/library/configparser.html">configparser</a></li>
<li><a href="https://github.com/rbarrois/aionotify">aionotify</a></li>
</ul>
<h2>Installing</h2>
<p>You can easily install errol with pip:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>errol
</code></pre></div>
<p>Note: errol can be installed in a <a href="https://virtualenv.pypa.io/en/stable/userguide/">virtualenv</a>.</p>
<p>You can also clone the git repository:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>clone<span class="w"> </span>https://gitlab.com/jnanar/errol.git
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>errol
$<span class="w"> </span>python3<span class="w"> </span>setup.py<span class="w"> </span>install
</code></pre></div>
<p>On Archlinux:</p>
<div class="highlight"><pre><span></span><code>A PKGBUILD will be available soon.
</code></pre></div>
<h2>Configuration</h2>
<p>You need to provide information about the XMPP account.</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">cat</span><span class="w"> </span><span class="n">config</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">ini</span>
<span class="o">[</span><span class="n">XMPP</span><span class="o">]</span>
<span class="n">pubsub</span><span class="o">=</span><span class="n">pubsub</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">org</span>
<span class="n">node</span><span class="o">=</span><span class="n">be</span><span class="p">.</span><span class="n">agayon</span><span class="p">.</span><span class="nl">errol</span><span class="p">:</span><span class="mi">0</span>
<span class="n">room</span><span class="o">=</span><span class="n">chat</span><span class="nv">@chat</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">org</span>
<span class="n">jid</span><span class="o">=</span><span class="n">jid</span><span class="nv">@example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">errol</span>
<span class="n">password</span><span class="o">=</span><span class="n">pass</span>
<span class="n">ressource_receiver</span><span class="o">=-</span><span class="n">receiver</span>
<span class="n">ressource_sender</span><span class="o">=-</span>
<span class="n">nick_sender</span><span class="o">=</span><span class="n">example_sender</span>
<span class="n">nick_receiver</span><span class="o">=</span><span class="n">example_receiver</span>
<span class="n">receiver</span><span class="o">=</span><span class="n">jid</span><span class="nv">@example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">errol</span><span class="o">-</span><span class="n">receiver</span>
</code></pre></div>
<ul>
<li>jid : the jabber account</li>
<li>password: the xmpp password</li>
<li>pubsub: the PubSub server (publish activity)</li>
<li>room: the MUC (chatroom) where errol display information.</li>
</ul>
<p>The files will be sent by jid@example.org/errol-0 and received by jid@example.org/errol-receiver
. The nicks are the usernames used on the MUC.</p>
<h2>Flying</h2>
<p><a href="https://www.flickr.com/photos/widnr/8716853721/"><img alt="Photo credit: Wisconsin Department of Natural Resources https://www.flickr.com/photos/widnr/" src="https://blog.agayon.be/images/owl7.jpg" style="width: 704px; height: auto; max-width: 100%;"/></a>
Photo credit: <a href="https://www.flickr.com/photos/widnr/">Wisconsin Department of Natural Resources</a>, Great Grey Owl at Mauston</p>
<p>Once installed, Errol can be launched in a terminal.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>errol<span class="w"> </span>--help
usage:<span class="w"> </span>errol<span class="w"> </span><span class="o">[</span>-h<span class="o">]</span><span class="w"> </span><span class="o">[</span>-e<span class="w"> </span>EVENTS<span class="o">]</span><span class="w"> </span><span class="o">[</span>-f<span class="w"> </span>FILE<span class="o">]</span><span class="w"> </span><span class="o">[</span>-d<span class="o">]</span><span class="w"> </span>-p<span class="w"> </span>PATH<span class="w"> </span>-c<span class="w"> </span>COMMAND
Automatic<span class="w"> </span>XMPP<span class="w"> </span>file<span class="w"> </span>sender<span class="w"> </span>and<span class="w"> </span>directory<span class="w"> </span>watcher
optional<span class="w"> </span>arguments:
<span class="w"> </span>-h,<span class="w"> </span>--help<span class="w"> </span>show<span class="w"> </span>this<span class="w"> </span><span class="nb">help</span><span class="w"> </span>message<span class="w"> </span>and<span class="w"> </span><span class="nb">exit</span>
<span class="w"> </span>-e<span class="w"> </span>EVENTS,<span class="w"> </span>--events<span class="w"> </span>EVENTS
<span class="w"> </span>Number<span class="w"> </span>of<span class="w"> </span>events<span class="w"> </span>to<span class="w"> </span>watch<span class="w"> </span><span class="o">(</span>delete,<span class="w"> </span>create<span class="w"> </span>modify<span class="o">)</span><span class="w"> </span><span class="k">in</span>
<span class="w"> </span>the<span class="w"> </span>directory.<span class="w"> </span>Once<span class="w"> </span>reached,<span class="w"> </span>the<span class="w"> </span>program<span class="w"> </span>stops.
<span class="w"> </span>-f<span class="w"> </span>FILE,<span class="w"> </span>--file<span class="w"> </span>FILE<span class="w"> </span>Config<span class="w"> </span>file<span class="w"> </span>containing<span class="w"> </span>XMPP<span class="w"> </span>parameters
<span class="w"> </span>-d,<span class="w"> </span>--debug<span class="w"> </span><span class="nb">set</span><span class="w"> </span>logging<span class="w"> </span>to<span class="w"> </span>DEBUG
<span class="w"> </span>-p<span class="w"> </span>PATH,<span class="w"> </span>--path<span class="w"> </span>PATH<span class="w"> </span>The<span class="w"> </span>path<span class="w"> </span>watched.
<span class="w"> </span>-c<span class="w"> </span>COMMAND,<span class="w"> </span>--command<span class="w"> </span>COMMAND
<span class="w"> </span>The<span class="w"> </span>executed<span class="w"> </span>command:<span class="w"> </span>xmpp<span class="w"> </span>or<span class="w"> </span>watcher
</code></pre></div>
<h3>In Hogwarts</h3>
<p>If you want to watch the directory /tmp/sender, the following command can be used:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>errol<span class="w"> </span>-f<span class="w"> </span>config.example.ini<span class="w"> </span>-p<span class="w"> </span>/tmp/sender<span class="w"> </span>-c<span class="w"> </span>watcher
</code></pre></div>
<p>All modified or new files created in the watched location will be sent by XMPP.</p>
<h3>In Azkaban</h3>
<p>If you want to receive the files, you have to launch Errol with the following command line.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>errol<span class="w"> </span>-f<span class="w"> </span>config.example.ini<span class="w"> </span>-p<span class="w"> </span>/tmp/receiver<span class="w"> </span>-c<span class="w"> </span>xmpp
</code></pre></div>
<p>All the received files will be stored in the directory defined with the option '-p'.</p>
<h2>License</h2>
<p>This project is licensed under the GPLv3 - see the <a href="https://gitlab.com/jnanar/errol/blob/master/LICENCE.txt">LICENSE.txt</a> file for details</p>
<h1>Why not X or Y?</h1>
<p><a href="https://www.flickr.com/photos/volvob12b/37310719232/"><img alt="Photo credit: Bernard Spragg. NZ https://www.flickr.com/photos/volvob12b/37310719232/" src="https://blog.agayon.be/images/owl3.jpg" style="width: 821px; height: auto; max-width: 100%;"/></a>
Photo credit: <a href="https://www.flickr.com/photos/volvob12b/">Bernard Spragg. NZ</a>, Great Grey Owl (Strix nebulosa)</p>
<p>There are plenty solutions for this kind of needs. Some of them are more mature. I choose XMPP for several reasons:</p>
<ul>
<li><a href="https://agayon.be">agayon.be</a> already provides a up-to-date XMPP server with all the needed XEPs enabled.</li>
<li>I do not want to open additional port on the client that performs the LaTeX compilation.</li>
<li>I wanted to learn to work with XMPP for machine to machine communications and use PubSub notifications (because why not?).</li>
</ul>
<p>Among the alternatives, I could have build the service on top of:</p>
<ul>
<li>sockets</li>
<li>HTTP file transfer</li>
<li>REST API</li>
<li>SSH and remote commands.</li>
<li>rsync</li>
<li>...</li>
</ul>
<h2>Acknowledgments</h2>
<p><a href="https://www.flickr.com/photos/115391424@N05/36873334554/"><img alt="Photo credit: lasta29, Great grey owl, Osaka Tennoji Zoo https://www.flickr.com/photos/115391424@N05/36873334554/" src="https://blog.agayon.be/images/owl5.jpg" style="width: 750px; height: auto; max-width: 100%;"/></a>
Photo credit: <a href="https://www.flickr.com/photos/115391424@N05/">lasta29</a>, Great grey owl, Osaka Tennoji Zoo</p>
<ul>
<li>The french XMPP community (availaible on sat@chat.jabberfr.org, jabberfr@chat.jabberfr.org, ...)</li>
<li><a href="https://lab.louiz.org/poezio/slixmpp">The Slixmpp maintainers</a> Florent Le Coz, Mathieu Pasquet for their nice library.</li>
<li>Emmanuel Gil Peyrot (Link mauve) for its reactivity.</li>
<li>Goffi from the <a href="https://salut-a-toi.org/">Salut à Toi</a> project for his explanations and his disponibility.</li>
</ul>
<h1 id="links">Links</h1>
<ul>
<li><a href="https://gitlab.com/jnanar/errol">Git repository</a></li>
<li><a href="https://pypi.python.org/pypi/errol">pypi package</a></li>
<li><a href="https://salut-a-toi.org/">Salut à Toi</a></li>
<li><a href="https://docs.python.org/3/library/asyncio.html">asyncio </a></li>
<li><a href="https://lab.louiz.org/poezio/slixmpp">slixmpp</a></li>
<li><a href="https://github.com/rbarrois/aionotify">aionotify</a></li>
</ul>Using sat-pubsub, a great pubsub component2017-11-26T20:00:00+01:002019-05-30T16:00:00+02:00Arnaudtag:blog.agayon.be,2017-11-26:/sat_pubsub.html
<p>In the continuity with the <a href="https://blog.agayon.be/sat_jp.html">previous post about jp</a>, the following article present <a href="https://wiki.goffi.org/wiki/Libervia/en#S.C3.A0T_PubSub">sat_pubsub</a>, a XMPP Publish-Subscribe (Pubsub) Service Component, build for the need of the « <a href="https://salut-a-toi.org/">Salut à Toi</a> » project.</p>
<p><a href="https://salut-a-toi.org/">Salut à toi</a> (SàT) is a unique XMPP client. As its official description says, it's a "multipurpose, multi front-end, free (libre) and decentralized communication tool". It has been actively developed by Jérôme Poisson (Goffi) and Adrien Cossa (Souliane) since 2008.</p>
<p>sat_pubsub allows us to use our own up-to-date persistent pubsub service.</p>
<p>This article is composed of several sections</p>
<ul>
<li><a href="#why">Why sat_pubsub ?</a> </li>
<li><a href="#install">Let's install it !</a></li>
<li><a href="#test">Let's use it !</a></li>
<li>A note about <a href="#agayon">Agayon XMPP service</a></li>
<li><a href="#future">Future of SàT</a></li>
<li><a href="#links">Links</a></li>
</ul>
<p>In the continuity with the <a href="https://blog.agayon.be/sat_jp.html">previous post about jp</a>, the following article present <a href="https://wiki.goffi.org/wiki/Libervia/en#S.C3.A0T_PubSub">sat_pubsub</a>, a XMPP Publish-Subscribe (Pubsub) Service Component, build for the need of the « <a href="https://salut-a-toi.org/">Salut à Toi</a> » project.</p>
<p><a href="https://salut-a-toi.org/">Salut à toi</a> (SàT) is a unique XMPP client. As its official description says, it's a "multipurpose, multi front-end, free (libre) and decentralized communication tool". It has been actively developed by Jérôme Poisson (Goffi) and Adrien Cossa (Souliane) since 2008.</p>
<p>sat_pubsub allows us to use our own up-to-date persistent pubsub service.</p>
<p>This article is composed of several sections</p>
<ul>
<li><a href="#why">Why sat_pubsub ?</a> </li>
<li><a href="#install">Let's install it !</a></li>
<li><a href="#test">Let's use it !</a></li>
<li>A note about <a href="#agayon">Agayon XMPP service</a></li>
<li><a href="#future">Future of SàT</a></li>
<li><a href="#links">Links</a></li>
</ul>
<h1 id="why">Why sat_pubsub ?</h1>
<p>There are <strong>three</strong> main reasons to use sat_pubsub.</p>
<p>First, the XMPP servers come with variable pubsub support. Using an external component allows users to benefits constant features independent of the XMPP server or provider.</p>
<p>Secondly, sat_pubub is the only free (libre) implementation that support Pubsub MAM (Message Archive Management). It allows one to research a pubsub node. This feature is used to research blog post in the database.</p>
<p>Thirdly, sat_pubsub is used to test new functionalities like the ability to restrict blog post to specific group of contacts (think about circles on Google+). [ref]See the following <a href="https://www.goffi.org/blog/goffi/S%C3%A0T_DOTCLEAR_IMPORT_BLOG_default_goffi_69%3A2012%2F06%2F24%2FFine-access-tuning-for-PubSub">blog post</a> for more information.[/ref]. </p>
<h1 id="install">Let's install it !</h1>
<p>This section describes the installation process of sat_pubsub in a python virtualenv. Unfortunately, I have encountered difficulties to install twisted, a dependency of SàT, in a virtualenv with pip because my production machine has no compiler. I managed to avoid the problem by installing python2-twisted (the distribution package) then the virtualenv was created with the option <code>--system-site-packages</code> to give it access to the system packages. Thereafter, it is possible to install sat_tmp, a small python module that monkey patch wokkel, a dependency of sat_pubsub.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>python2-twisted
<span class="o">[</span>...<span class="o">]</span>
$<span class="w"> </span>hg<span class="w"> </span>clone<span class="w"> </span>http://repos.goffi.org/sat_tmp
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>sat_tmp/
$<span class="w"> </span>virtualenv<span class="w"> </span>env<span class="w"> </span>-ppython2.7<span class="w"> </span>--system-site-packages
$<span class="w"> </span><span class="nb">source</span><span class="w"> </span>env/bin/activate
<span class="o">(</span>env<span class="o">)</span>$<span class="w"> </span>python2<span class="w"> </span>setup.py<span class="w"> </span>install
<span class="o">(</span>env<span class="o">)</span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>../
<span class="o">(</span>env<span class="o">)</span>$<span class="w"> </span>hg<span class="w"> </span>clone<span class="w"> </span>https://repos.goffi.org/sat_pubsub
<span class="o">(</span>env<span class="o">)</span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>sat_pubsub
<span class="o">(</span>env<span class="o">)</span>$<span class="w"> </span>python2<span class="w"> </span>setup.py<span class="w"> </span>install
</code></pre></div>
<h2>Setup the database</h2>
<p>According to the official <a href="https://wiki.goffi.org/wiki/Libervia/en#S.C3.A0T_PubSub">documentation</a>, we need to create the database and install the SQL schema.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>-u<span class="w"> </span>postgres<span class="w"> </span>createuser<span class="w"> </span>-d<span class="w"> </span>-P<span class="w"> </span><span class="sb">`</span>whoami<span class="sb">`</span>
$<span class="w"> </span>createdb<span class="w"> </span>pubsub
$<span class="w"> </span>psql<span class="w"> </span>pubsub<span class="w"> </span><<span class="w"> </span>sat_pubsub/db/pubsub.sql
</code></pre></div>
<h2>Configure Prosody</h2>
<p>In order to use sat_pubsub, we need to declare it in our prosody config file. Once again, the information is available in the <a href="https://wiki.goffi.org/wiki/Libervia/en#S.C3.A0T_PubSub">documentation</a>. Two files need to be modified: </p>
<ul>
<li>/etc/prosody/prosody.cfg.lua</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p">[...]</span>
<span class="n">modules_enabled</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">[...]</span>
<span class="s2">"delegation"</span><span class="p">;</span>
<span class="s2">"privilege"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">[...]</span>
</code></pre></div>
<ul>
<li>/etc/prosody/conf.avail/your_domain.cfg.lua</li>
</ul>
<p>This file defines the domain configuration of your XMPP server.</p>
<p>This is a copy of the configuration for the agayon server. The pubsub component is available at the address pubsub.agayon.be.
The following configuration:</p>
<ul>
<li>enables pubsub MAM,</li>
<li>uses the component for microblogging activity,</li>
<li>announces the pubsub nodes and items in <a href="https://xmpp.org/extensions/xep-0030.html">disco requests</a>,</li>
<li>lets the component access the roster and presence informations (used for <a href="https://xmpp.org/extensions/xep-0163.html">PEP</a>).</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">VirtualHost</span> <span class="s2">"agayon.be"</span>
<span class="n">enabled</span> <span class="o">=</span> <span class="kc">true</span>
<span class="p">[...]</span>
<span class="n">privileged_entities</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">[</span><span class="s2">"pubsub.agayon.be"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">roster</span> <span class="o">=</span> <span class="s2">"get"</span><span class="p">;</span>
<span class="n">message</span> <span class="o">=</span> <span class="s2">"outgoing"</span><span class="p">;</span>
<span class="n">presence</span> <span class="o">=</span> <span class="s2">"roster"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="n">delegations</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">[</span><span class="s2">"urn:xmpp:mam:1"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">filtering</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"node"</span><span class="p">};</span>
<span class="n">jid</span> <span class="o">=</span> <span class="s2">"pubsub.agayon.be"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">[</span><span class="s2">"http://jabber.org/protocol/pubsub"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">jid</span> <span class="o">=</span> <span class="s2">"pubsub.agayon.be"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">[</span><span class="s2">"http://jabber.org/protocol/pubsub#owner"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">jid</span> <span class="o">=</span> <span class="s2">"pubsub.agayon.be"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">[</span><span class="s2">"https://salut-a-toi/protocol/schema:0"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">jid</span> <span class="o">=</span> <span class="s2">"pubsub.agayon.be"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">[</span><span class="s2">"http://jabber.org/protocol/disco#items:*"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">jid</span> <span class="o">=</span> <span class="s2">"pubsub.agayon.be"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="n">Component</span> <span class="s2">"pubsub.agayon.be"</span>
<span class="n">component_secret</span> <span class="o">=</span> <span class="s2">"shared_secret"</span>
<span class="n">modules_enabled</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"delegation"</span><span class="p">,</span> <span class="s2">"privilege"</span><span class="p">}</span>
</code></pre></div>
<h2>Launch</h2>
<div class="highlight"><pre><span></span><code><span class="o">(</span>env<span class="o">)</span>$<span class="w"> </span>twistd<span class="w"> </span>sat_pubsub<span class="w"> </span>--jid<span class="o">=</span>pubsub.agayon.be<span class="w"> </span>--secret<span class="o">=</span>shared_secret<span class="w"> </span>--rhost<span class="o">=</span><span class="m">127</span>.0.0.1<span class="w"> </span><span class="se">\</span>
--rport<span class="o">=</span><span class="m">5347</span><span class="w"> </span>--backend<span class="o">=</span>pgsql<span class="w"> </span>--dbuser<span class="o">=</span>user<span class="w"> </span>--dbpass<span class="o">=</span>pass_psql<span class="w"> </span>--dbname<span class="o">=</span>pubsub<span class="w"> </span>--dbhost<span class="o">=</span>localhost<span class="w"> </span>--dbport<span class="o">=</span><span class="m">5432</span>
</code></pre></div>
<h1 id="test">Let's use it !</h1>
<p>Once prosody is restarted, the component is accessible through any XMPP client. See the following example with <a href="https://blog.agayon.be/sat_jp.html">jp</a>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>info<span class="w"> </span>disco<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-c
Features
http://jabber.org/protocol/disco#items
http://jabber.org/protocol/pubsub#auto-create
http://jabber.org/protocol/pubsub#config-node
http://jabber.org/protocol/pubsub#create-nodes
http://jabber.org/protocol/pubsub#delete-any
http://jabber.org/protocol/pubsub#delete-nodes
http://jabber.org/protocol/pubsub#groupblog
http://jabber.org/protocol/pubsub#instant-nodes
http://jabber.org/protocol/pubsub#item-ids
http://jabber.org/protocol/pubsub#meta-data
http://jabber.org/protocol/pubsub#outcast-affiliation
http://jabber.org/protocol/pubsub#persistent-items
http://jabber.org/protocol/pubsub#publish
http://jabber.org/protocol/pubsub#publish-options
http://jabber.org/protocol/pubsub#publisher-affiliation
http://jabber.org/protocol/pubsub#purge-nodes
http://jabber.org/protocol/pubsub#retract-items
http://jabber.org/protocol/pubsub#retrieve-affiliations
http://jabber.org/protocol/pubsub#retrieve-default
http://jabber.org/protocol/pubsub#retrieve-items
http://jabber.org/protocol/pubsub#retrieve-subscriptions
http://jabber.org/protocol/pubsub#subscribe
https://salut-a-toi.org/spec/pubsub_admin:0
https://salut-a-toi/protocol/schema:0
jabber:iq:version
urn:xmpp:mam:2
urn:xmpp:order-by:0
Identities
┌───────┬─────────┬───────────────────────────┐
│catego<span class="w"> </span>│<span class="w"> </span><span class="nb">type</span><span class="w"> </span>│<span class="w"> </span>name<span class="w"> </span>│
├┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│pubsub<span class="w"> </span>│<span class="w"> </span>service<span class="w"> </span>│<span class="w"> </span>Salut<span class="w"> </span>à<span class="w"> </span>Toi<span class="w"> </span>pubsub<span class="w"> </span>service│
└───────┴─────────┴───────────────────────────┘
Items
┌─────────────────┬───────────────────┬─┐
│entity<span class="w"> </span>│<span class="w"> </span>node<span class="w"> </span>│<span class="w"> </span>│
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┤
│pubsub.agayon.be<span class="w"> </span>│<span class="w"> </span>be.agayon.errol:0<span class="w"> </span>│<span class="w"> </span>│
│pubsub.agayon.be<span class="w"> </span>│<span class="w"> </span>tests<span class="w"> </span>│<span class="w"> </span>│
│pubsub.agayon.be<span class="w"> </span>│<span class="w"> </span>Agayon<span class="w"> </span>│<span class="w"> </span>│
└─────────────────┴───────────────────┴─┘
</code></pre></div>
<p>The following commands are used to create a node, subscribe to it and edit an item (see <a href="https://blog.agayon.be/sat_jp.html">the previous article</a>).</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>node<span class="w"> </span>create<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-n<span class="w"> </span>node_name
$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>subscribe<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-n<span class="w"> </span>node_name
$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>edit<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-n<span class="w"> </span>node_name
</code></pre></div>
<p>The default text editor is then opened. It is possible to directly edit a XML file. This command is useful for debugging purpose.</p>
<h1 id="agayon">About Agayon XMPP service</h1>
<p>All the tests described in this blog are realized on the Agayon.be XMPP service. It is not opened for registration but having this field of experimentation is great to learn, practice
and question the potential uses of XMPP. I use it with several accounts, depending on my use:</p>
<ul>
<li>obtain statistics about the server with <a href="https://gitlab.com/r1dScripts/chatty_server">chatty_server</a>, a <a href="https://blog.agayon.be/chatty_server.html">small python bot</a>.</li>
<li>write memos to save links in a dummy account,</li>
<li>automatic file transfer, </li>
<li>daily logs sender with <a href="https://sourceforge.net/projects/logwatch/files/">logwatch</a></li>
<li>...</li>
</ul>
<p><a href="https://xmpp.org/extensions/xep-0375.html">Most moderns XEPs</a> are enabled in order to provide an up-to-date experience: <a href="https://xmpp.org/extensions/xep-0163.html">PEP</a>, <a href="https://xmpp.org/extensions/xep-0280.html">Carbon</a>, <a href="https://xmpp.org/extensions/xep-0198.html">Stream Management</a>, <a href="https://xmpp.org/extensions/xep-0060.html">persistent pubsub</a>, <a href="https://xmpp.org/extensions/xep-0313.html">MAM</a>, <a href="https://xmpp.org/extensions/xep-0368.html">SRV records over TLS</a> (useful to pass blocking WiFi accesses), etc.</p>
<h1 id="future">Future of SàT</h1>
<p>In a near future, the SàT project will make great announcements about new uses of pubsub. I hope this article will make you want to install sat_pubsub and experiment with nodes, data and notifications.</p>
<p>Stay tuned !</p>
<h1 id="links">Links</h1>
<ul>
<li><a href="https://repos.goffi.org/sat_pubsub/">sat_pubsub repository</a></li>
<li><a href="https://repos.goffi.org/sat_pubsub/file/tip/INSTALL">Install notes</a> (maybe outdated)</li>
<li><a href="https://blog.agayon.be/sat_jp.html">The previous article about JP</a></li>
</ul>JP, a powerful command line interface for Salut-à-Toi2017-10-17T20:00:00+02:002019-05-30T16:00:00+02:00Arnaudtag:blog.agayon.be,2017-10-17:/sat_jp.html
<p><a href="https://salut-a-toi.org/">Salut à toi</a> is a unique XMPP client. As its official description says, it's a "multipurpose, multi front-end, free (libre) and decentralized communication tool". It has been actively developed by Jérôme Poisson (Goffi) and Adrien Cossa (Souliane) since 2008. Today, I will focus on the use of "<a href="https://wiki.goffi.org/wiki/Jp/en">JP</a>", a command-line interface. It can be used to send or receive files directly from a shell, pipe commands to or from XMPP, use XMPP easily in a script and of course play with pubsub nodes.</p>
<p><a href="https://salut-a-toi.org/">Salut à toi</a> is a unique XMPP client. As its official description says, it's a "multipurpose, multi front-end, free (libre) and decentralized communication tool". It has been actively developed by Jérôme Poisson (Goffi) and Adrien Cossa (Souliane) since 2008. Today, I will focus on the use of "<a href="https://wiki.goffi.org/wiki/Jp/en">JP</a>", a command-line interface. It can be used to send or receive files directly from a shell, pipe commands to or from XMPP, use XMPP easily in a script and of course play with pubsub nodes.</p>
<p>The following article describes uses of JP. Several of them are availabled in the <a href="https://repos.goffi.org/sat/">trunk version of Salut-à-Toi</a>.</p>
<h1>Introduction</h1>
<p>JP can be used to launch complexes commands in script, for debugging purpose or to explore XMPP services.
JP is the command line interface and it connect to the daemon Salut-à-Toi (SàT). You can share the session between the front-ends:</p>
<ul>
<li><a href="https://wiki.goffi.org/wiki/Jp/en">JP</a> of course.</li>
<li><a href="https://wiki.goffi.org/wiki/Primitivus/en">Primitivus</a> as a console front-end.</li>
<li><a href="https://wiki.goffi.org/wiki/Libervia/en">Livervia</a>, a web based front-end.</li>
<li><a href="https://salut-a-toi.org/static/images/screenshots/sententia/sententia.png">Sententia</a> an Emacs front-end (WIP).</li>
<li><a href="https://www.goffi.org/blog/goffi/9ff9bb64-1e1f-4334-80e5-a86f0b6b795b?tag=jabber-xmpp">Cagou</a> an original mobile XMPP client based on SàT (WIP).</li>
</ul>
<p>If you want to send a file easily, discover which services are available on a server, send messages in your scripts, manage your xmpp account, control video player, edit your blog post with your favorite editor, pipe streams, manage your pubsub nodes etc, JP is for you!</p>
<h1>Usage</h1>
<p>JP can be used as a command line tools or in a small shell environment.</p>
<p>First, we need to configure and launch the daemon sat.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sat
</code></pre></div>
<p>Your default profile will be connected. If you have no profile, JP can be used to create one. </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>profile<span class="w"> </span>create<span class="w"> </span>-j<span class="w"> </span>my_jid@example.org<span class="w"> </span>-x<span class="w"> </span>mypassword<span class="w"> </span>profile_name
$<span class="w"> </span>jp<span class="w"> </span>profile<span class="w"> </span>connect<span class="w"> </span>-p<span class="w"> </span>profile_name
</code></pre></div>
<p>Your password is saved in the sat settings.
You can connect automatically with the option <code>-c</code>. It should be noted that SàT defines a default profile. It can be bypassed with the option <code>-p</code>. [ref]It is possible to use <code>jp profile modify</code> to set the default profile.[/ref]</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"test message"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jp<span class="w"> </span>message<span class="w"> </span>send<span class="w"> </span>recipient@example.org<span class="w"> </span>-p<span class="w"> </span>profile_name<span class="w"> </span>-c
</code></pre></div>
<p>You can obtain help about a command with the option <code>-h</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>-h
usage:<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span><span class="o">[</span>-h<span class="o">]</span>
<span class="w"> </span><span class="o">{</span>get,delete,edit,subscribe,unsubscribe,subscriptions,node,affiliations,search,hook,uri<span class="o">}</span>
<span class="w"> </span>...
positional<span class="w"> </span>arguments:
<span class="w"> </span><span class="o">{</span>get,delete,edit,subscribe,unsubscribe,subscriptions,node,affiliations,search,hook,uri<span class="o">}</span>
<span class="w"> </span>get<span class="w"> </span>get<span class="w"> </span>pubsub<span class="w"> </span>item<span class="o">(</span>s<span class="o">)</span>
<span class="w"> </span>delete<span class="w"> </span>delete<span class="w"> </span>an<span class="w"> </span>item
<span class="w"> </span>edit<span class="w"> </span>edit<span class="w"> </span>an<span class="w"> </span>existing<span class="w"> </span>or<span class="w"> </span>new<span class="w"> </span>pubsub<span class="w"> </span>item
<span class="w"> </span>subscribe<span class="w"> </span>subscribe<span class="w"> </span>to<span class="w"> </span>a<span class="w"> </span>node
<span class="w"> </span>unsubscribe<span class="w"> </span>unsubscribe<span class="w"> </span>from<span class="w"> </span>a<span class="w"> </span>node
<span class="w"> </span>subscriptions<span class="w"> </span>retrieve<span class="w"> </span>all<span class="w"> </span>subscriptions<span class="w"> </span>on<span class="w"> </span>a<span class="w"> </span>service
<span class="w"> </span>node<span class="w"> </span>node<span class="w"> </span>handling
<span class="w"> </span>affiliations<span class="w"> </span>retrieve<span class="w"> </span>all<span class="w"> </span>affiliations<span class="w"> </span>on<span class="w"> </span>a<span class="w"> </span>service
<span class="w"> </span>search<span class="w"> </span>search<span class="w"> </span>items<span class="w"> </span>corresponding<span class="w"> </span>to<span class="w"> </span>filters
<span class="w"> </span>hook<span class="w"> </span>trigger<span class="w"> </span>action<span class="w"> </span>on<span class="w"> </span>Pubsub<span class="w"> </span>notifications
<span class="w"> </span>uri<span class="w"> </span>build<span class="w"> </span>URI
optional<span class="w"> </span>arguments:
<span class="w"> </span>-h,<span class="w"> </span>--help<span class="w"> </span>show<span class="w"> </span>this<span class="w"> </span><span class="nb">help</span><span class="w"> </span>message<span class="w"> </span>and<span class="w"> </span><span class="nb">exit</span>
</code></pre></div>
<p>JP is a Swiss army knife. Let's discover its possibilities through a few examples.</p>
<h1>Examples</h1>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>-h
usage:<span class="w"> </span>jp<span class="w"> </span><span class="o">[</span>-h<span class="o">]</span><span class="w"> </span><span class="o">[</span>--version<span class="o">]</span>
<span class="w"> </span><span class="o">{</span>file,input,uri,message,event,info,account,param,debug,ad-hoc,ticket,invitation,profile,shell,avatar,pipe,pubsub,bookmarks,roster,identity,blog<span class="o">}</span>
<span class="w"> </span>...
This<span class="w"> </span>software<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span><span class="nb">command</span><span class="w"> </span>line<span class="w"> </span>tool<span class="w"> </span><span class="k">for</span><span class="w"> </span>XMPP.
Get<span class="w"> </span>the<span class="w"> </span>latest<span class="w"> </span>version<span class="w"> </span>at<span class="w"> </span>http://salut-a-toi.org
optional<span class="w"> </span>arguments:
<span class="w"> </span>-h,<span class="w"> </span>--help<span class="w"> </span>show<span class="w"> </span>this<span class="w"> </span><span class="nb">help</span><span class="w"> </span>message<span class="w"> </span>and<span class="w"> </span><span class="nb">exit</span>
<span class="w"> </span>--version<span class="w"> </span>show<span class="w"> </span>programʼs<span class="w"> </span>version<span class="w"> </span>number<span class="w"> </span>and<span class="w"> </span><span class="nb">exit</span>
Available<span class="w"> </span>commands:
<span class="w"> </span><span class="o">{</span>file,input,uri,message,event,info,account,param,debug,ad-hoc,ticket,invitation,profile,shell,avatar,pipe,pubsub,bookmarks,roster,identity,blog<span class="o">}</span>
<span class="w"> </span>file<span class="w"> </span>File<span class="w"> </span>sending/receiving
<span class="w"> </span>input<span class="w"> </span>launch<span class="w"> </span><span class="nb">command</span><span class="w"> </span>with<span class="w"> </span>external<span class="w"> </span>input
<span class="w"> </span>uri<span class="w"> </span>XMPP<span class="w"> </span>URI<span class="w"> </span>parsing/generation
<span class="w"> </span>message<span class="w"> </span>messages<span class="w"> </span>handling
<span class="w"> </span>event<span class="w"> </span>event<span class="w"> </span>management
<span class="w"> </span>info<span class="w"> </span>Get<span class="w"> </span>various<span class="w"> </span>pieces<span class="w"> </span>of<span class="w"> </span>information<span class="w"> </span>on<span class="w"> </span>entities
<span class="w"> </span>account<span class="w"> </span>XMPP<span class="w"> </span>account<span class="w"> </span>management
<span class="w"> </span>param<span class="w"> </span>Save/load<span class="w"> </span>parameters<span class="w"> </span>template
<span class="w"> </span>debug<span class="w"> </span>debugging<span class="w"> </span>tools
<span class="w"> </span>ad-hoc<span class="w"> </span>Ad-hoc<span class="w"> </span>commands
<span class="w"> </span>ticket<span class="w"> </span>tickets<span class="w"> </span>handling
<span class="w"> </span>invitation<span class="w"> </span>invitation<span class="w"> </span>of<span class="w"> </span>user<span class="o">(</span>s<span class="o">)</span><span class="w"> </span>without<span class="w"> </span>XMPP<span class="w"> </span>account
<span class="w"> </span>profile<span class="w"> </span>profile<span class="w"> </span>commands
<span class="w"> </span>shell<span class="w"> </span>launch<span class="w"> </span>jp<span class="w"> </span><span class="k">in</span><span class="w"> </span>shell<span class="w"> </span><span class="o">(</span>REPL<span class="o">)</span><span class="w"> </span>mode
<span class="w"> </span>avatar<span class="w"> </span>avatar<span class="w"> </span>uploading/retrieving
<span class="w"> </span>pipe<span class="w"> </span>stream<span class="w"> </span>piping<span class="w"> </span>through<span class="w"> </span>XMPP
<span class="w"> </span>pubsub<span class="w"> </span>PubSub<span class="w"> </span>nodes/items<span class="w"> </span>management
<span class="w"> </span>bookmarks<span class="w"> </span>manage<span class="w"> </span>bookmarks
<span class="w"> </span>roster<span class="w"> </span>Manage<span class="w"> </span>an<span class="w"> </span>entityʼs<span class="w"> </span>roster
<span class="w"> </span>identity<span class="w"> </span>identity<span class="w"> </span>management
<span class="w"> </span>blog<span class="w"> </span>blog/microblog<span class="w"> </span>management
</code></pre></div>
<h2>Send a message</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Hello World"</span><span class="w"> </span>><span class="w"> </span>filetest
$<span class="w"> </span>jp<span class="w"> </span>message<span class="w"> </span>send<span class="w"> </span>recipient@example.org<span class="w"> </span><<span class="w"> </span>filetest
</code></pre></div>
<p>or</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"test jp"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jp<span class="w"> </span>message<span class="w"> </span>send<span class="w"> </span>recipient@example.org
</code></pre></div>
<h2>Send files</h2>
<p>The following command allows to send the file <code>file.txt</code> to recipient@example.org.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>file<span class="w"> </span>send<span class="w"> </span>file.txt<span class="w"> </span>recipient@example.org
</code></pre></div>
<h2>Receive files</h2>
<p>The following command allows to receive a file in the /tmp directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>file<span class="w"> </span>receive<span class="w"> </span>-p<span class="w"> </span>agayon<span class="w"> </span>--path<span class="w"> </span>/tmp
</code></pre></div>
<h2>Get information about a server</h2>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>info<span class="w"> </span>version<span class="w"> </span>agayon.be
Client<span class="w"> </span>name:<span class="w"> </span>Prosody
Client<span class="w"> </span>version:<span class="w"> </span><span class="m">0</span>.10.0
Operating<span class="w"> </span>System:<span class="w"> </span>Linux
</code></pre></div>
<h2>Get disco information</h2>
<h3>Query a Server</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>info<span class="w"> </span>disco<span class="w"> </span>agayon.be
Features
http://jabber.org/protocol/commands
http://jabber.org/protocol/disco#info
http://jabber.org/protocol/disco#items
http://jabber.org/protocol/pubsub#auto-create
http://jabber.org/protocol/pubsub#config-node
http://jabber.org/protocol/pubsub#create-nodes
http://jabber.org/protocol/pubsub#delete-any
http://jabber.org/protocol/pubsub#delete-nodes
http://jabber.org/protocol/pubsub#groupblog
http://jabber.org/protocol/pubsub#instant-nodes
http://jabber.org/protocol/pubsub#item-ids
http://jabber.org/protocol/pubsub#meta-data
http://jabber.org/protocol/pubsub#outcast-affiliation
http://jabber.org/protocol/pubsub#persistent-items
http://jabber.org/protocol/pubsub#publish
http://jabber.org/protocol/pubsub#publish-options
http://jabber.org/protocol/pubsub#publisher-affiliation
http://jabber.org/protocol/pubsub#purge-nodes
http://jabber.org/protocol/pubsub#retract-items
http://jabber.org/protocol/pubsub#retrieve-affiliations
http://jabber.org/protocol/pubsub#retrieve-default
http://jabber.org/protocol/pubsub#retrieve-items
http://jabber.org/protocol/pubsub#retrieve-subscriptions
http://jabber.org/protocol/pubsub#subscribe
https://salut-a-toi.org/spec/pubsub_admin:0
https://salut-a-toi/protocol/schema:0
jabber:iq:auth
jabber:iq:last
jabber:iq:private
jabber:iq:roster
jabber:iq:time
jabber:iq:version
msgoffline
urn:xmpp:blocking
urn:xmpp:carbons:1
urn:xmpp:carbons:2
urn:xmpp:http:upload
urn:xmpp:http:upload:0
urn:xmpp:mam:0
urn:xmpp:order-by:0
urn:xmpp:ping
urn:xmpp:time
vcard-temp
Identities
┌───────┬──────┬─────────────────┐
│catego<span class="w"> </span>│<span class="w"> </span><span class="nb">type</span><span class="w"> </span>│<span class="w"> </span>name<span class="w"> </span>│
├┄┄┄┄┄┄┄┼┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│store<span class="w"> </span>│<span class="w"> </span>file<span class="w"> </span>│<span class="w"> </span>HTTP<span class="w"> </span>File<span class="w"> </span>Upload│
│pubsub<span class="w"> </span>│<span class="w"> </span>pep<span class="w"> </span>│<span class="w"> </span>Prosody<span class="w"> </span>│
│server<span class="w"> </span>│<span class="w"> </span>im<span class="w"> </span>│<span class="w"> </span>Prosody<span class="w"> </span>│
└───────┴──────┴─────────────────┘
Extensions
urn:xmpp:http:upload
<span class="w"> </span>type:<span class="w"> </span>text-single
<span class="w"> </span>var:<span class="w"> </span>max-file-size
<span class="w"> </span>value:<span class="w"> </span><span class="m">1048576</span>
urn:xmpp:http:upload:0
<span class="w"> </span>type:<span class="w"> </span>text-single
<span class="w"> </span>var:<span class="w"> </span>max-file-size
<span class="w"> </span>value:<span class="w"> </span><span class="m">1048576</span>
Items
┌──────────────────────┬──┬────────────────────┐
│entity<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>name<span class="w"> </span>│
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│test-pubsub.agayon.be<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│
│upload.agayon.be<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│
│auth.agayon.be<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│
│proxy.agayon.be<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│
│pubsub.agayon.be<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>│
│chat.agayon.be<span class="w"> </span>│<span class="w"> </span>│<span class="w"> </span>Agayon.be<span class="w"> </span>chatrooms│
└──────────────────────┴──┴────────────────────┘
</code></pre></div>
<h3>Query an account</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>info<span class="w"> </span>disco<span class="w"> </span>info@agayon.be
Items
┌───────────────┬───────────────────────────────────────────────────┬─┐
│entity<span class="w"> </span>│<span class="w"> </span>node<span class="w"> </span>│<span class="w"> </span>│
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┤
│info@agayon.be<span class="w"> </span>│<span class="w"> </span>eu.siacs.conversations.axolotl.bundles:1948395845<span class="w"> </span>│<span class="w"> </span>│
│info@agayon.be<span class="w"> </span>│<span class="w"> </span>http://jabber.org/protocol/activity<span class="w"> </span>│<span class="w"> </span>│
│info@agayon.be<span class="w"> </span>│<span class="w"> </span>http://jabber.org/protocol/mood<span class="w"> </span>│<span class="w"> </span>│
│info@agayon.be<span class="w"> </span>│<span class="w"> </span>eu.siacs.conversations.axolotl.devicelist<span class="w"> </span>│<span class="w"> </span>│
│info@agayon.be<span class="w"> </span>│<span class="w"> </span>storage:bookmarks<span class="w"> </span>│<span class="w"> </span>│
└───────────────┴───────────────────────────────────────────────────┴─┘
</code></pre></div>
<h2>Manage pubsub nodes</h2>
<p>The following examples show how to manage pubsub nodes on the service <a href="http://pubsub.agayon.be">pubsub.agayon.be</a>.</p>
<h3>Create a node</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>node<span class="w"> </span>create<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-n<span class="w"> </span>node_name
</code></pre></div>
<h3>Subscribe to a node</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>subscribe<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-n<span class="w"> </span>node_name
</code></pre></div>
<h3>Edit</h3>
<p>The edit command allows to edit an item under a node.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>pubsub<span class="w"> </span>edit<span class="w"> </span>-s<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>-n<span class="w"> </span>node_name
</code></pre></div>
<p>The default text editor is opened. It is possible to directly edit a XML file. This command is useful for debugging purpose.</p>
<p>If you want to edit a post without having to edit xml directly, use <a href="#blog_edit">jp blog edit</a>. </p>
<p>As an example, you can try to edit the following xml file.</p>
<div class="highlight"><pre><span></span><code><span class="nt"><entry</span><span class="w"> </span><span class="na">xmlns=</span><span class="s">"http://www.w3.org/2005/Atom"</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><author></span>
<span class="w"> </span><span class="nt"><name></span>author@example.com<span class="nt"></name></span>
<span class="w"> </span><span class="nt"></author></span>
<span class="w"> </span><span class="nt"><generator></span>JP<span class="w"> </span>(SàT)<span class="nt"></generator></span>
<span class="w"> </span><span class="nt"><title></span>I<span class="w"> </span>am<span class="w"> </span>a<span class="w"> </span>pubsub<span class="w"> </span>post<span class="w"> </span>!<span class="nt"></title></span>
<span class="w"> </span><span class="nt"><content></span>This<span class="w"> </span>is<span class="w"> </span>the<span class="w"> </span>content<span class="w"> </span>of<span class="w"> </span>this<span class="w"> </span>great<span class="w"> </span>post.<span class="nt"></content></span>
<span class="nt"></entry></span>
</code></pre></div>
<p>Once the file is saved, a notification appears in Gajim. For now, the name of the post is a hash.</p>
<p><img alt="pubsub post on Gajim" src="https://blog.agayon.be/images/pubsub_1.png" style="width: 272px; height: auto; max-width: 100%;"/></p>
<p><img alt="pubsub post on Gajim" src="https://blog.agayon.be/images/pubsub_2.png" style="width: 870px; height: auto; max-width: 100%;"/></p>
<h2>Manage your XMPP Blog</h2>
<p>It is possible to manage a post with XMPP. It is based on <a href="https://xmpp.org/extensions/xep-0163.html">PEP</a>, a simplified version of pubsub. If your server support PEP, JP can help you to manage this blog easily.</p>
<h3 id="blog_edit">Publish a post</h3>
<p>First, you need to define your preferred syntax. In this example, I select markdown. This option can also be set in any other frontend (e.g. Primitivus). Whereafter you can edit a new post with the syntax jp blog edit.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>param<span class="w"> </span><span class="nb">set</span><span class="w"> </span>Composition<span class="w"> </span>Syntax<span class="w"> </span>markdown<span class="w"> </span>-p<span class="w"> </span>agayon
$<span class="w"> </span>jp<span class="w"> </span>blog<span class="w"> </span>edit
</code></pre></div>
<p>Your favorite editor open and you can edit your blog post with markdown syntax. When you save and close it, another file open. You can edit your settings:</p>
<div class="highlight"><pre><span></span><code><span class="o">{</span>
<span class="w"> </span><span class="s2">"allow_comments"</span>:<span class="w"> </span>true,
<span class="o">}</span>
</code></pre></div>
<p>You can verify the accessibility of your post with the following command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>blog<span class="w"> </span>get<span class="w"> </span>
content:<span class="w"> </span>
Great<span class="w"> </span><span class="nb">test</span><span class="w"> </span>!
Make<span class="w"> </span>testing<span class="w"> </span>great<span class="w"> </span>again<span class="w"> </span>!
Yuuuuuuge<span class="w"> </span>publication.
</code></pre></div>
<p>It is also possible to modify your last blog post simply with <code>jp blog edit --last-item</code>.</p>
<p>Your blog post are also visible on other clients like <a href="https://movim.eu/">Movim</a> (see below) and interfaces like <a href="https://wiki.goffi.org/wiki/Libervia/en">Livervia</a>.</p>
<p><img alt="Post on movim" src="https://blog.agayon.be/images/movim_post.png.webp" style="width: 1158px; height: auto; max-width: 100%;"/></p>
<h2>Use jp shell:</h2>
<p>In order to ease debugging of services, JP comes with a shell interface.
You only need to launch <code>jp shell</code>. You can obtain help by typing <code>?</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>shell
cmd<span class="w"> </span>pubsub
pubsub><span class="w"> </span>?
Shell<span class="w"> </span>commands:
Documented<span class="w"> </span>commands<span class="w"> </span><span class="o">(</span><span class="nb">type</span><span class="w"> </span><span class="nb">help</span><span class="w"> </span><topic><span class="o">)</span>:
<span class="o">========================================</span>
cmd<span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">help</span><span class="w"> </span>shell<span class="w"> </span>use_clear<span class="w"> </span>version
debug<span class="w"> </span><span class="nb">exit</span><span class="w"> </span>quit<span class="w"> </span>use<span class="w"> </span>verbose<span class="w"> </span>whoami<span class="w"> </span>
Action<span class="w"> </span>commands:
positional<span class="w"> </span>arguments:
<span class="w"> </span><span class="o">{</span>get,delete,edit,subscribe,unsubscribe,subscriptions,node,affiliations,search,hook,uri<span class="o">}</span>
<span class="w"> </span>get<span class="w"> </span>get<span class="w"> </span>pubsub<span class="w"> </span>item<span class="o">(</span>s<span class="o">)</span>
<span class="w"> </span>delete<span class="w"> </span>delete<span class="w"> </span>an<span class="w"> </span>item
<span class="w"> </span>edit<span class="w"> </span>edit<span class="w"> </span>an<span class="w"> </span>existing<span class="w"> </span>or<span class="w"> </span>new<span class="w"> </span>pubsub<span class="w"> </span>item
<span class="w"> </span>subscribe<span class="w"> </span>subscribe<span class="w"> </span>to<span class="w"> </span>a<span class="w"> </span>node
<span class="w"> </span>unsubscribe<span class="w"> </span>unsubscribe<span class="w"> </span>from<span class="w"> </span>a<span class="w"> </span>node
<span class="w"> </span>subscriptions<span class="w"> </span>retrieve<span class="w"> </span>all<span class="w"> </span>subscriptions<span class="w"> </span>on<span class="w"> </span>a<span class="w"> </span>service
<span class="w"> </span>node<span class="w"> </span>node<span class="w"> </span>handling
<span class="w"> </span>affiliations<span class="w"> </span>retrieve<span class="w"> </span>all<span class="w"> </span>affiliations<span class="w"> </span>on<span class="w"> </span>a<span class="w"> </span>service
<span class="w"> </span>search<span class="w"> </span>search<span class="w"> </span>items<span class="w"> </span>corresponding<span class="w"> </span>to<span class="w"> </span>filters
<span class="w"> </span>hook<span class="w"> </span>trigger<span class="w"> </span>action<span class="w"> </span>on<span class="w"> </span>Pubsub<span class="w"> </span>notifications
<span class="w"> </span>uri<span class="w"> </span>build<span class="w"> </span>URI
</code></pre></div>
<h3>Select a command</h3>
<div class="highlight"><pre><span></span><code>><span class="w"> </span>cmd<span class="w"> </span>pubsub
pubsub><span class="w"> </span>use<span class="w"> </span>node<span class="w"> </span>urn:xmpp:microblog:0
</code></pre></div>
<h3>Navigate into commands</h3>
<div class="highlight"><pre><span></span><code>><span class="w"> </span>cmd<span class="w"> </span>bookmarks/list
bookmarks/list><span class="w"> </span>-c
...<span class="w"> </span>CMD<span class="w"> </span>result
bookmarks/list><span class="w"> </span>cmd<span class="w"> </span>..
bookmarks><span class="w"> </span>cmd
>
</code></pre></div>
<h3>Example: List bookmarks</h3>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>jp<span class="w"> </span>shell
><span class="w"> </span>cmd<span class="w"> </span>bookmarks
bookmarks><span class="w"> </span>cmd<span class="w"> </span>list
bookmarks/list><span class="w"> </span>?
Shell<span class="w"> </span>commands:
Documented<span class="w"> </span>commands<span class="w"> </span><span class="o">(</span><span class="nb">type</span><span class="w"> </span><span class="nb">help</span><span class="w"> </span><topic><span class="o">)</span>:
<span class="o">========================================</span>
cmd<span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nb">help</span><span class="w"> </span>shell<span class="w"> </span>use_clear<span class="w"> </span>version
debug<span class="w"> </span><span class="nb">exit</span><span class="w"> </span>quit<span class="w"> </span>use<span class="w"> </span>verbose<span class="w"> </span>whoami<span class="w"> </span>
Action<span class="w"> </span>commands:
optional<span class="w"> </span>arguments:
<span class="w"> </span>-h,<span class="w"> </span>--help<span class="w"> </span>show<span class="w"> </span>this<span class="w"> </span><span class="nb">help</span><span class="w"> </span>message<span class="w"> </span>and<span class="w"> </span><span class="nb">exit</span>
<span class="w"> </span>-p<span class="w"> </span>PROFILE,<span class="w"> </span>--profile<span class="w"> </span>PROFILE
<span class="w"> </span>Use<span class="w"> </span>PROFILE<span class="w"> </span>profile<span class="w"> </span>key<span class="w"> </span><span class="o">(</span>default:<span class="w"> </span>@DEFAULT@<span class="o">)</span>
<span class="w"> </span>--pwd<span class="w"> </span>PASSWORD<span class="w"> </span>Password<span class="w"> </span>used<span class="w"> </span>to<span class="w"> </span>connect<span class="w"> </span>profile,<span class="w"> </span><span class="k">if</span><span class="w"> </span>necessary
<span class="w"> </span>-c,<span class="w"> </span>--connect<span class="w"> </span>Connect<span class="w"> </span>the<span class="w"> </span>profile<span class="w"> </span>before<span class="w"> </span>doing<span class="w"> </span>anything<span class="w"> </span><span class="k">else</span>
<span class="w"> </span>-l<span class="w"> </span><span class="o">{</span>all,local,private,pubsub<span class="o">}</span>,<span class="w"> </span>--location<span class="w"> </span><span class="o">{</span>all,local,private,pubsub<span class="o">}</span>
<span class="w"> </span>storage<span class="w"> </span>location<span class="w"> </span><span class="o">(</span>default:<span class="w"> </span>all<span class="o">)</span>
<span class="w"> </span>-t<span class="w"> </span><span class="o">{</span>muc,url<span class="o">}</span>,<span class="w"> </span>--type<span class="w"> </span><span class="o">{</span>muc,url<span class="o">}</span>
<span class="w"> </span>bookmarks<span class="w"> </span><span class="nb">type</span><span class="w"> </span><span class="o">(</span>default:<span class="w"> </span>muc<span class="o">)</span>
bookmarks/list><span class="w"> </span>-t<span class="w"> </span>muc
private:
<span class="w"> </span>Movim<span class="w"> </span><span class="o">[</span>movim@conference.movim.eu<span class="o">]</span><span class="w"> </span><span class="o">(</span>*<span class="o">)</span>
<span class="w"> </span>Archlinux<span class="w"> </span>-<span class="w"> </span>blah<span class="w"> </span>blah<span class="w"> </span><span class="o">[</span>archlinux-fr@chat.jabberfr.org<span class="o">]</span>
<span class="w"> </span>bot<span class="w"> </span><span class="o">[</span>bot@chat.agayon.be<span class="o">]</span><span class="w"> </span><span class="o">(</span>*<span class="o">)</span>
</code></pre></div>
<h3>Example: get disco information</h3>
<div class="highlight"><pre><span></span><code>><span class="w"> </span>cmd<span class="w"> </span>info
info><span class="w"> </span>use<span class="w"> </span>jid<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>
info><span class="w"> </span>disco
Features
http://jabber.org/protocol/disco#info
http://jabber.org/protocol/disco#items
http://jabber.org/protocol/pubsub
http://jabber.org/protocol/pubsub#create-nodes
http://jabber.org/protocol/pubsub#delete-items
http://jabber.org/protocol/pubsub#delete-nodes
http://jabber.org/protocol/pubsub#instant-nodes
http://jabber.org/protocol/pubsub#item-ids
http://jabber.org/protocol/pubsub#publish
http://jabber.org/protocol/pubsub#publisher-affiliation
http://jabber.org/protocol/pubsub#purge-nodes
http://jabber.org/protocol/pubsub#retract-items
http://jabber.org/protocol/pubsub#retrieve-items
http://jabber.org/protocol/pubsub#retrieve-subscriptions
http://jabber.org/protocol/pubsub#subscribe
Identities
┌───────┬─────────┬───────────────────────┐
│catego<span class="w"> </span>│<span class="w"> </span><span class="nb">type</span><span class="w"> </span>│<span class="w"> </span>name<span class="w"> </span>│
├┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│pubsub<span class="w"> </span>│<span class="w"> </span>service<span class="w"> </span>│<span class="w"> </span>Prosody<span class="w"> </span>PubSub<span class="w"> </span>Service│
└───────┴─────────┴───────────────────────┘
</code></pre></div>
<h1>More</h1>
<ul>
<li><a href="https://www.goffi.org/blog/goffi/S%C3%A0T_DOTCLEAR_IMPORT_BLOG_default_goffi_77%3A2013%2F02%2F22%2FExport-command-to-a-contact-%28with-video%29?tag=jabber-xmpp-en">Export command (input/output) to a contact</a></li>
<li><a href="https://www.goffi.org/blog/goffi/S%C3%A0T_DOTCLEAR_IMPORT_BLOG_default_goffi_83%3A2014%2F02%2F18%2FA-universal-remote-for-your-softwares?tag=jabber-xmpp-en">Create a universal remote for VLC (media player)</a></li>
</ul>
<h1>Some alternatives</h1>
<h4>SleekXMPP</h4>
<p>SleekXMPP is a xmpp python library. It provide two examples of pubsub clients. The <a href="https://github.com/fritzy/SleekXMPP/blob/develop/examples/pubsub_client.py">first, <code>pubsub_client.py</code></a> can manage nodes and the <a href="https://github.com/fritzy/SleekXMPP/blob/develop/examples/pubsub_events.py">second one, pubsub_events.py,</a> can manage events.</p>
<div class="highlight"><pre><span></span><code>./pubsub_client.py<span class="w"> </span>-j<span class="w"> </span>jid<span class="w"> </span>--password<span class="o">=</span>passwd<span class="w"> </span>pubsub.agayon.be<span class="w"> </span>create<span class="w"> </span>node_name
</code></pre></div>
<h3>Gajim</h3>
<p><a href="https://gajim.org/">Gajim</a> is a great desktop client. It can be used to post on pubsub nodes too.</p>
<p><img alt="pubsub post on Gajim" src="https://blog.agayon.be/images/pubsub_3.png.webp" style="width: 450px; height: auto; max-width: 100%;"/></p>
<h3>Movim</h3>
<p><a href="https://movim.eu/">Movim</a> is a social network platform based on XMPP. It massively use pubsub to allow users to interact.</p>
<h1>Conclusions</h1>
<p>Salut-à-Toi is a great XMPP client. JP, its command-line interface, is a powerful tool. It can be useful to debug services, pubsub nodes and play with 'social' capacities of XMPP.
It is actively developed and new features are coming. We will be soon able to make todolist or fill a bug ticket on a XMPP based system.</p>
<h1>Links</h1>
<ul>
<li><a href="https://salut-a-toi.org/">Salut à toi</a></li>
<li>SàT room: <a href="xmpp:sat@chat.jabberfr.org?join">sat@chat.jabberfr.org</a></li>
<li><a href="https://wiki.goffi.org/wiki/Jp/en">JP</a></li>
<li><a href="https://repos.goffi.org/sat/">SàT repository</a></li>
<li><a href="https://aur.archlinux.org/packages/sat-xmpp/">Archlinux package</a></li>
<li><a href="https://gajim.org/">Gajim</a></li>
<li><a href="https://movim.eu/">Movim</a></li>
</ul>Opus helps you to find a job2017-09-17T19:00:00+02:002017-09-17T19:00:00+02:00Arnaudtag:blog.agayon.be,2017-09-17:/opus_en.html
<p>Finding a job may be a time and energy-consuming task. Opus is a program written to help people to send their application more easily.
It can be used to generate personalized cover letter. I began to write it several years ago and I took the time to enhance it with the aim to help other people.</p>
<p>Finding a job may be a time and energy-consuming task. Opus is a program written to help people to send their application more easily.
It can be used to generate personalized cover letter. I began to write it several years ago and I took the time to enhance it with the aim to help other people.</p>
<p>During this experience, I got better at python programming, I learned graphical interfaces with pyQt and how to make a windows self extracting executable.</p>
<p>Opus is basically a "mass mailing system" based on templates. The user adapts his cover letter with keywords that are substituted with data coming from an excel file.</p>
<p><strong>Opus is still in an early stage. User is responsible of its use and of the messages that are sent.</strong></p>
<h1 id="install">Install</h1>
<p>Opus is a opensource software. There is a self extracting executable availaible for windows but it is possible to use it from the sources.</p>
<p>Opus is written in python. It needs the following dependencies:</p>
<ul>
<li>appdirs</li>
<li>lxml</li>
<li>xlrd</li>
<li>PyQt5</li>
</ul>
<p>Just clone the git repository after installing the dependencies in a python virtualenv.</p>
<div class="highlight"><pre><span></span><code> git clone https://gitlab.com/jnanar/opus.git
cd opus
virtualenv venv
source venv/bin/activate
pip install .
</code></pre></div>
<p>You can test it with its demo data and configure it later (see section <a href="#getting_started">getting started</a>).</p>
<div class="highlight"><pre><span></span><code> python3 opus.py
</code></pre></div>
<p>It is also possible to download the sources as a zip file on the <a href="https://gitlab.com/jnanar/opus/tree/master">repository</a>.</p>
<h1 id="getting_started">Getting started</h1>
<p><strong>TLDR (windows)</strong>: Download <a href="https://agayon.be/repository/opus/opus_win_1.0.1.exe">Opus for windows</a> and launch it. See the generated files in directory <code>%temp%\opus\data\sent</code>. Edit <code>%temp%\opus\data\list.example.xlsx</code>, generate the letters and see what happens.</p>
<div class="highlight"><pre><span></span><code>sha256sum<span class="w"> </span>opus_win_1.0.1.exe<span class="w"> </span>
9d94f1fc41d4dc26632a0dce13120c68b4454d390f4eb59bebe1f350bb048d89<span class="w"> </span>opus_win_1.0.1.exe
</code></pre></div>
<ol>
<li>Download <a href="https://agayon.be/repository/opus/opus_start_pack_en.zip">Opus start pack</a> which contains a sample configuration file and templates. </li>
<li>Rename the configuration file to opus.ini and personalize it. (see section <a href="#config">Configuration</a>).</li>
<li>Edit or copy the repository (data/list.example.xlsx) with Libreoffice or Excel. (see <a href="#conftable">Repository edition</a>)</li>
<li>Personalize the cover letters. Two templates are available: spontaneous application and application for an offer. (see <a href="#conf_letters">Letters edition</a>)</li>
<li>Launch Opus. Two interfaces are available: CLI (in french) and a GUI. (see <a href="#interfaces">Interfaces</a>). I will assume that you use the graphical interface.</li>
<li>You can configure Opus by editing the settings. The best idea is to select the configuration file of the Opus start pack and rename the templates. You can link your own templates in the 'file' section. Once you are ready, generate the letters. </li>
<li>The files are generated in the output directory (see <a href="#config">Configuration</a>).</li>
<li><strong>Once you have edited the letters and the mails</strong> (remove the bold texted that was substituted), you can convert it to PDF. If Libreoffice 5 is installed, Opus can convert all the document at once. if not, you can convert the .docx to PDF with Word.</li>
<li>You may now send the mail. A copy is sent to your adress.
Once the mails are sent, you must edit the first column of the repository (xlsx file). Only empty first cells allows the generation of the letters.This prevent to sent multiple times the same application. I strongly advise to move the generated documents in another folder. It will help you to keep in mind what you have already sent.</li>
</ol>
<p>I strongly suggest to sent the first letters to your email in order to see what the recruter will get.</p>
<h1 id="interfaces">Interfaces</h1>
<p>Two interfaces are available: cli and gui. The launch <code>opus.py</code> with option <code>-i</code> or <code>--interface</code> to select it. By default the graphical interface is chosen.</p>
<h1>Graphical interface(GUI)</h1>
<p><img alt="Opus Gui: Cover letter generation" src="https://blog.agayon.be/images/opus/opus_gui_1.png" style="width: 407px; height: auto; max-width: 100%;"/>
<img alt="Opus Gui: Settings" src="https://blog.agayon.be/images/opus/opus_gui_2_file_selector.png" style="width: 630px; height: auto; max-width: 100%;"/>
<img alt="Opus Gui: About" src="https://blog.agayon.be/images/opus/opus_gui_3.png" style="width: 490px; height: auto; max-width: 100%;"/></p>
<h1>Command line interface (CLI)</h1>
<p>This interace is usefull but it is only avaible in french for now.</p>
<p><img alt="Opus Cli" src="https://blog.agayon.be/images/opus/opus-cli.png" style="width: 731px; height: auto; max-width: 100%;"/></p>
<h1 id="config">Configuration</h1>
<p>The <code>ini</code> file has 3 sections.</p>
<h2>Personal information</h2>
<ul>
<li>Your full name</li>
<li>The subject of mails for spontaneous applications</li>
<li>the subject of mails when responding to offers</li>
<li>Your lang (EN).</li>
</ul>
<h2>The files</h2>
<ul>
<li>Templates of letters</li>
<li>Your CV (PDF file)</li>
<li>The repository (data/list.example.xlsx by default),</li>
<li>emails templates</li>
<li>output directory (/data/sent/ by default)</li>
</ul>
<h2>Send emails</h2>
<p>You need to configure the smtp options in order to be able to send emails. These parameters depends on your email provider (eg gmail, hotmail, etc)</p>
<p>The parameters are:</p>
<ul>
<li>smtp server address</li>
<li>user name</li>
<li>password (clear text in the <code>ini</code> file).</li>
<li>port number.</li>
</ul>
<p>If one of those parameters do not need to be set, just indicate <code>None</code>. The following example show a server that does not need authentication:</p>
<div class="highlight"><pre><span></span><code>server = example.com
user = None
password = None
port = 25
</code></pre></div>
<h1 id="conftable">Repository edition</h1>
<p>Each line of the <code>xlsx</code> file corresponds to an application. The information of each column are processed and used in the cover letter and the email. The editable fields are:</p>
<ol>
<li><strong>sent</strong> indicate something if the application must not be sent. For example if you already sent the email to this company.</li>
<li><strong>company</strong>: company name</li>
<li><strong>sector</strong></li>
<li><strong>nom</strong>: if you sent the application to a person whose name is known.</li>
<li><strong>gender</strong>: the gender of the recipient (Sir, madam, Sir or madam)</li>
<li><strong>interest</strong>: your interest for the company</li>
<li><strong>offer</strong>: the offer reference</li>
<li><strong>subject</strong>: the reference of the offer to put in the subject mail.</li>
<li><strong>email</strong>: the email recipient</li>
<li><strong>language</strong>: lang: EN or FR</li>
<li><strong>web, telephone, number, information</strong>: complementary useful information. These data are not used yet.</li>
</ol>
<h1 id="conf_letters">Edition of the cover letters</h1>
<p>Several fields are available</p>
<ul>
<li><strong>TIME</strong>: the date of the day</li>
<li><strong>GENDER</strong>: the gender of the recipient (Sir, madam, Sir or madam)</li>
<li><strong>INTEREST</strong>: your interest for the company</li>
<li><strong>COMPANY</strong>: company name</li>
<li><strong>SECTOR</strong>: the sector of the company</li>
<li><strong>OFFER</strong>: the reference of the offer.</li>
</ul>
<p>The same fields are usable in the emails.</p>
<h1>Help !</h1>
<ul>
<li>The log is located in <code>%temp%\opus\opus.log</code> on windows (exe file) and in the main directory otherwise.</li>
<li>You can contact me, I will try my best to help users.</li>
</ul>
<h1>About the Windows program</h1>
<ul>
<li>I generated it with <a href="https://anthony-tuininga.github.io/cx_Freeze/">cx_Freeze</a> (see <a href="https://gitlab.com/jnanar/opus/blob/master/setup.py">setup.py</a>) and <a href="https://sourceforge.net/projects/s-zipsfxbuilder/">7z SFX Builder</a>. </li>
</ul>
<p>The following configuration file was used:</p>
<div class="highlight"><pre><span></span><code><span class="p">;</span><span class="err">!</span><span class="nv">@Install</span><span class="err">@!</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span><span class="err">!</span>
<span class="n">RunProgram</span><span class="o">=</span><span class="ss">"opus_win.exe"</span>
<span class="p">;</span><span class="n">Config</span><span class="w"> </span><span class="k">file</span><span class="w"> </span><span class="n">generated</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="mi">7</span><span class="n">z</span><span class="w"> </span><span class="n">SFX</span><span class="w"> </span><span class="n">Builder</span><span class="w"> </span><span class="n">v2</span><span class="mf">.1</span><span class="p">.)</span>
<span class="p">;</span><span class="err">!</span><span class="nv">@InstallEnd</span><span class="err">@!</span>
<span class="mi">7</span><span class="n">zSFXBuilder_7zArchive</span><span class="o">=</span><span class="nl">C</span><span class="p">:</span><span class="err">\</span><span class="n">Users</span><span class="err">\</span><span class="n">xx</span><span class="err">\</span><span class="n">windows_setup</span><span class="err">\</span><span class="n">opus</span><span class="mf">.7</span><span class="n">z</span>
<span class="mi">7</span><span class="n">zSFXBuilder_SFXName</span><span class="o">=</span><span class="nl">C</span><span class="p">:</span><span class="err">\</span><span class="n">Users</span><span class="err">\</span><span class="n">xx</span><span class="err">\</span><span class="n">windows_setup</span><span class="err">\</span><span class="n">opus_win</span><span class="p">.</span><span class="n">exe</span>
<span class="mi">7</span><span class="n">zSFXBuilder_SFXIcon</span><span class="o">=</span><span class="nl">C</span><span class="p">:</span><span class="err">\</span><span class="n">Users</span><span class="err">\</span><span class="n">xx</span><span class="err">\</span><span class="n">icon</span><span class="err">\</span><span class="n">opus</span><span class="p">.</span><span class="n">ico</span>
<span class="mi">7</span><span class="n">zSFXBuilder_UseDefMod</span><span class="o">=</span><span class="mi">7</span><span class="n">zsd_All</span>
</code></pre></div>
<h1>Conclusion</h1>
<p>Good luck !</p>
<h1>Links</h1>
<ul>
<li><a href="https://agayon.be/repository/opus/opus_win_1.0.1.exe">Opus for windows</a></li>
<li><a href="https://agayon.be/repository/opus/opus_start_pack_en.zip">Start pack</a></li>
<li><a href="https://gitlab.com/jnanar/opus/">Source code</a></li>
<li><a href="https://anthony-tuininga.github.io/cx_Freeze/">cx_Freeze</a></li>
<li><a href="http://www.7-zip.org/">7zip</a></li>
<li><a href="https://sourceforge.net/projects/s-zipsfxbuilder/">7z SFX Builder</a></li>
</ul>Opus le bouchon un peu plus loin...2017-08-22T22:00:00+02:002017-08-22T22:00:00+02:00Arnaudtag:blog.agayon.be,2017-08-22:/opus_2.html
<p>... et devient compatible avec Windows. Précédement sur ce site, je vous ai présenté le logiciel <a href="https://blog.agayon.be/presentation_opus.html">Opus</a>. Il s'agit d'un système de publipostage permettant d'automatiser la recherche d'emploi. Ayant reçu des demandes pour le porter sous windows, j'ai réalisé un portage sous forme d'un fichier exécutable auto-extractible. Pour les plus néophytes, on clique et ça se lance.</p>
<p>... et devient compatible avec Windows. Précédement sur ce site, je vous ai présenté le logiciel <a href="https://blog.agayon.be/presentation_opus.html">Opus</a>. Il s'agit d'un système de publipostage permettant d'automatiser la recherche d'emploi. Ayant reçu des demandes pour le porter sous windows, j'ai réalisé un portage sous forme d'un fichier exécutable auto-extractible. Pour les plus néophytes, on clique et ça se lance.</p>
<h1>Nouveautés</h1>
<ul>
<li>La fenêtre de configuration a été revue, notamment la sélection des fichiers et dossier de destination. </li>
<li>Le champs du mot de passe SMTP est à présent masqué. Notez que ce mot de passe est stocké en clair dans le fichier de configuration.</li>
<li>Sous windows, il est possible de convertir automatiquement les lettres au format PDF. Cette fonctionnalité nécessite que Libreoffice 5 (la dernière version à ce jour) soit installé.</li>
</ul>
<h1>Utilisation</h1>
<ul>
<li><a href="https://agayon.be/repository/opus/opus_win_1.0.exe">Téléchargez le programme</a>.</li>
<li>Les plus précautionneux pourront vérifier la signature du binaire:</li>
</ul>
<div class="highlight"><pre><span></span><code>sha256sum<span class="w"> </span>opus_win_1.0.exe
4c47c75048609e40bf0a70594e508978513912fdb7c2d2c371a3667beb22d5f1<span class="w"> </span>opus_win_1.0.exe
</code></pre></div>
<ul>
<li>Téléchargez le pack de démarrage <a href="https://agayon.be/repository/opus/opus_start_pack.zip">Opus start pack</a> contenant un fichier de configuration de démarrage et des données d'exemples. Décompressez-le dans un dossier de votre choix. Pour plus de facilités, vous pouvez renommer le fichier de configuration <code>opus.exemple.ini</code>.</li>
<li>Lancez le programme.</li>
<li>Ouvrez la fenêtre de configuration et chargez le fichier <code>.ini</code>. </li>
<li>Modifiez les paramètres à votre guise et faites pointer les fichiers canevas (lettres, mails, CV, ...) vers les données d'exemple. Vous pourrez adapter ces canevas et les renommez par la suite.</li>
<li>Cliquez sur OK. Vos modifications sont sauvées dans le fichier de configuration.</li>
<li>Générez les lettres de motivations en cliquant sur <em>Generate</em>.</li>
<li>Pour plus d'explications concernant l'utilisation et la configuration du programme, je vous renvoie vers l'<a href="https://blog.agayon.be/presentation_opus.html">article précédent</a>.</li>
</ul>
<p>Il est important de noter qu'Opus ne laisse pas de trace. Une fois que vous le quittez, les fichiers temporaires sont détruits. Il ne nécessite pas d'installation et ne va pas polluer votre machine avec des clés de registres et autres. Lorsque vous voulez le réutiliser, rechargez simplement votre fichier de configuration pour régénérer des candidatures.</p>
<h1>Captures d'écran</h1>
<p><img alt="Generate Windows" src="https://blog.agayon.be/images/opus/opus_gui_1.png" style="width: 407px; height: auto; max-width: 100%;"/>
<img alt="Nouvelle configuration" src="https://blog.agayon.be/images/opus/opus_gui_2_file_selector.png" style="width: 630px; height: auto; max-width: 100%;"/>
<img alt="Nouvelle configuration" src="https://blog.agayon.be/images/opus/opus_gui_4.png" style="width: 610px; height: auto; max-width: 100%;"/></p>
<h1>Futur</h1>
<ul>
<li>Un bouton enregistrer sous pourrait être ajouté pour faciliter l'enregistrement de la configuration.</li>
<li>Le programme prend en compte deux canevas de mails pour la génération en français et anglais. Pour l'instant, le choix des canevas est identique quelque soit la langue.</li>
</ul>
<h1>En cas de soucis</h1>
<ul>
<li>Les messages d'erreurs sont sauvés dans le fichier opus.log. Sous windows, ce fichier est disponible dans le dossier <code>%temp%\opus</code>. Il s'agit du raccourcis vers votre dossier temporaire.</li>
<li>Sinon,n'hésitez pas à me contacter. Je ferai de mon mieux pour résoudre les bugs et aider les utilisateurs.</li>
</ul>
<h1>En conclusion: Bonne recherche !</h1>
<h2>Détails techniques</h2>
<p>Le programme a été réalisé à l'aide du module python <a href="https://anthony-tuininga.github.io/cx_Freeze/">cx_Freeze</a>. Il permet de générer des exécutables pour Windows, Linux et Mac à partir de programmes Python (2 et 3). Le fichier <a href="https://gitlab.com/jnanar/opus/blob/master/setup.py">setup.py</a> a été adapté pour incorporer les éléments nécessaires sous windows. Un dossier contenant une version statique et toutes les dépendances sont générées dans le dossier <code>build</code>. La création d'un seul exécutable compressé a été réalisée à l'aide de l'utilitaire <a href="http://www.7-zip.org/">7zip</a>.</p>
<h3>Création de l'archive</h3>
<p>Cette section décrit la création d'un exécutable pour windows à partir d'un paquet python pour les personnes intéressées.</p>
<h3>Construction du paquet</h3>
<p>La première étape consiste à construire le paquet à l'aide de cx_Freeze.[ref]Sous Windows, le dossier platform n'est pas ajouté automatiquement dans le dossier de destination. Il est ajouté au cours du build. Voir <a href="https://gitlab.com/jnanar/opus/blob/master/setup.py">setup.py</a>[/ref]
La construction du paquet est simplement réalisée avec la commande suivante:</p>
<div class="highlight"><pre><span></span><code><span class="n">python</span> <span class="n">setup</span><span class="o">.</span><span class="n">py</span> <span class="n">build</span>
</code></pre></div>
<h3>Créer un seul exécutable</h3>
<p>Le paquet généré consiste en un gros dossier contenant le programme <code>opus_win.exe</code> et toutes ses dépendances. Ça marche, mais ce n'est pas très pratique à distribuer. Les étapes suivantes consistent à utiliser 7zip pour tout placer dans un fichier compressé auto-extractible (sfx). </p>
<ol>
<li>Compresser le dossier cible à l'aide de 7zip au format 7z.</li>
<li>Générer un fichier de configuration. Personnellement, je me suis aidé de l'excellent <a href="https://sourceforge.net/projects/s-zipsfxbuilder/">7z SFX Builder</a>. Bien qu'il soit abandonné, il fonctionne toujours. </li>
<li>Générer le fichier opus_win.exe !</li>
</ol>
<p>Opus pour windows est généré à l'aide du fichier de configuration suivant:</p>
<div class="highlight"><pre><span></span><code><span class="p">;</span><span class="err">!</span><span class="nv">@Install</span><span class="err">@!</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span><span class="err">!</span>
<span class="n">RunProgram</span><span class="o">=</span><span class="ss">"opus_win.exe"</span>
<span class="p">;</span><span class="n">Config</span><span class="w"> </span><span class="k">file</span><span class="w"> </span><span class="n">generated</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="mi">7</span><span class="n">z</span><span class="w"> </span><span class="n">SFX</span><span class="w"> </span><span class="n">Builder</span><span class="w"> </span><span class="n">v2</span><span class="mf">.1</span><span class="p">.)</span>
<span class="p">;</span><span class="err">!</span><span class="nv">@InstallEnd</span><span class="err">@!</span>
<span class="mi">7</span><span class="n">zSFXBuilder_7zArchive</span><span class="o">=</span><span class="nl">C</span><span class="p">:</span><span class="err">\</span><span class="n">Users</span><span class="err">\</span><span class="n">xx</span><span class="err">\</span><span class="n">windows_setup</span><span class="err">\</span><span class="n">opus</span><span class="mf">.7</span><span class="n">z</span>
<span class="mi">7</span><span class="n">zSFXBuilder_SFXName</span><span class="o">=</span><span class="nl">C</span><span class="p">:</span><span class="err">\</span><span class="n">Users</span><span class="err">\</span><span class="n">xx</span><span class="err">\</span><span class="n">windows_setup</span><span class="err">\</span><span class="n">opus_win</span><span class="p">.</span><span class="n">exe</span>
<span class="mi">7</span><span class="n">zSFXBuilder_SFXIcon</span><span class="o">=</span><span class="nl">C</span><span class="p">:</span><span class="err">\</span><span class="n">Users</span><span class="err">\</span><span class="n">xx</span><span class="err">\</span><span class="n">icon</span><span class="err">\</span><span class="n">opus</span><span class="p">.</span><span class="n">ico</span>
<span class="mi">7</span><span class="n">zSFXBuilder_UseDefMod</span><span class="o">=</span><span class="mi">7</span><span class="n">zsd_All</span>
</code></pre></div>
<p>Il est également possible de créer un exécutable en se passant de <em>7z SFX Builder</em>. La documentation est disponible <a href="https://sevenzip.osdn.jp/chm/cmdline/switches/sfx.htm">sur le site japonnais de 7zip</a>.</p>
<h1>Liens</h1>
<ul>
<li><a href="https://agayon.be/repository/opus/opus_win_1.0.exe">Opus pour windows</a></li>
<li><a href="https://blog.agayon.be/presentation_opus.html">Présentation</a></li>
<li><a href="https://agayon.be/repository/opus/opus_start_pack.zip">Pack de démarrage</a></li>
<li><a href="https://gitlab.com/jnanar/opus/">Code source</a></li>
<li><a href="https://anthony-tuininga.github.io/cx_Freeze/">cx_Freeze</a></li>
<li><a href="http://www.7-zip.org/">7zip</a></li>
<li><a href="https://sourceforge.net/projects/s-zipsfxbuilder/">7z SFX Builder</a></li>
<li><a href="https://sevenzip.osdn.jp/chm/cmdline/switches/sfx.htm">Créer une archive SFX</a></li>
</ul>Where is Charlie (part 3)2017-08-03T20:00:00+02:002017-08-03T20:00:00+02:00Arnaudtag:blog.agayon.be,2017-08-03:/work_in_progress_3.html
<p><a href="https://blog.agayon.be/work_in_progress_1.html">Previously</a> on <a href="https://blog.agayon.be/work_in_progress_2.html">agayon.be</a>, I mentioned a new small project: a SLAM device.</p>
<p>It was finished recently and this article presents its final form.</p>
<p>After many trials, the chosen design includes an optical rotary encoder disk with 360 teeth in order to provide a theoretical resolution of one degree. The Arduino code sends the data through serial connection in a compatible way with the <a href="https://github.com/simondlevy/xvlidar">xvlidar library</a>. Therefore, the lidar module can be used with the <a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> module of Simon D. Levy. Moreover, the data are being sent through XMPP to another computer where the map can be plotted.</p>
<p><a href="https://blog.agayon.be/work_in_progress_1.html">Previously</a> on <a href="https://blog.agayon.be/work_in_progress_2.html">agayon.be</a>, I mentioned a new small project: a SLAM device.</p>
<p>It was finished recently and this article presents its final form.</p>
<p>After many trials, the chosen design includes an optical rotary encoder disk with 360 teeth in order to provide a theoretical resolution of one degree. The Arduino code sends the data through serial connection in a compatible way with the <a href="https://github.com/simondlevy/xvlidar">xvlidar library</a>. Therefore, the lidar module can be used with the <a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> module of Simon D. Levy. Moreover, the data are being sent through XMPP to another computer where the map can be plotted.</p>
<h1>Gathering data</h1>
<p>The encoder wheel displays 360 markers in order to provide a theoretical resolution of one degree [ref]The disk was generated by using the following website: <a href="http://www.bushytails.net/~randyg/encoder/encoderwheel.html">http://www.bushytails.net/~randyg/encoder/encoderwheel.html</a>.[/ref]. The Arduino nano sends the data through serial connection. A <a href="https://gitlab.com/lidar1d2/lidar1d2/blob/master/serial_acquisition.py">Python script</a> collects the data and </p>
<ul>
<li>sends it through XMPP to another device where it is analyzed.</li>
<li>save it in a CSV file on the SD card for future use.</li>
</ul>
<p>This script saves </p>
<ul>
<li>the distance, </li>
<li>time between detections (called timer), </li>
<li>the global and local counter which are the counter gathered in interrupt functions. They are triggered by the optical disk for each degree. In other words, local counter is equal to 360 after each rotation and global rotation is equal to the number of counts since the beginning of rotation.</li>
</ul>
<p><img alt="Delta T data" src="https://blog.agayon.be/images/deltaT.png" style="width: 730px; height: auto; max-width: 100%;"/></p>
<p>The figure displays the timer as a function of the counter. As you can see, the timer drastically increases periodically. These values correspond to the 300 ms stops of the disk after a full rotation. The Arduino nano detects it and sends a "EOR" message (End of Rotation) by serial which is used by the python script.</p>
<p>As the device is thought to be portable, I wanted it to be run on a Raspberry Pi 2. The lidar is wireless and almost cordless. The data is sent by Wifi, the electronic cards (Pi and micro-controllers) are powered with the help of a 5000 mAh power bank and the only cord is the 5V alimentation of the motors. </p>
<h1>Optical Rotary Encoder Disks</h1>
<p>In a first attempt, the encoder disk was printed on transparent film. Unfortunately, accurate optical detection is not easy. Depending on the rotation speed or the number of steps and the printing quality, it was not trivial to obtain the 360 bits per tour. In order to obtain more accurate data, the disk was printed on a standard paper and the white stripes were cut out. The resulting disk was glued between two layers of transparent film. Over the course of a full rotation, 962 steps of the small stepper motor are needed to match the 360 teeth (<a href="work_in_progress_1.html">remember the belt and the pulleys</a>). Every tooth is detected when the distance measurement is deactivated. When the lidar is activated, the process is slower and about 180 values are detected over a full rotation. As a result, undetected angles must be filled with null values [ref]Because null values give unaesthetic maps, a distance of 1 mm is saved for undetected angles[/ref].</p>
<p><img alt="Optical Rotary Encoder Disks" src="https://blog.agayon.be/images/360.png" style="width: 510px; height: auto; max-width: 100%;"/></p>
<h1>Drawing maps</h1>
<p>As the lidar is running, the map is drawn on another computer in a practical way with the help of another <a href="https://gitlab.com/lidar1d2/lidar1d2/blob/master/lidar1d2.py">small script</a> even if the Pi could save a PNG file on the SD card. This script can be used in two ways.</p>
<ol>
<li>The data is gathered from XMPP and the map is drawn in real time [ref]The variables for this settings are: file = True and xmpp_instance = False. An option parser will be added soon in order to make is more user-friendly[/ref]. Each data serie is saved in a csv file (one line per 360° scan).</li>
<li>The data is read from the previously written CSV file [ref]The variables for this settings are: file = False and xmpp_instance = True.[/ref].</li>
</ol>
<p>The data is managed in a compatible way with the <a href="https://github.com/simondlevy/xvlidar">xvlidar library</a>. Therefore, my lidar module is compatible with the <a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> library of Simon D. Levy.</p>
<h1>Pictures and video</h1>
<p>Here are two pictures of my living room.</p>
<p><img alt="SLAM map living room 1" src="https://blog.agayon.be/images/SLAM_living1.png" style="width: 1000px; height: auto; max-width: 100%;"/>
<img alt="SLAM map living room 2" src="https://blog.agayon.be/images/SLAM_living2.png" style="width: 1000px; height: auto; max-width: 100%;"/> </p>
<p>The following video dislays the acquisition process in a entrance hall.</p>
<video controls="controls" height="480" src="/images/lidar_encoder.mp4" width="640">A small video with the whole assembled parts</video>
<h1>Notes</h1>
<p>During the building of the device, several questions arised and the outcome was uncertain. The following list shows some answers. I let you find the questions. ;-)</p>
<ul>
<li>The slip ring works like a charm</li>
<li>The motor rotation is quite constant. I do not regret using a stepper motor instead of a simple DC motor. A 360° rotation is performed by counting steps. The rotation is paused for several milliseconds. This pause triggers the emission of data through serial connection.</li>
<li>The Arduino IDE is better and better on Linux. I had synchronization problems on Linux in the past when unplugging the micro-controller but everything is fine in the last versions.</li>
<li>BreezySLAM is working with my data even if I had to use the <em>Deterministic_SLAM</em> function. <em>RMHC_SLAM</em> tried to dynamically find anew position of the lidar even if it stayed still.</li>
<li>I can send the data from the lidar through XMPP and analyze them on another computer on a big screen. Speed is not a concern.</li>
<li>I can easily reuse my LCD from Adafruit to print useful information about the process.</li>
</ul>
<h1>Future</h1>
<p>In the following months, I will try to mount the lidar on the agayon in order to map entire rooms and not only immediate environment of a single point.</p>
<h1>Links</h1>
<ul>
<li><a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> and <a href="http://aerospacerobotics.com/blogs/learn/14881837-unraveling-breezyslam">http://aerospacerobotics.com/blogs/learn/14881837-unraveling-breezyslam</a></li>
<li><a href="https://gitlab.com/lidar1d2/lidar1d2">Gitlab repository</a></li>
<li>Arduino code of <a href="https://gitlab.com/lidar1d2/Arduino_nano">Nano</a> and <a href="https://gitlab.com/lidar1d2/Arduino_mega">Mega</a></li>
</ul>
<h1>Footnotes</h1>Where is Charlie (follow up)2017-04-01T20:00:00+02:002017-04-01T20:00:00+02:00Arnaudtag:blog.agayon.be,2017-04-01:/work_in_progress_2.html<p>A follow up to <a href="https://blog.agayon.be/work_in_progress_1.html">my previous post</a>: building a <a href="https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping">SLAM</a> device to map a room in 2 or 3 dimensions.</p><p>Previously on <a href="https://blog.agayon.be/work_in_progress_1.html">agayon.be</a>, I mentioned a new small project. I started to build a <a href="https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping">SLAM</a> device. After several tests and trials, I managed to assemble a small rotating platform. The lowest part is made of several elements:</p>
<ul>
<li>a wooden platform</li>
<li>a <a href="https://shop.mchobby.be/moteurs-pas-a-pas/767-moteur-pas-a-pas-5v-32-pas-reduction-116--3232100007673.html">5V stepper motor (BYJ48)</a></li>
<li>an electronic breadboard used for the connections</li>
<li>an <a href="https://www.arduino.cc/en/Main/arduinoBoardNano">arduino nano compatible board</a></li>
<li>a <a href="http://www.robotshop.com/eu/en/slip-ring-flange-736.html">slip ring</a></li>
</ul>
<p>The rotating part hold the <a href="http://www.robotshop.com/en/lidar-lite-3-laser-rangefinder.html">Lidar</a> and a <a href="https://shop.mchobby.be/makerbeam/85-servo-standard-tower-pro-sg5010-3232100000858.html">5V servo motor</a>.</p>
<h1>Pictures and video</h1>
<h2>The lower part of the platform: stepper motor, timing belt and pulleys</h2>
<p><img alt="The lower part of the platform: stepper motor, timing belt and pulleys" src="https://blog.agayon.be/images/lower_1.jpg" style="width: 625px; height: auto; max-width: 100%;"/></p>
<h2>On the other face, a slip ring allows the transmission of electric signal between static and rotating parts</h2>
<p><img alt="On the other face, a slip ring allows the transmission of electric signal between static and rotating parts. " src="https://blog.agayon.be/images/lower2.jpg" style="width: 482px; height: auto; max-width: 100%;"/></p>
<h2>This wood board gives a better idea of the final product</h2>
<p><img alt="This wood board gives a better idea of the final product" src="https://blog.agayon.be/images/upper_1.jpg" style="width: 750px; height: auto; max-width: 100%;"/></p>
<h2>The Lidar is able to rotate at 180°</h2>
<p><img alt="Alt Text" src="https://blog.agayon.be/images/lidar.jpg" style="width: 769px; height: auto; max-width: 100%;"/></p>
<h2>The assembled platform with the Arduino nano</h2>
<p><img alt="Alt Text" src="https://blog.agayon.be/images/platform.jpg" style="width: 702px; height: auto; max-width: 100%;"/></p>
<h2>Finally, a small video with the whole assembled parts</h2>
<p>The Lidar performs the distance measurements every 500 ms.</p>
<video controls="controls" height="480" src="/images/lidar.mp4" width="640">A small video with the whole assembled parts</video>
<p>The laser can rotate on a horizontal plane at 360° and at 180° on a vertical plane. My aim is to obtain a picture of a hemisphere. The Arduino transmits the data by batch. It is planned to interface this device with the <a href="https://github.com/simondlevy/BreezyLidar">BreezyLidar</a> and <a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> Python libraries. Therefore, it will communicate through a serial interface similarly to other popular Lidar like Hokuyo <a href="https://www.hokuyo-aut.jp/02sensor/07scanner/urg_04lx.html">URG-04LX</a> or the <a href="http://www.robotshop.com/en/rplidar-360-laser-scanner.html">RP Lidar</a>.</p>
<h1>Code</h1>
<h2>Arduino nano</h2>
<p>The Nano controls:</p>
<ul>
<li>the Lidar</li>
<li>the servo</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="cm">/* Distance measurments</span>
<span class="cm"> by Arnaud Joset <www.agayon.be></span>
<span class="cm"> This example code is released under the GPLV3 licence.</span>
<span class="cm"> This code is inspired by the work of:</span>
<span class="cm"> - Sweep by BARRAGAN <http://barraganstudio.com></span>
<span class="cm"> - LIDARLite Arduino Library GetDistancePwm (Garmin)</span>
<span class="cm"> modified 30 March 2017</span>
<span class="cm"> by Arnaud Joset</span>
<span class="cm">*/</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><Servo.h></span>
<span class="n">Servo</span><span class="w"> </span><span class="n">myservo</span><span class="p">;</span><span class="w"> </span><span class="c1">// create servo object to control a servo</span>
<span class="c1">// twelve servo objects can be created on most boards</span>
<span class="kt">int</span><span class="w"> </span><span class="n">mini</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="c1">// minimum position Servo</span>
<span class="kt">int</span><span class="w"> </span><span class="n">maxi</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">180</span><span class="p">;</span><span class="w"> </span><span class="c1">// Maximum position Servo</span>
<span class="kt">int</span><span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">90</span><span class="p">;</span><span class="w"> </span><span class="c1">// variable to store the servo position</span>
<span class="n">boolean</span><span class="w"> </span><span class="n">Servo_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">pulseWidth</span><span class="p">;</span>
<span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">currentMillis</span><span class="p">,</span><span class="w"> </span><span class="n">previousServoMillis</span><span class="p">,</span><span class="w"> </span><span class="n">previousLidarMillis</span><span class="p">;</span>
<span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">intervalServo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="n">intervalLidar</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">500</span><span class="w"> </span><span class="p">;</span>
<span class="kt">int</span><span class="w"> </span><span class="n">triggerPin</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span>
<span class="kt">int</span><span class="w"> </span><span class="n">monitorPin</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3</span><span class="p">;</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">setup</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="mi">9600</span><span class="p">);</span><span class="w"> </span><span class="c1">// Start serial communications</span>
<span class="w"> </span><span class="c1">// TRIGGER = RESISTANCE 1k OHM = YELLOW</span>
<span class="w"> </span><span class="c1">// MONITOR = NO RESISTANCE</span>
<span class="w"> </span><span class="n">pinMode</span><span class="p">(</span><span class="n">triggerPin</span><span class="p">,</span><span class="w"> </span><span class="n">OUTPUT</span><span class="p">);</span><span class="w"> </span><span class="c1">// Set pin 2 as trigger pin</span>
<span class="w"> </span><span class="n">digitalWrite</span><span class="p">(</span><span class="n">triggerPin</span><span class="p">,</span><span class="w"> </span><span class="n">LOW</span><span class="p">);</span><span class="w"> </span><span class="c1">// Set trigger LOW for continuous read</span>
<span class="w"> </span><span class="n">pinMode</span><span class="p">(</span><span class="n">monitorPin</span><span class="p">,</span><span class="w"> </span><span class="n">INPUT</span><span class="p">);</span><span class="w"> </span><span class="c1">// Set pin 3 as monitor pin</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Start"</span><span class="p">);</span>
<span class="w"> </span><span class="n">myservo</span><span class="p">.</span><span class="n">attach</span><span class="p">(</span><span class="mi">9</span><span class="p">);</span><span class="w"> </span><span class="c1">// attaches the servo on pin 9 to the servo object</span>
<span class="w"> </span><span class="c1">// pin 9 et 10 pour servo shield</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">loop</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">lidar_test</span><span class="p">();</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">lidar_test</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">currentMillis</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">millis</span><span class="p">();</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">currentMillis</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">previousServoMillis</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">intervalServo</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">previousServoMillis</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">currentMillis</span><span class="p">;</span>
<span class="w"> </span><span class="n">Servo_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">setServo</span><span class="p">(</span><span class="n">Servo_state</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">currentMillis</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">previousLidarMillis</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">intervalLidar</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">previousLidarMillis</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">currentMillis</span><span class="p">;</span>
<span class="w"> </span><span class="n">pulseWidth</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">pulseIn</span><span class="p">(</span><span class="n">monitorPin</span><span class="p">,</span><span class="w"> </span><span class="n">HIGH</span><span class="p">);</span><span class="w"> </span><span class="c1">// Count how long the pulse is high in microseconds</span>
<span class="w"> </span><span class="c1">//Serial.print("Pulsewidth ");</span>
<span class="w"> </span><span class="c1">//Serial.println(pulseWidth);</span>
<span class="w"> </span><span class="c1">// If we get a reading that isn't zero, let's print it</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pulseWidth</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">pulseWidth</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">pulseWidth</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">10</span><span class="p">;</span><span class="w"> </span><span class="c1">// 10usec = 1 cm of distance</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="s">"Distance : "</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="n">pulseWidth</span><span class="p">);</span><span class="w"> </span><span class="c1">// Print the distance</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">" cm"</span><span class="p">);</span><span class="w"> </span><span class="c1">// Print the distance</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="c1">//////////////////////////////////////////</span>
<span class="c1">// ---- SERVO ---</span>
<span class="n">boolean</span><span class="w"> </span><span class="nf">setServo</span><span class="p">(</span><span class="n">boolean</span><span class="w"> </span><span class="n">Servo_state</span><span class="p">)</span><span class="c1">// balayage servo de 20 à 180°</span>
<span class="p">{</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">increment</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">Servo_state</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Serial.println("Rising ngle");</span>
<span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">increment</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Serial.println("Falling angle");</span>
<span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="n">increment</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1">//Serial.print("Pos = ");</span>
<span class="w"> </span><span class="c1">//Serial.println(pos);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pos</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="n">mini</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Servo_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mini</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pos</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">maxi</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Servo_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">maxi</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">myservo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">pos</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">Servo_state</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<h2>The Mega with motor shield V2.3</h2>
<p>This board controls the stepper motor. </p>
<div class="highlight"><pre><span></span><code><span class="cm">/* </span>
<span class="cm">This is a test sketch for the Adafruit assembled Motor Shield for Arduino v2</span>
<span class="cm">It won't work with v1.x motor shields! Only for the v2's with built in PWM</span>
<span class="cm">control</span>
<span class="cm">For use with the Adafruit Motor Shield v2 </span>
<span class="cm">----> http://www.adafruit.com/products/1438</span>
<span class="cm">*/</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><Wire.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><Adafruit_MotorShield.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"utility/Adafruit_MS_PWMServoDriver.h"</span>
<span class="c1">// Create the motor shield object with the default I2C address</span>
<span class="n">Adafruit_MotorShield</span><span class="w"> </span><span class="n">AFMS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Adafruit_MotorShield</span><span class="p">();</span><span class="w"> </span>
<span class="c1">// Or, create it with a different I2C address (say for stacking)</span>
<span class="c1">// Adafruit_MotorShield AFMS = Adafruit_MotorShield(0x61); </span>
<span class="c1">// Connect a stepper motor with 513 steps per revolution (xx degree)</span>
<span class="c1">// to motor port #2 (M3 and M4)</span>
<span class="n">Adafruit_StepperMotor</span><span class="w"> </span><span class="o">*</span><span class="n">myMotor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AFMS</span><span class="p">.</span><span class="n">getStepper</span><span class="p">(</span><span class="mi">513</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">);</span>
<span class="cm">/*</span>
<span class="cm"> * Wiring stepper 28 BYJ-48 5V</span>
<span class="cm"> * bleue: M4 top</span>
<span class="cm"> * pink: M3 bottom</span>
<span class="cm"> * red : GND</span>
<span class="cm"> * orange: M3 top</span>
<span class="cm"> * yellow : M4 bottom</span>
<span class="cm"> * On the mega, the top and bottom refer to the position of the connection </span>
<span class="cm"> when the user can read the text on the shield.</span>
<span class="cm">*/</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">setup</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="mi">9600</span><span class="p">);</span><span class="w"> </span><span class="c1">// set up Serial library at 9600 bps</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Stepper test!"</span><span class="p">);</span>
<span class="w"> </span><span class="n">AFMS</span><span class="p">.</span><span class="n">begin</span><span class="p">();</span><span class="w"> </span><span class="c1">// create with the default frequency 1.6KHz</span>
<span class="w"> </span><span class="c1">//AFMS.begin(1000); // OR with a different frequency, say 1KHz</span>
<span class="w"> </span><span class="n">myMotor</span><span class="o">-></span><span class="n">setSpeed</span><span class="p">(</span><span class="mi">25</span><span class="p">);</span><span class="w"> </span><span class="c1">// 5 rpm </span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">loop</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">myMotor</span><span class="o">-></span><span class="n">step</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span><span class="w"> </span><span class="n">FORWARD</span><span class="p">,</span><span class="w"> </span><span class="n">DOUBLE</span><span class="p">);</span><span class="w"> </span>
<span class="w"> </span><span class="n">test</span><span class="p">();</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">test</span><span class="p">(){</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Single coil steps"</span><span class="p">);</span>
<span class="w"> </span><span class="n">myMotor</span><span class="o">-></span><span class="n">step</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span><span class="w"> </span><span class="n">FORWARD</span><span class="p">,</span><span class="w"> </span><span class="n">SINGLE</span><span class="p">);</span><span class="w"> </span>
<span class="w"> </span><span class="n">myMotor</span><span class="o">-></span><span class="n">step</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span><span class="w"> </span><span class="n">BACKWARD</span><span class="p">,</span><span class="w"> </span><span class="n">SINGLE</span><span class="p">);</span><span class="w"> </span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Double coil steps"</span><span class="p">);</span>
<span class="w"> </span><span class="n">myMotor</span><span class="o">-></span><span class="n">step</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span><span class="w"> </span><span class="n">FORWARD</span><span class="p">,</span><span class="w"> </span><span class="n">DOUBLE</span><span class="p">);</span><span class="w"> </span>
<span class="w"> </span><span class="n">myMotor</span><span class="o">-></span><span class="n">step</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span><span class="w"> </span><span class="n">BACKWARD</span><span class="p">,</span><span class="w"> </span><span class="n">DOUBLE</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<h1>Future</h1>
<p>The following steps are:</p>
<ul>
<li>Use an encoder disk to determine the direction of measurement.</li>
<li>Write the measurement procedure and the serial protocol in order to use the <a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> library.</li>
</ul>Opus, une aide à la recherche d'emploi2017-03-22T19:00:00+01:002017-03-13T21:00:00+01:00Arnaudtag:blog.agayon.be,2017-03-22:/presentation_opus.html<p>Opus est un programme d'aide à la recherche d'emploi. Il facilite l'envoi de lettres de motivation et CV générés à partir de canevas personnalisés.</p><p>Opus est un programme d'aide à la recherche d'emploi. Je l'ai codé lorsque je recherchais un boulot. Après avoir reçu retour positifs sur l'existence d'un tel programme, je l'ai amélioré afin de le rendre portable et facilement installable. Cette expérience m'a permis de m'améliorer en programmation et de toucher aux interfaces graphiques. À présent, ce programme est disponible pour le plus grand nombre. </p>
<p>Opus facilite l'envoi de lettres de motivation et CV générés à partir de canevas personnalisés. Il permet également d'envoyer des emails adaptés au destinataire. Les données sont extraites à partir d'éléments encodés préalablement dans un tableur. </p>
<p>En d'autres termes, c'est un système de publipostage ciblé pour la recherche d'emploi.</p>
<p><strong>Opus est en version alpha. L'utilisateur est responsable de son utilisation est des messages envoyés.</strong></p>
<h1 id="install">Installation</h1>
<p>Opus est un programme écrit en python. Il nécessite les dépendances suivantes (installables via pip):</p>
<ul>
<li>appdirs</li>
<li>lxml</li>
<li>xlrd</li>
<li>PyQt5</li>
</ul>
<p>Pour l'instant, le plus simple est de cloner le dépôt git après avoir installé les dépendances dans un environement virtuel. Un script d'installation est prévu prochainement.</p>
<div class="highlight"><pre><span></span><code> git clone https://gitlab.com/jnanar/opus.git
cd opus
virtualenv venv
source venv/bin/activate
pip install .
</code></pre></div>
<p>Configuration du programme (voir section <a href="#methodologie">méthodologie</a>)</p>
<div class="highlight"><pre><span></span><code> python3 opus.py
</code></pre></div>
<p>Par ailleurs, si vous voulez utiliser l'interface graphique, vous devez également installer pyQt 5.</p>
<p>Il est également possible de télécharger les sources sous forme d'archive zip depuis le <a href="https://gitlab.com/jnanar/opus/tree/master">dépôt</a>.</p>
<h1 id="methodologie">Méthodologie</h1>
<p>La procédure d'utilisation est composée de 9 étapes.</p>
<ol>
<li>Copie du fichier de configuration opus.example.ini vers opus.ini.</li>
<li>Personalisation du fichier de configuration (voir la section <a href="#config">Configuration</a>).</li>
<li>Edition du répertoire (par défaut data/list.xlsx) à l'aide d'un tableur (OpenOffice, libreOffice, MS Office, etc). (voir section <a href="#conftable">Édition du tableau</a>)</li>
<li>Personalisation des lettres de motivations. A l'heure actuelle, deux canevas sont prévus: le premier pour les candidatures spontanées et le second en cas de réponse à une offre.
Il est primordial de soigner ces lettres afin qu'elles soient uniques et personnelles. Opus permet d'adapter légèrement les lettres en fonction du destinataire, de la compagnie mais si la lettre est mauvaise, Opus, ne vous sera d'aucune utilité. (voir section <a href="#conf_letters">Édition des lettres</a>)</li>
<li>Lancement du programme opus.py. Deux interfaces sont disponibles: une textuelle et une interface graphique (voir section <a href="#interfaces">Interfaces</a>).
La première étape consiste à générer les lettres et les emails. Les canevas des documents sont spécifiés dans le fichier de configuration. Les fichiers générés sont placés dans le répertoire data/sent par défaut (option out_dir du fichier de configuration). Il est indispensable de les éditer. En effet, les modification apparaissent en gras et il est nécessaire de vérifier les accords, la tournure des phrases et la cohérence de l'ensemble.</li>
<li>L'étape suivante consiste à convertir les lettres au format PDF. </li>
<li>La dernière étape consiste à envoyer les mails. Une copie est envoyée à votre adresse en copie cachée. </li>
<li>Une fois les mails envoyés, il est nécessaire d'éditer le fichier le tableau en indiquant un commentaire dans la première colonne afin de ne pas renvoyer plusieurs fois la même candidature par inadvertance.</li>
<li>Il est conseillé de déplacer les fichiers docx, PDF et les mails dans un sous-dossier afin de garder une trace des éléments envoyés (tracabilité et éventuelles justification auprès des services administratifs).</li>
</ol>
<p>Le mieux est probablement de réaliser plusieurs tests avec votre adresse personnelle.</p>
<p>Attention, la génération des PDF n'est pas encore possible sous windows. La solution nécessite pour l'instant la présence de de l'exacutable <code>soffice</code> pour réaliser cette tâche. Un script <code>.bat</code> sera préparé prochainement. </p>
<p>D'autres possibilités existent et seront codées si une demande est formulée dans ce sens. En attendant, il est possible de générer les PDF manuellement dans votre éditeur de document préféré.</p>
<h1 id="interfaces">Interfaces</h1>
<p>Deux interfaces sont disponibles: textuelle et graphique. Elles sont toutes les deux faciles d'utilisation et l'utilisateur peut les choisir au moyen de l'option <code>-i</code> ou <code>--interface</code> du script de lancement <code>opus.py</code>. Par défaut, l'interface graphique est sélectionnée.</p>
<h1>Interface graphique (GUI)</h1>
<p>L'interface graphique est intuitive. Elle permet de réaliser toutes les tâches d'Opus mais d'également de choisir à la volée un fichier de configuration et d'enregistrer les modifications réalisées. L'état du logitiel est affiché dans une barre d'état, ce qui permet de suivre l'avancement des opérations facilement.
<img alt="Opus Gui: Cover letter generation" src="https://blog.agayon.be/images/opus/opus_gui_1.png" style="width: 407px; height: auto; max-width: 100%;"/>
<img alt="Opus Gui: Settings" src="https://blog.agayon.be/images/opus/opus_gui_2.png" style="width: 603px; height: auto; max-width: 100%;"/>
<img alt="Opus Gui: About" src="https://blog.agayon.be/images/opus/opus_gui_3.png" style="width: 490px; height: auto; max-width: 100%;"/></p>
<h1>Interface textuelle (CLI)</h1>
<p>L'interface textuelle permet d'utiliser Opus de manière linéaire et facile. Le programme pose des questions (pour l'instant uniquement en français). A la fin de chaque étape, le script demande si l'utilisateur a fait le nécessaire pour passer à l'étape suivante. Il est possible de relancer plusieurs fois Opus pour sauter certaines étapes et reprendre un travail en cours (par exemple pour convertir en PDF des lettres préalablement éditées).
<img alt="Opus Cli" src="https://blog.agayon.be/images/opus/opus-cli.png" style="width: 731px; height: auto; max-width: 100%;"/></p>
<h1 id="config">Configuration</h1>
<p>Le fichier <code>.ini</code> contient différents champs. </p>
<h2>Informations personnelles</h2>
<p>Les informations personnelles comportent:</p>
<ul>
<li>votre nom complet,</li>
<li>le sujet des mails envoyés pour une candidature spontanée,</li>
<li>le sujet des mails envoyés en réponse à une offre,</li>
<li>votre langue (pour l'instant, seul le français est supporté). </li>
</ul>
<h2>Les fichiers</h2>
<p>Les informations de la section <code>files</code> concernent les </p>
<ul>
<li>les canevas des lettres de motivation,</li>
<li>Le fichier contenant le CV,</li>
<li>les fichiers générés,</li>
<li>le répertoire utilisé (data/list.xlsx par défaut),</li>
<li>les canevas de mails</li>
<li>le dossier de destination des fichiers générés (par défaut /data/sent/)</li>
</ul>
<h2>Envoi de courrier électronique</h2>
<p>L'envoi courrier électronique nécessite l'utilisation d'un serveur smtp. Les paramètres de configuration dépendent de votre fournisseur mail (gmail, hotmail, votre fournisseur d'accès internet, etc). Les paramètres sont:</p>
<ul>
<li>l'adresse du serveur</li>
<li>le nom d'utilisateur</li>
<li>le mot de passe (stocké en clair pour l'instant)</li>
<li>le numéro de port.</li>
</ul>
<p>Si un de ces éléments ne nécessite pas d'être configuré, veuillez indiquer <code>None</code>. Voir l'exemple ci-dessous pour un serveur ne nécessitant pas d'authentification:</p>
<div class="highlight"><pre><span></span><code>server = example.com
user = None
password = None
port = 25
</code></pre></div>
<p>Les informations concernant ces paramètres sont disponibles auprès du fournisseur de votre boîte mail.</p>
<h1 id="conftable">Édition du tableau</h1>
<p>Chaque ligne du répertoire correspond à l'envoi d'une candidature. Les informations renseignées dans les différentes colonnes sont interprétées et utilisées dans la lettre de motivation et/ou le mail envoyé. Les différents champs sont:</p>
<ol>
<li><strong>sent</strong> indiquez quelque chose si la candidature a dété été envoyée. Le programme lis cette ligne uniquement si ce champs est vide. Il s'agit d'une sécurité pour éviter d'envoyer plusieurs fois le même mail mais également d'une aide pour faciliter le suivit des offres.</li>
<li><strong>company</strong>: le nom de la société.</li>
<li><strong>sector</strong>: le domaine dans lequel la société opère.</li>
<li><strong>nom</strong>: si vous adressez directement votre candidature à une personne en particulier, il s'agit de son nom.</li>
<li><strong>gender</strong>: le genre de la personne à qui vous adressez votre candidature. Au choix: "Madame"; "Monsieur"; "Madame, monsieur"</li>
<li><strong>interest</strong>: votre intérêt à postuler dans cette société.</li>
<li><strong>offer</strong>: la référence de l'offre à laquelle vous répondez. Laissez ce champs vide si vous postulez spontanément.</li>
<li><strong>subject</strong>: un complément d'information dans le mail que vous envoyez (référence d'une offre par exemple).</li>
<li><strong>email</strong>: l'adresse mail à laquelle le mail doit être envoyé.</li>
<li><strong>language</strong>: la langue de la candidature. Pour l'instant, seul le français est supporté.</li>
<li><strong>web, telephone, number, information</strong>: renseignements supplémentaires qui permettent de concentrer d'autres informations sur la société.</li>
</ol>
<h1 id="conf_letters">Edition des lettres</h1>
<p>Plusieurs éléments sont éditables.</p>
<ul>
<li><strong>TIME</strong>: la date du jour sera spécifiée</li>
<li><strong>GENDER</strong>: Monsieur; Madame ou Madame, Monsieur</li>
<li><strong>INTEREST</strong>: votre intérêt pour la société</li>
<li><strong>COMPANY</strong>: le nom de la société</li>
<li><strong>SECTOR</strong>: le domaine d'activité de la société</li>
<li><strong>OFFER</strong>: la référence de l'offre à laquelle vous répondez.</li>
</ul>
<p>Les éléments des mails sont identiques.</p>
<h1>Futur</h1>
<p>En l'état, le programme est utilisable et il répondait à mes besoins lorsque je m'en servais. A l'avenir, plusieurs pistes d'améliorations sont envisageables:</p>
<ul>
<li>Création d'un exécutable utilisable sous Windows si je reçois des demandes dans ce sens.</li>
</ul>
<p>Bonne recherche !</p>Work in progress2017-02-09T20:00:00+01:002017-02-09T20:00:00+01:00Arnaudtag:blog.agayon.be,2017-02-09:/work_in_progress_1.html<p>I recently purchased some new products. The aim is to build a mapping device for the Agayon. This subproject will be built from a <a href="http://www.robotshop.com/en/lidar-lite-3-laser-rangefinder.html">Lidar Lite v3</a>, a 5V <a href="https://shop.mchobby.be/moteurs-pas-a-pas/767-moteur-pas-a-pas-5v-32-pas-reduction-116--3232100007673.html">stepper motor</a>, <a href="http://www.robotshop.com/eu/en/mxls-01-hitec-timing-belt-pulley.html">pulleys</a> ,<a href="http://www.robotshop.com/eu/en/8-mxl-timing-belt.html">belt</a>, a <a href="https://www.arduino.cc/en/Main/arduinoBoardNano">compatible arduino nano v3</a>, <a href="http://www.robotshop.com/eu/en/slip-ring-flange-736.html">slip ring</a> and a homemade wooden chassis.</p><h1>Where is Charlie ?</h1>
<p>In robotic, SLAM stands for <a href="https://fr.wikipedia.org/wiki/Simultaneous_localization_and_mapping">Simultaneous Localization And Mapping</a>. This technique is used to
create a map of the surroundings of the robot (see <a href="https://blog.agayon.be/work_in_progress_1.html#links">examples</a>).</p>
<p>I recently purchased some new products. The aim is to build a mapping device for the Agayon. This small project will be built from </p>
<ul>
<li>a <a href="http://www.robotshop.com/en/lidar-lite-3-laser-rangefinder.html">Lidar Lite v3</a>, </li>
<li>a 5V <a href="https://shop.mchobby.be/moteurs-pas-a-pas/767-moteur-pas-a-pas-5v-32-pas-reduction-116--3232100007673.html">stepper motor</a>, </li>
<li>two pulleys (<a href="http://www.robotshop.com/eu/en/mxls-01-hitec-timing-belt-pulley.html">a small one</a>and <a href="http://www.robotshop.com/eu/en/mxl-60-tooth-timing-pulley.html">a larger one</a>), </li>
<li>a <a href="http://www.robotshop.com/eu/en/8-mxl-timing-belt.html">timing belt</a> </li>
<li>a <a href="https://www.arduino.cc/en/Main/arduinoBoardNano">compatible arduino nano v3</a>,</li>
<li>a <a href="http://www.robotshop.com/eu/en/slip-ring-flange-736.html">slip ring</a>,</li>
<li>an <a href="https://shop.mchobby.be/shields/379-shield-de-controle-moteur-motor-shield-v2-3232100003798-adafruit.html?search_query=motor+shield&results=147">Arduino motor shield V2.3</a>,</li>
<li>and a homemade wooden chassis. </li>
</ul>
<p>This module will a part of the <a href="https://www.agayon.be">Agayon</a>. It aims to map a room in 2 dimensions but 3 dimensional mapping should be possible by using a small servomotor (see this <a href="https://www.youtube.com/watch?v=gCpCGkwwy8I">example</a>).</p>
<h1>About the chassis</h1>
<p>Furthermore, four <a href="https://shop.mchobby.be/proximite/561-senseur-ultrason-hc-sr04-3232100005617.html?search_query=ultrason&results=14">ultrasonic sensors</a> have been
mounted on the Agayon. Their purpose is to prevent it to get too close of surrounding walls. This function is complementary to the SLAM system. Indeed, these sensors are closer to the ground and the lidar could not see obstacles at 15-20 cm to the ground.</p>
<h2 id="links">Links</h2>
<p>The mapping system is inspired by the following nice projects:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=as3NhsMiWXo">https://www.youtube.com/watch?v=as3NhsMiWXo</a></li>
<li><a href="https://www.youtube.com/watch?v=nIvOWfxFefc">https://www.youtube.com/watch?v=nIvOWfxFefc</a></li>
<li><a href="http://blog.trossenrobotics.com/2015/07/13/lidar-lite-360-mirror-scanner/">http://blog.trossenrobotics.com/2015/07/13/lidar-lite-360-mirror-scanner/</a></li>
<li><a href="https://hackaday.io/project/4087-360-degree-lidar-lite-scanner">https://hackaday.io/project/4087-360-degree-lidar-lite-scanner</a></li>
<li><a href="https://github.com/simondlevy/BreezySLAM">BreezySLAM</a> and <a href="http://aerospacerobotics.com/blogs/learn/14881837-unraveling-breezyslam">http://aerospacerobotics.com/blogs/learn/14881837-unraveling-breezyslam</a></li>
</ul>Atom feeds2017-01-12T19:00:00+01:002017-01-12T19:00:00+01:00Arnaudtag:blog.agayon.be,2017-01-12:/feeds.html<p>This post shows the list of ATOM feeds for the blog.</p><p>This post shows the list of ATOM feeds for the blog.</p>
<p>There is a feed for the entire activity of the blog: <a href="https://blog.agayon.be/feeds/all.atom.xml">https://blog.agayon.be/feeds/all.atom.xml</a></p>
<p>Feeds are also available per</p>
<ul>
<li>
<p>Category</p>
<ul>
<li>XMPP: <a href="https://blog.agayon.be/feeds/xmpp.atom.xml">https://blog.agayon.be/feeds/xmpp.atom.xml</a></li>
<li>Misc <a href="https://blog.agayon.be/feeds/misc.atom.xml">https://blog.agayon.be/feeds/misc.atom.xml</a></li>
</ul>
</li>
<li>
<p>Tag</p>
<ul>
<li>XMPP: <a href="https://blog.agayon.be/feeds/tag-xmpp.atom.xml">https://blog.agayon.be/feeds/tag-xmpp.atom.xml</a></li>
<li>Python: <a href="https://blog.agayon.be/feeds/tag-python.atom.xml">https://blog.agayon.be/feeds/tag-python.atom.xml</a></li>
<li>Django <a href="https://blog.agayon.be/feeds/tag-django.atom.xml">https://blog.agayon.be/feeds/tag-django.atom.xml</a></li>
</ul>
</li>
</ul>
<p>More categories and tags will be created with the writing of new articles: <a href="https://en.wikipedia.org/wiki/Do_it_yourself">DIY</a>, Robots ,
<a href="https://www.agayon.be">Agayon</a>, ...</p>
<p>Stay tuned !</p>Authentication without password using XMPP on a Django website2017-01-11T19:00:00+01:002017-02-09T19:00:00+01:00Arnaudtag:blog.agayon.be,2017-01-11:/xmpp_auth_django.html<p>This article describes the authentication with XMPP on a Django powered website. When you authenticate on a website, the domain validate your identity before letting you access confidential information. They are several ways perform this validation and the use of passwords is the most popular. Another method is the use of a token generator i.e. a small device that generate a secret passphrase that you copy on a website. Today I will present you another authentication method without password using XMPP.</p><p>This article describes the authentication with XMPP on a Django powered website.</p>
<h1>Authentication without password</h1>
<p>When you authenticate on a website, the domain validate your identity before letting you access confidential information. They are several ways perform this
validation and the use of passwords is the most popular. Another method is the use of a token generator i.e. a small device that generate a secret passphrase that you
copy on a website. Today I will present you another authentication method without password using XMPP.</p>
<h2>XMPP authentication</h2>
<p>XMPP has a nice authentication mechanism. It is normalized in the XMPP extension <a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a>. It may be used on website.
There are 4 steps.</p>
<ol>
<li>The user visits its favorite website and go to the login section.</li>
<li>The user enter its jid (XMPP address) in a form and click on a button to authenticate.</li>
<li>The website send a XMPP request to the user asking if he wants to login on the website. The request display also a code that must be identical on the website and
the XMPP client in order to validate the request.</li>
<li>The user validate the request on its XMPP client and therefore he is login on the website.</li>
</ol>
<p>There are plenty XMPP clients: <a href="https://gajim.org">Gajim</a>, <a href="https://salut-a-toi.org/">Salut-à-toi</a>, <a href="https://movim.eu/">Movim</a>, <a href="https://conversations.im/">Conversation</a>, <a href="http://poez.io">Poezio</a>, <a href="https://pidgin.im/">Pidgin</a>, <a href="http://psi-im.org/">Psi</a> etc. Several of them work on mobile, on webpage or on Desktop. Therefore, it is
possible to authenticate easily on a website using your smartphone, Desktop or another platform easily without password.</p>
<p>Note: if the client does not support the XEP-0070, there is a fallback mechanism where the user send back the validation code in a chat window. Therefore, it is possible to authenticate with all XMPP clients.</p>
<h2>Examples</h2>
<h3>Gajim</h3>
<p><img alt="Gajim XEP-0070" src="https://blog.agayon.be/images/XEP-0070-Gajim.png" style="width: 514px; height: auto; max-width: 100%;"/></p>
<h3>Salut à toi (Primitivus)</h3>
<p><img alt="Primitivus XEP-0070" src="https://blog.agayon.be/images/XEP-0070-primitivus.png.webp" style="width: 658px; height: auto; max-width: 100%;"/></p>
<p>The following section presents the implementation of this mechanism on a Django website.</p>
<h1>Use XMPP authentification mechanism with Django</h1>
<h2>Make it easy with HTTPAuthenticationOverXMPP</h2>
<p>In this section, the XMPP part is managed by a component written by "Chteufleur". This <a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">component</a>
is easy to use. It manage the XMPP session and the web developeur just have to make a request to the component and it sends a return code:</p>
<ul>
<li>200 : User accepts the request</li>
<li>400 : One or more mandatory parameter(s) is missing</li>
<li>401 : User denies the request or timeout</li>
<li>520 : Unknown error appends</li>
<li>523 : Server is unreachable</li>
</ul>
<p>The installation procedure is described in the Readme file of the project
(<a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP</a>).</p>
<h2>Django files</h2>
<p>The view manage the form fields and send the jid and validation code (<code>transaction_id</code>) to a module called <code>XmppBackend</code>. The <code>transaction_id</code> is generated when the form is accessed. Its value is kept in memory by using the session mechanism of Django (see section <a href="https://blog.agayon.be/xmpp_auth_django.html#settings.py">Settings.py</a>).</p>
<p>Several files are needed to obtained the desired result. The following sections describes them.</p>
<h3>Forms.py</h3>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">django</span> <span class="kn">import</span> <span class="n">forms</span>
<span class="k">class</span> <span class="nc">AuthForm</span><span class="p">(</span><span class="n">forms</span><span class="o">.</span><span class="n">Form</span><span class="p">):</span>
<span class="n">username</span> <span class="o">=</span> <span class="n">forms</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">help_text</span><span class="o">=</span><span class="s2">"(XMPP jid)"</span><span class="p">)</span>
</code></pre></div>
<h3>HTML template</h3>
<div class="highlight"><pre><span></span><code>{% extends "base.html" %}
{% block content %}
{% if form.errors %}
<span class="p"><</span><span class="nt">p</span><span class="p">></span>Your username is invalid. Please try again.<span class="p"></</span><span class="nt">p</span><span class="p">></span>
{% endif %}
<span class="p"><</span><span class="nt">form</span> <span class="na">method</span><span class="o">=</span><span class="s">"post"</span> <span class="na">action</span><span class="o">=</span><span class="s">"{% url 'login' %}"</span><span class="p">></span>
{% csrf_token %}
<span class="p"><</span><span class="nt">table</span><span class="p">></span>
{{form.as_p}}
<span class="p"></</span><span class="nt">table</span><span class="p">></span>
<span class="p"><</span><span class="nt">input</span> <span class="na">type</span><span class="o">=</span><span class="s">"submit"</span> <span class="na">value</span><span class="o">=</span><span class="s">"Login"</span> <span class="na">id</span><span class="o">=</span><span class="s">"Login"</span> <span class="na">name</span><span class="o">=</span><span class="s">"login"</span><span class="p">/></span>
<span class="p"></</span><span class="nt">form</span><span class="p">></span>
Your validation code: {{ transaction_id|linebreaks }}
<span class="p"><</span><span class="nt">strong</span><span class="p">></span>{{ status_msg|linebreaks }}<span class="p"></</span><span class="nt">strong</span><span class="p">></span>
{% endblock %}
</code></pre></div>
<h3>Views.py</h3>
<p><code>views.py</code> reads the content of the POST and sends the result to <code>xmpp_auth</code>. It also handles the session and the <code>transaction_id</code> generation.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">render</span>
<span class="kn">from</span> <span class="nn">django.contrib.auth</span> <span class="kn">import</span> <span class="n">login</span>
<span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">HttpResponse</span>
<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">xmpp_auth</span>
<span class="kn">from</span> <span class="nn">.forms</span> <span class="kn">import</span> <span class="n">AuthForm</span>
<span class="k">def</span> <span class="nf">index</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'index.html'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">xmpp_authentification</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">xb</span> <span class="o">=</span> <span class="n">xmpp_auth</span><span class="o">.</span><span class="n">XmppBackend</span><span class="p">()</span>
<span class="n">transaction_id</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">status_msg</span> <span class="o">=</span> <span class="s2">""</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'POST'</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">transaction_id</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'transaction_id'</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">KeyError</span><span class="p">:</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="s1">'user_logged_in'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'fail.html'</span><span class="p">)</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">AuthForm</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="p">)</span>
<span class="c1"># check whether it's valid:</span>
<span class="k">if</span> <span class="n">form</span><span class="o">.</span><span class="n">is_valid</span><span class="p">():</span>
<span class="n">username</span> <span class="o">=</span> <span class="n">form</span><span class="o">.</span><span class="n">cleaned_data</span><span class="p">[</span><span class="s1">'username'</span><span class="p">]</span>
<span class="n">user</span><span class="p">,</span> <span class="n">status_code</span> <span class="o">=</span> <span class="n">xb</span><span class="o">.</span><span class="n">authenticate</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">transaction_id</span><span class="o">=</span><span class="n">transaction_id</span><span class="p">)</span>
<span class="k">if</span> <span class="n">user</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">login</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
<span class="c1"># Redirect to a success page.</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="s1">'user_logged_in'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'success.html'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">status_code</span> <span class="o">==</span> <span class="mi">401</span><span class="p">:</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="s1">'user_logged_in'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">status_msg</span> <span class="o">=</span> <span class="s2">"User </span><span class="si">{}</span><span class="s2"> refused to authenticate."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">username</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="s1">'user_logged_in'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'fail.html'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="s1">'user_logged_in'</span><span class="p">]</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">transaction_id</span> <span class="o">=</span> <span class="n">xb</span><span class="o">.</span><span class="n">id_generator</span><span class="p">(</span><span class="mi">6</span><span class="p">)</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="s1">'transaction_id'</span><span class="p">]</span> <span class="o">=</span> <span class="n">transaction_id</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">AuthForm</span><span class="p">()</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'registration/login.html'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'form'</span><span class="p">:</span> <span class="n">form</span> <span class="p">,</span> <span class="s1">'transaction_id'</span> <span class="p">:</span> <span class="n">transaction_id</span><span class="p">,</span>
<span class="s1">'status_msg'</span><span class="p">:</span> <span class="n">status_msg</span><span class="p">})</span>
</code></pre></div>
<h3>xmpp_auth.py</h3>
<p>This module makes the following request to the component:</p>
<div class="highlight"><pre><span></span><code><span class="nf">GET</span> <span class="nn">/auth?jid=user%40host%2fresource;domain=example.net;method=POST;transaction_id=what_you_want;timeout=120</span> <span class="kr">HTTP</span><span class="o">/</span><span class="m">1.1</span>
</code></pre></div>
<p>The component send back a return code. In case of success, the system try to find the user in the database. If this user does not exist, it is created. The system described here is simple and the code must be adapted for more complex website (profile creation, additionnal data etc).</p>
<p><code>id_generator</code> is called by <code>views.py</code> and by default, it send a code made of 8 characters (both letters and digits) but it is possible to adapt easily this behavior.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">string</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="kn">import</span> <span class="n">User</span>
<span class="k">class</span> <span class="nc">XmppBackend</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Authenticate with the XMPP 00-70 XEP</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">transaction_id</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">def</span> <span class="nf">get_transaction_id</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">transaction_id</span>
<span class="k">def</span> <span class="nf">set_transaction_id</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">transaction_id</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">transaction_id</span> <span class="o">=</span> <span class="n">transaction_id</span>
<span class="k">def</span> <span class="nf">authenticate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">username</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">transaction_id</span> <span class="o">=</span> <span class="kc">None</span><span class="p">):</span>
<span class="c1"># Check the token and return a user.</span>
<span class="n">timeout</span> <span class="o">=</span> <span class="mi">300</span>
<span class="n">payload</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'jid'</span><span class="p">:</span> <span class="n">username</span><span class="p">,</span> <span class="s1">'domain'</span><span class="p">:</span> <span class="s1">'agayon.be'</span><span class="p">,</span> <span class="s1">'method'</span><span class="p">:</span> <span class="s1">'POST'</span><span class="p">,</span> <span class="s1">'timeout'</span><span class="p">:</span> <span class="n">timeout</span><span class="p">,</span>
<span class="s1">'transaction_id'</span><span class="p">:</span> <span class="n">transaction_id</span><span class="p">}</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'https://auth.agayon.be/auth'</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
<span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="n">username</span><span class="p">)</span>
<span class="k">except</span> <span class="n">User</span><span class="o">.</span><span class="n">DoesNotExist</span><span class="p">:</span>
<span class="c1"># Create a new user. There's no need to set a password</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="n">username</span><span class="p">)</span>
<span class="n">user</span><span class="o">.</span><span class="n">is_staff</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">user</span><span class="o">.</span><span class="n">is_superuser</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">user</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">return</span> <span class="n">user</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span>
<span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">401</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"User </span><span class="si">{}</span><span class="s2"> refused to authenticate"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">username</span><span class="p">),</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">None</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span>
<span class="k">return</span> <span class="kc">None</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span>
<span class="k">def</span> <span class="nf">id_generator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">chars</span><span class="o">=</span><span class="n">string</span><span class="o">.</span><span class="n">ascii_letters</span> <span class="o">+</span> <span class="n">string</span><span class="o">.</span><span class="n">digits</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">transaction_id</span> <span class="o">=</span> <span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">chars</span><span class="p">)</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">size</span><span class="p">))</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">transaction_id</span>
</code></pre></div>
<h3 id="settings.py">Settings.py</h3>
<p>The setting of the website must be adapted to your needs. In this simple example, the sessions must be enabled (it is the case by default). Our example use cached session but you can use cookies or even databases. See the <a href="https://docs.djangoproject.com/en/1.10/topics/auth/default/">excellent documentation of Django</a> for additional information.</p>
<div class="highlight"><pre><span></span><code><span class="n">LOGIN_URL</span> <span class="o">=</span> <span class="s1">'/path/to/login/'</span>
<span class="n">CACHES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'default'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'BACKEND'</span><span class="p">:</span> <span class="s1">'django.core.cache.backends.memcached.MemcachedCache'</span><span class="p">,</span>
<span class="s1">'LOCATION'</span><span class="p">:</span> <span class="s1">'unix:/tmp/memcached.sock'</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h1>Links</h1>
<ul>
<li><a href="https://xmpp.org/extensions/xep-0070.html">XEP-0070</a></li>
<li><a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">Authentication component by Chteufleur</a></li>
<li><a href="https://linuxfr.org/news/authentifiez-vous-sans-mot-de-passe-grace-a-xmpp">Description of the mechanism</a> (french)</li>
<li><a href="https://docs.djangoproject.com/en/1.10/topics/auth/default/">Django authentification</a></li>
</ul>
<h1>Credits</h1>
<ul>
<li>The image comes from the post on <a href="https://linuxfr.org/news/authentifiez-vous-sans-mot-de-passe-grace-a-xmpp">Linuxfr</a> (by Chteufleur).</li>
<li>The description of the XMPP component comes from <a href="https://git.kingpenguin.tk/chteufleur/HTTPAuthentificationOverXMPP">its repository</a> (by Chteufleur).</li>
</ul>Chatty Server2017-01-10T19:00:00+01:002017-01-13T18:00:00+01:00Arnaudtag:blog.agayon.be,2017-01-10:/chatty_server.html<p>Chatty server is a XMPP bot programmed to run on a server. It can be used as a dynamic toto list; to provide the travel time by car, foot and public transport between two adresses; to give useful information about the load of the server and to give the status of the server. The status of the account change as a function of the load and memory usage.</p><p>Chatty server is a <a href="http://xmpp.org">XMPP</a> bot programmed to run on a server. It can be used</p>
<ul>
<li>as a dynamic toto list.</li>
<li>to provide the travel time by car, foot and public transport between two adresses.</li>
<li>to give useful information about the load of the server.</li>
<li>to give the status of the server. The status of the account change depending on the load and memory usage.</li>
</ul>
<p>The following sections describes the usage of chatty_server.</p>
<h1>When it start</h1>
<p>At start, the program set a status base on the load and the memory usage.</p>
<div class="highlight"><pre><span></span><code><span class="err"></span><span class="mi">18</span><span class="err">:</span><span class="mi">40</span><span class="err">:</span><span class="mi">29</span><span class="w"> </span><span class="err"></span><span class="n">server</span><span class="err">:</span><span class="w"> </span><span class="n">Agayon</span><span class="p">.</span><span class="n">be</span><span class="w"> </span><span class="n">en</span><span class="w"> </span><span class="n">ligne</span>
<span class="err"></span><span class="mi">18</span><span class="err">:</span><span class="mi">40</span><span class="err">:</span><span class="mi">30</span><span class="w"> </span><span class="err"></span><span class="n">server</span><span class="err">:</span><span class="w"> </span><span class="n">How</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="n">doin</span><span class="s1">' ? Load: 0.12 0.05 0.05 Memory (Go): 1.08GiB/1.96GiB (35.2%)</span>
<span class="s1">18:40:30 server is now Available (How you doin'</span><span class="w"> </span><span class="vm">?</span><span class="w"> </span><span class="k">Load</span><span class="err">:</span><span class="w"> </span><span class="mf">0.12</span><span class="w"> </span><span class="mf">0.05</span><span class="w"> </span><span class="mf">0.05</span><span class="w"> </span><span class="n">Memory</span><span class="w"> </span><span class="p">(</span><span class="k">Go</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="mf">1.08</span><span class="n">GiB</span><span class="o">/</span><span class="mf">1.96</span><span class="n">GiB</span><span class="w"> </span><span class="p">(</span><span class="mf">35.2</span><span class="o">%</span><span class="p">))</span>
<span class="k">Get</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">list</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">available</span><span class="w"> </span><span class="n">commands</span>
</code></pre></div>
<p>The list of command is send whenever the user send a message to the bot.</p>
<div class="highlight"><pre><span></span><code>18:40:58 jnanar: help
18:40:58 server: Available Commands : status, reminder, travel
</code></pre></div>
<h2>Calculate a travel time</h2>
<p>Calculate a travel time is easy and fast. You need to define a Google Maps API key to use this function. You can also, define</p>
<ul>
<li>your address,</li>
<li>the summary key: it is the summary of this travel( a highway, a long road, etc),</li>
<li>the shortest distance between work and home,</li>
<li>as well as the duration of this particular travel.</li>
</ul>
<p>When the distance differs or time differs too much from these data, the bot will remind you to take another road.</p>
<div class="highlight"><pre><span></span><code><span class="mi">18</span><span class="o">:</span><span class="mi">22</span><span class="o">:</span><span class="mi">22</span><span class="w"> </span><span class="err"></span><span class="n">jnanar</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">travel</span>
<span class="err"></span><span class="mi">18</span><span class="o">:</span><span class="mi">22</span><span class="o">:</span><span class="mi">23</span><span class="w"> </span><span class="err"></span><span class="n">server</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">Usage</span><span class="o">:</span><span class="w"> </span><span class="n">travel</span><span class="w"> </span><span class="n">car</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="n">stop</span>
<span class="n">conveyance</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">car</span><span class="o">,</span><span class="w"> </span><span class="n">bus</span><span class="o">,</span><span class="w"> </span><span class="n">metro</span><span class="o">,</span><span class="w"> </span><span class="n">tramway</span><span class="w"> </span><span class="n">etc</span><span class="o">.</span>
<span class="n">Example</span><span class="o">:</span><span class="w"> </span><span class="n">travel</span><span class="w"> </span><span class="n">car</span><span class="w"> </span><span class="n">rue</span><span class="w"> </span><span class="n">du</span><span class="w"> </span><span class="n">parc</span><span class="w"> </span><span class="mi">4000</span><span class="w"> </span><span class="n">Liège</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="n">Place</span><span class="w"> </span><span class="n">saint</span><span class="w"> </span><span class="n">Lambert</span><span class="w"> </span><span class="mi">15</span><span class="w"> </span><span class="mi">4000</span><span class="w"> </span><span class="n">Liège</span>
<span class="err"></span>
</code></pre></div>
<h2>Examples</h2>
<div class="highlight"><pre><span></span><code><span class="mi">18</span><span class="o">:</span><span class="mi">42</span><span class="o">:</span><span class="mi">08</span><span class="w"> </span><span class="err"></span><span class="n">jnanar</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">travel</span><span class="w"> </span><span class="n">car</span><span class="w"> </span><span class="n">rue</span><span class="w"> </span><span class="n">du</span><span class="w"> </span><span class="n">parc</span><span class="w"> </span><span class="mi">4000</span><span class="w"> </span><span class="n">Liège</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="n">Place</span><span class="w"> </span><span class="n">saint</span><span class="w"> </span><span class="n">Lambert</span><span class="w"> </span><span class="mi">15</span><span class="w"> </span><span class="mi">4000</span><span class="w"> </span><span class="n">Liège</span>
<span class="err"></span><span class="mi">18</span><span class="o">:</span><span class="mi">42</span><span class="o">:</span><span class="mi">09</span><span class="w"> </span><span class="err"></span><span class="n">server</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">travel</span><span class="w"> </span><span class="n">car</span>
<span class="n">travel</span><span class="w"> </span><span class="n">rue</span><span class="w"> </span><span class="n">du</span><span class="w"> </span><span class="n">parc</span><span class="w"> </span><span class="mi">4000</span><span class="w"> </span><span class="n">liège</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">place</span><span class="w"> </span><span class="n">saint</span><span class="w"> </span><span class="n">lambert</span><span class="w"> </span><span class="mi">15</span><span class="w"> </span><span class="mi">4000</span><span class="w"> </span><span class="n">liège</span>
<span class="n">Summary</span><span class="o">:</span><span class="w"> </span><span class="n">Quai</span><span class="w"> </span><span class="n">Orban</span>
<span class="n">Distance</span><span class="o">:</span><span class="w"> </span><span class="mf">2.46</span><span class="w"> </span><span class="n">km</span>
<span class="n">Duration</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">traffic</span><span class="o">:</span><span class="w"> </span><span class="mf">9.22</span><span class="w"> </span><span class="n">min</span>
<span class="n">Duration</span><span class="w"> </span><span class="o">(</span><span class="n">normal</span><span class="o">):</span><span class="w"> </span><span class="mf">8.05</span><span class="w"> </span><span class="n">min</span>
</code></pre></div>
<p>In order to use the travel time function you need to define an API key, a work location and a home location in the file named "libs/credentials.py". Some shortcuts
are available: h2w and w2h for home and work location.</p>
<div class="highlight"><pre><span></span><code><span class="mi">18</span><span class="o">:</span><span class="mi">49</span><span class="o">:</span><span class="mi">10</span><span class="w"> </span><span class="err"></span><span class="n">jnanar</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">travel</span><span class="w"> </span><span class="n">h2w</span>
<span class="err"></span><span class="mi">18</span><span class="o">:</span><span class="mi">49</span><span class="o">:</span><span class="mi">11</span><span class="w"> </span><span class="err"></span><span class="n">server</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">home</span><span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">work</span><span class="w"> </span><span class="n">address</span>
<span class="n">Summary</span><span class="o">:</span><span class="w"> </span><span class="n">E42</span>
<span class="n">Distance</span><span class="o">:</span><span class="w"> </span><span class="mf">7.81</span><span class="w"> </span><span class="n">km</span>
<span class="n">Duration</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">traffic</span><span class="o">:</span><span class="w"> </span><span class="mf">11.03</span><span class="w"> </span><span class="n">min</span>
<span class="n">Duration</span><span class="w"> </span><span class="o">(</span><span class="n">normal</span><span class="o">):</span><span class="w"> </span><span class="mf">11.13</span><span class="w"> </span><span class="n">min</span>
</code></pre></div>
<h2>Obtain the status of the server</h2>
<div class="highlight"><pre><span></span><code><span class="mi">18</span><span class="o">:</span><span class="mi">34</span><span class="o">:</span><span class="mi">13</span><span class="w"> </span><span class="err"></span><span class="n">jnanar</span><span class="err"></span><span class="o">:</span><span class="w"> </span><span class="n">status</span>
<span class="err"></span><span class="mi">18</span><span class="o">:</span><span class="mi">34</span><span class="o">:</span><span class="mi">13</span><span class="w"> </span><span class="err"></span><span class="n">server</span><span class="o">:</span>
<span class="n">Load</span><span class="o">:</span><span class="w"> </span><span class="mf">2.12</span><span class="w"> </span><span class="mf">2.26</span><span class="w"> </span><span class="mf">2.31</span><span class="w"> </span><span class="n">Memory</span><span class="w"> </span><span class="o">(</span><span class="n">Go</span><span class="o">):</span><span class="w"> </span><span class="mf">3.94</span><span class="n">GiB</span><span class="o">/</span><span class="mf">11.23</span><span class="n">GiB</span><span class="w"> </span><span class="o">(</span><span class="mf">17.6</span><span class="o">%)</span>
<span class="n">CPUS</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="o">(</span><span class="s1">'14.5%'</span><span class="o">,</span><span class="w"> </span><span class="s1">'11.9%'</span><span class="o">,</span><span class="w"> </span><span class="s1">'19.0%'</span><span class="o">,</span><span class="w"> </span><span class="s1">'10.9%'</span><span class="o">,</span><span class="w"> </span><span class="s1">'35.7%'</span><span class="o">,</span><span class="w"> </span><span class="s1">'7.6%'</span><span class="o">)</span>
<span class="n">Memory</span><span class="w"> </span><span class="o">(</span><span class="n">Go</span><span class="o">):</span><span class="w"> </span><span class="mf">3.94</span><span class="n">GiB</span><span class="o">/</span><span class="mf">11.23</span><span class="n">GiB</span><span class="w"> </span><span class="o">(</span><span class="mf">17.6</span><span class="o">%)</span>
</code></pre></div>
<h2>Use it as a reminder or a TODOlist.</h2>
<div class="highlight"><pre><span></span><code><span class="mi">18</span>:<span class="mi">42</span>:<span class="mi">45</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="nv">list</span>
<span class="mi">18</span>:<span class="mi">42</span>:<span class="mi">45</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Todo</span><span class="w"> </span><span class="nv">list</span>
<span class="mi">18</span>:<span class="mi">42</span>:<span class="mi">57</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="nv">write</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">blog</span><span class="w"> </span><span class="nv">article</span><span class="w"> </span><span class="mi">5</span><span class="nv">min</span>
<span class="mi">18</span>:<span class="mi">42</span>:<span class="mi">57</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Reminder</span>:<span class="w"> </span><span class="nv">write</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">blog</span><span class="w"> </span><span class="nv">article</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">48</span>:<span class="mi">04</span>.<span class="mi">797349</span>
<span class="mi">18</span>:<span class="mi">43</span>:<span class="mi">23</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="mi">2</span><span class="nv">h</span><span class="w"> </span><span class="nv">program</span><span class="w"> </span><span class="nv">my</span><span class="w"> </span><span class="nv">personal</span><span class="w"> </span><span class="nv">robot</span>
<span class="mi">18</span>:<span class="mi">43</span>:<span class="mi">23</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Reminder</span>:<span class="w"> </span><span class="nv">program</span><span class="w"> </span><span class="nv">my</span><span class="w"> </span><span class="nv">personal</span><span class="w"> </span><span class="nv">robot</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">20</span>:<span class="mi">43</span>:<span class="mi">30</span>.<span class="mi">599113</span>
<span class="mi">18</span>:<span class="mi">43</span>:<span class="mi">29</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="nv">list</span>
<span class="mi">18</span>:<span class="mi">43</span>:<span class="mi">29</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Todo</span><span class="w"> </span><span class="nv">list</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="nv">write</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">blog</span><span class="w"> </span><span class="nv">article</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">48</span>:<span class="mi">04</span>.<span class="mi">797349</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="nv">program</span><span class="w"> </span><span class="nv">my</span><span class="w"> </span><span class="nv">personal</span><span class="w"> </span><span class="nv">robot</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">20</span>:<span class="mi">43</span>:<span class="mi">30</span>.<span class="mi">599113</span>
<span class="mi">18</span>:<span class="mi">43</span>:<span class="mi">57</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="mi">1</span><span class="nv">min</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">test</span>
<span class="mi">18</span>:<span class="mi">43</span>:<span class="mi">57</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Reminder</span>:<span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">test</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">45</span>:<span class="mi">04</span>.<span class="mi">779812</span>
<span class="mi">18</span>:<span class="mi">44</span>:<span class="mi">05</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="nv">list</span>
<span class="mi">18</span>:<span class="mi">44</span>:<span class="mi">05</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Todo</span><span class="w"> </span><span class="nv">list</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="nv">write</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">blog</span><span class="w"> </span><span class="nv">article</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">48</span>:<span class="mi">04</span>.<span class="mi">797349</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="nv">program</span><span class="w"> </span><span class="nv">my</span><span class="w"> </span><span class="nv">personal</span><span class="w"> </span><span class="nv">robot</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">20</span>:<span class="mi">43</span>:<span class="mi">30</span>.<span class="mi">599113</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">test</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">45</span>:<span class="mi">04</span>.<span class="mi">779812</span>
<span class="mi">18</span>:<span class="mi">45</span>:<span class="mi">01</span><span class="w"> </span><span class="nv">server</span>:
<span class="o">---</span>
<span class="nv">Reminder</span>:<span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">test</span>
<span class="o">---</span>
<span class="mi">18</span>:<span class="mi">45</span>:<span class="mi">10</span><span class="w"> </span><span class="nv">jnanar</span>:<span class="w"> </span><span class="nv">reminder</span><span class="w"> </span><span class="nv">list</span>
<span class="mi">18</span>:<span class="mi">45</span>:<span class="mi">10</span><span class="w"> </span><span class="nv">server</span>:<span class="w"> </span><span class="nv">Todo</span><span class="w"> </span><span class="nv">list</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="nv">write</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">blog</span><span class="w"> </span><span class="nv">article</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">48</span>:<span class="mi">04</span>.<span class="mi">797349</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">due</span><span class="w"> </span><span class="nv">program</span><span class="w"> </span><span class="nv">my</span><span class="w"> </span><span class="nv">personal</span><span class="w"> </span><span class="nv">robot</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">20</span>:<span class="mi">43</span>:<span class="mi">30</span>.<span class="mi">599113</span>
<span class="nv">todo</span>:<span class="w"> </span><span class="nv">done</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">test</span><span class="w"> </span><span class="nv">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">18</span>:<span class="mi">45</span>:<span class="mi">04</span>.<span class="mi">779812</span>
</code></pre></div>
<p>chatty_server is written in python v3 and is licensed under the GPLv3 licence.</p>
<h1>Install</h1>
<h2>Install the dependencies</h2>
<ul>
<li><a href="https://github.com/fritzy/SleekXMPP">SleekXMPP</a></li>
<li><a href="https://github.com/googlemaps/google-maps-services-python">Google Maps python API</a></li>
</ul>
<h2>Clone the repository</h2>
<ul>
<li>git clone git@gitlab.com:r1dScripts/chatty_server.git</li>
<li>Modify libs/credentials.example.py and rename it as libs/credentials.py.</li>
</ul>First Post2017-01-10T18:00:00+01:002017-01-10T18:00:00+01:00Arnaudtag:blog.agayon.be,2017-01-10:/first_post.html<p>First post of the Agayon</p><p>The blog of <a href="https://www.agayon.be">Agayon.be</a> is finally online.</p>
<p>I will post here some short articles about topics I am interested in (robotic, python, django, etc).</p>
<p>Stay tuned!</p>