Journal of a Bot - Mischttps://blog.agayon.be/2024-02-04T12:00:00+01:00AgayonTodo2024-02-04T12:00:00+01:002024-02-04T12:00:00+01:00Arnaudtag:blog.agayon.be,2024-02-04:/agayontodo.html
<p>For years, I have used <a href="https://www.mytinytodo.net/">myTinyTodo</a> to manage my to-do lists. It is super light, fast and easy to use.
At some point, I struggled to update it to use my up-to-date version of PHP. Last year, I wanted to explore other languages than Python. I decided to start a small project to replace myTinyTodo. As I wanted to explore <a href="https://go.dev/">Go</a> and <a href="https://react.dev/">React</a>, I created two projects to build my to-do list system.</p>
<p>The project fulfills my needs for now. The IU is not perfect, and it lacks a lot of functionalities of MyTinyTodo but I don't really need them for now.</p>
<h1>Development</h1>
<p>I named the project AgayonTodo. The repositories can be found here:</p>
<ul>
<li><a href="https://gitlab.com/jnanar/agayontodo_js">Javascript frontend</a></li>
<li><a href="https://gitlab.com/jnanar/agayontodo">Go backend</a></li>
</ul>
<p>The Javascript frontend relies on React and calls the API to display the data. I used the <a href="https://github.com/facebook/create-react-app/"> create-react-app</a> tool to build the whole thing and followed the main tutorial. Even if I had to download the whole internet of dependencies, it is pretty efficient.
I used the <a href="https://bulma.io/">Bulma</a> CSS Framework because I wanted to try something else than Bootstrap. It can be used without Javascript, which could be helpful in some of my other projects.</p>
<p>The Go backend service relies on <a href="https://gorm.io/index.html">gorm</a>, an ORM in Go. The service only provides a small <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD API</a> and some small tools because I don't need much at the moment. I did not want to do all the SQL requests by myself, especially if I want to add <a href="https://en.wikipedia.org/wiki/Access-control_list">access control</a> in the future. I thought it would be better to learn how to use that popular library.</p>
<p>For years, I have used <a href="https://www.mytinytodo.net/">myTinyTodo</a> to manage my to-do lists. It is super light, fast and easy to use.
At some point, I struggled to update it to use my up-to-date version of PHP. Last year, I wanted to explore other languages than Python. I decided to start a small project to replace myTinyTodo. As I wanted to explore <a href="https://go.dev/">Go</a> and <a href="https://react.dev/">React</a>, I created two projects to build my to-do list system.</p>
<p>The project fulfills my needs for now. The IU is not perfect, and it lacks a lot of functionalities of MyTinyTodo but I don't really need them for now.</p>
<h1>Development</h1>
<p>I named the project AgayonTodo. The repositories can be found here:</p>
<ul>
<li><a href="https://gitlab.com/jnanar/agayontodo_js">Javascript frontend</a></li>
<li><a href="https://gitlab.com/jnanar/agayontodo">Go backend</a></li>
</ul>
<p>The Javascript frontend relies on React and calls the API to display the data. I used the <a href="https://github.com/facebook/create-react-app/"> create-react-app</a> tool to build the whole thing and followed the main tutorial. Even if I had to download the whole internet of dependencies, it is pretty efficient.
I used the <a href="https://bulma.io/">Bulma</a> CSS Framework because I wanted to try something else than Bootstrap. It can be used without Javascript, which could be helpful in some of my other projects.</p>
<p>The Go backend service relies on <a href="https://gorm.io/index.html">gorm</a>, an ORM in Go. The service only provides a small <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD API</a> and some small tools because I don't need much at the moment. I did not want to do all the SQL requests by myself, especially if I want to add <a href="https://en.wikipedia.org/wiki/Access-control_list">access control</a> in the future. I thought it would be better to learn how to use that popular library.</p>
<h1>Production</h1>
<p>The frontend is built and copied in a nginx virtual host and the backend static executable is run with a systemd unit. The system is basically launched like that. Running the frontend and the backend in the same virtual host prevents dealing with CORS issues.</p>
<div class="highlight"><pre><span></span><code><span class="n">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">listen</span><span class="w"> </span><span class="o">*</span><span class="p">:</span><span class="mi">443</span><span class="w"> </span><span class="n">ssl</span><span class="p">;</span>
<span class="w"> </span><span class="n">listen</span><span class="w"> </span><span class="p">[::]:</span><span class="mi">443</span><span class="w"> </span><span class="n">ssl</span><span class="w"> </span><span class="n">http2</span><span class="p">;</span>
<span class="w"> </span><span class="n">server_name</span><span class="w"> </span><span class="n">example</span><span class="o">.</span><span class="n">org</span><span class="p">;</span>
<span class="w"> </span><span class="n">ssl_certificate</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">example</span><span class="o">.</span><span class="n">pem</span><span class="p">;</span>
<span class="w"> </span><span class="n">ssl_certificate_key</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">example</span><span class="o">.</span><span class="n">key</span><span class="p">;</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="n">robots</span><span class="o">.</span><span class="n">txt</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">add_header</span><span class="w"> </span><span class="n">Content</span><span class="o">-</span><span class="n">Type</span><span class="w"> </span><span class="n">text</span><span class="o">/</span><span class="n">plain</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">200</span><span class="w"> </span><span class="s2">"User-agent: *</span><span class="se">\n</span><span class="s2">Disallow: /</span><span class="se">\n</span><span class="s2">"</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">error_log</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">error</span><span class="o">.</span><span class="n">log</span><span class="p">;</span>
<span class="w"> </span><span class="n">include</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">global</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">.</span><span class="n">conf</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="o">/</span><span class="n">srv</span><span class="o">/</span><span class="n">http</span><span class="o">/</span><span class="n">example</span><span class="o">.</span><span class="n">org</span><span class="p">;</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">auth_basic</span><span class="w"> </span><span class="s2">"Authorized users only"</span><span class="p">;</span>
<span class="w"> </span><span class="n">auth_basic_user_file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">security</span><span class="o">.</span><span class="n">txt</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">/</span><span class="n">api</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">auth_basic</span><span class="w"> </span><span class="s2">"Authorized users only"</span><span class="p">;</span>
<span class="w"> </span><span class="n">auth_basic_user_file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">security</span><span class="o">.</span><span class="n">txt</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_pass</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">8080</span><span class="o">/</span><span class="n">api</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h1>Captures</h1>
<p><img alt="Capture of AgayonTodo" src="https://blog.agayon.be/images/agayontodo/capture.png.webp" style="width: 1468px; height: auto; max-width: 100%;"/>
<img alt="Capture 3 of AgayonTodo" src="https://blog.agayon.be/images/agayontodo/capture3.png.webp" style="width: 846px; height: auto; max-width: 100%;"/></p>
<h1>Links</h1>
<ul>
<li><a href="https://www.mytinytodo.net/">myTinyTodo</a></li>
</ul>2023 Summary2023-12-26T19:00:00+01:002023-12-26T19:00:00+01:00Arnaudtag:blog.agayon.be,2023-12-26:/news_2023.html
<p>First I would like to wish all the readers (if there is any), a merry Christmas and a nice happy end of year.</p>
<p>The time flew since the last article. As I was more busy with my other hobby, improv theater, I had less time to hack stuff and build new things. Nevertheless, I will summarize here what I did in the past months. </p>
<h1>My First Thinkpad T</h1>
<p>First, I bought a refurbished <a href="https://www.lenovo.com/ee/et/laptops/thinkpad/t-series/ThinkPad-T480s/p/22TP2TT480S">Levovo Thinkpad T480s</a>. It is the best machine I've had. For around €300 it is really a great purchase from my point of view. I had already purchased a second-hand computer for some relatives at <a href="https://media-monster.be/en">Media Monster</a>, and I wanted to buy one myself to be more autonomous.</p>
<p>I am not a gamer and I don't have big needs. I just upgraded the RAM to 24Go in total, which is sufficient for my needs. After checking that everything was alright with Windows 11 installed by default, I formatted everything and <a href="https://support.lenovo.com/us/en/downloads/ds502226-bios-update-utility-bootable-cd-for-windows-10-64-bit-linux-thinkpad-t480s">updated the UFI Bios</a>. Then I installed
<a href="https://archlinux.org/">Arch Linux</a> with ext4 partition in an encrypted LVM. </p>
<h1>Sway</h1>
<p>For 15 years, I have been using <a href="https://www.enlightenment.org/">Enlightenment</a> as my primary DM. I loved it, especially the ability to change from one virtual desktop to another by moving the mouse outside the screen. With the arrival of this new computer, knowing that I would have a 14' screen, I wanted to be more efficient and to use the keyboard as much as possible. Therefore, I decided to try a modern tiling window manager. In the past, I already tried the tiling mode of Enlightenment, but I did not liked it in the long run. This time I installed <a href="https://swaywm.org/">Sway</a>, a tiling WM compatible with <a href="https://i3wm.org/">i3</a>. I love it. Even if at beginning, it was a little bit difficult to remember all the shortcuts. I feel really efficient, and I like to keep my hands on the super comfortable keyboard on the Thinkpad.
I also installed <a href="https://codeberg.org/dnkl/foot">Foot</a> as my primary <a href="https://en.wikipedia.org/wiki/Terminal_emulator">terminal emulator</a>.</p>
<h1>Neovim</h1>
<p>I also took the opportunity to start to use <a href="https://neovim.io/">Neovim</a> as my primary text editor. I wanted to try it for a really long time but never took the time for it. I recently discovered the <a href="https://neovim.io/doc/user/pi_tutor.html#%3ATutor">:Tutor mode of Neovim (vimtutor)</a> and adopted it as my primary editor. For now, I use it quite basically instead of <a href="https://www.nano-editor.org/">Nano</a> but I feel it will really improve my productivity and speed in the future.</p>
<h1>Cleanup of this blog</h1>
<p>This year, I also took the time to improve the performance of this blog. With time, and the growth of article, I observed that the blog was slow to load. The main issue was the use of non-optimized pictures or videos. I decided to use the <a href="https://developer.chrome.com/docs/lighthouse/overview">Lighthouse</a> tool natively available in Chrome/Chromium and follow the diagnostic help to improve the blog. At first, I was afraid it was not possible, and I would have to move the blog to <a href="https://gohugo.io/">Hugo</a>, another static site generator. Even if Hugo is more modern, I was not happy with that idea because it could mean that the RSS feed would have been republished. As this blog pushes updates to <a href="https://planet.jabber.org/">the jabber Planet</a>, I was afraid it would spam all users with old articles. After reading the diagnostics of Lighthouse, I was able to highly improve the performances of the blog. I can keep <a href="https://getpelican.com/">Pelican</a>, my current static blog generator. I even updated the blueidea theme according to the latest changes in <a href="https://github.com/getpelican/pelican/commits/master/pelican/themes/notmyidea">notmyidea</a>, the default theme of Pelican. Two pictures are better than one big sentence.</p>
<h2>Before</h2>
<p><img alt="ligthouse before" src="https://blog.agayon.be/images/lighthouse_before.webp" style="width: 720px; height: auto; max-width: 100%;"/></p>
<h2>After</h2>
<p><img alt="lighthouse after" src="https://blog.agayon.be/images/lighthouse_after.webp" style="width: 755px; height: auto; max-width: 100%;"/></p>
<h2>Strategy</h2>
<p>In order to improve the score, the following changes were made:</p>
<ul>
<li>Convert most of the images into webp, reduce the size and resolution of the biggest ones</li>
<li>Update the iframe settings of some embedded youtube video</li>
<li>Decrease the number of article per page</li>
<li>Improve some links to avoid "read more", "go" etc. generic descriptions</li>
<li>Activate <a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP2</a> on the blog virtualhost</li>
</ul>
Écouter des Podcasts MP3 en voiture2023-04-13T15:00:00+02:002023-04-13T15:00:00+02:00Arnaudtag:blog.agayon.be,2023-04-13:/rss_podcasts.html
<p>Pendant mes trajets en voiture, j'aime écouter des podcasts. Ma voiture supporte la connection Bluetooth avec un téléphone mais pour diverses raisons, je préfère utiliser une bonne vieille clé USB.
Je récupère les épisodes à l'aide de l'application <a href="https://apps.gnome.org/app/org.gnome.Podcasts/">GNOME Podcast</a> qui récupère les flux RSS/Atom.</p>
<p>Les scripts ci-dessous me permettent d'écouter <a href="https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451">La semaine des 5 heures</a> pendant des heures, étant donné qu'ils sont passés à une formule quotidienne. J'écoute également d'excellents podcasts de France Inter: <a href="https://www.radiofrance.fr/franceinter/podcasts/blockbusters-le-podcast">Blockbusters</a> et <a href="https://www.radiofrance.fr/franceculture/podcasts/la-science-cqfd">La Science, CQFD</a> sur France Culture.</p>
<h1>RTBF AUVIO</h1>
<p>Il y a quelques mois, le site <a href="https://auvio.rtbf.be/">Auvio</a> de la RTBF a changé et les liens des podcasts n'étaient plus autant mis en avant. Je suis nénmoins tombé sur ce <a href="https://libreantenne.radioactu.com/topic/40041-rtbf-flux-rss-des-podcast-introuvables/">forum</a> qui m'a aidé à trouver la solution:</p>
<blockquote>
<p>Le nom du programme en toutes lettres est remplacé par des chiffres, identiques au code d'Auvio, la plateforme de Replay/Podcast de la RTBF.</p>
</blockquote>
<p>Ainsi, la semaine de 5h, l'émission cinéma dont l'identifiant <code>1451</code> voit ses addresses évoluer comme ceci: </p>
<ul>
<li><strong>AUVIO</strong> https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451 </li>
<li><strong>Podcast</strong> http://rss.rtbf.be/media/rss/audio/1451.xml</li>
</ul>
<h1>Scripts de conversion et renommage</h1>
<p>Par ailleurs, ma voiture ne sait pas lire les fichiers <code>m2a</code> récupérés par l'application. Je les convertis à l'aide de <a href="https://ffmpeg.org/">FFMPEG</a> afin de les lire. J'utilise les scripts ci-dessous pour lire le tout.</p>
<p>Le premier converti les fichiers en mp3 et le second les renommes en fonction de la date de l'épisode.</p>
<p>Pendant mes trajets en voiture, j'aime écouter des podcasts. Ma voiture supporte la connection Bluetooth avec un téléphone mais pour diverses raisons, je préfère utiliser une bonne vieille clé USB.
Je récupère les épisodes à l'aide de l'application <a href="https://apps.gnome.org/app/org.gnome.Podcasts/">GNOME Podcast</a> qui récupère les flux RSS/Atom.</p>
<p>Les scripts ci-dessous me permettent d'écouter <a href="https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451">La semaine des 5 heures</a> pendant des heures, étant donné qu'ils sont passés à une formule quotidienne. J'écoute également d'excellents podcasts de France Inter: <a href="https://www.radiofrance.fr/franceinter/podcasts/blockbusters-le-podcast">Blockbusters</a> et <a href="https://www.radiofrance.fr/franceculture/podcasts/la-science-cqfd">La Science, CQFD</a> sur France Culture.</p>
<h1>RTBF AUVIO</h1>
<p>Il y a quelques mois, le site <a href="https://auvio.rtbf.be/">Auvio</a> de la RTBF a changé et les liens des podcasts n'étaient plus autant mis en avant. Je suis nénmoins tombé sur ce <a href="https://libreantenne.radioactu.com/topic/40041-rtbf-flux-rss-des-podcast-introuvables/">forum</a> qui m'a aidé à trouver la solution:</p>
<blockquote>
<p>Le nom du programme en toutes lettres est remplacé par des chiffres, identiques au code d'Auvio, la plateforme de Replay/Podcast de la RTBF.</p>
</blockquote>
<p>Ainsi, la semaine de 5h, l'émission cinéma dont l'identifiant <code>1451</code> voit ses addresses évoluer comme ceci: </p>
<ul>
<li><strong>AUVIO</strong> https://auvio.rtbf.be/emission/la-semaine-des-5-heures-1451 </li>
<li><strong>Podcast</strong> http://rss.rtbf.be/media/rss/audio/1451.xml</li>
</ul>
<h1>Scripts de conversion et renommage</h1>
<p>Par ailleurs, ma voiture ne sait pas lire les fichiers <code>m2a</code> récupérés par l'application. Je les convertis à l'aide de <a href="https://ffmpeg.org/">FFMPEG</a> afin de les lire. J'utilise les scripts ci-dessous pour lire le tout.</p>
<p>Le premier converti les fichiers en mp3 et le second les renommes en fonction de la date de l'épisode.</p>
<h2>Conversion en mp3</h2>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">EXT</span><span class="o">=</span><span class="s2">".m2a"</span>
<span class="nv">FILES</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>/*<span class="s2">"</span><span class="nv">$EXT</span><span class="s2">"</span>
<span class="nv">DIRECTORY</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
/usr/bin/python3<span class="w"> </span>podcast_rename.py<span class="w"> </span><span class="s2">"</span><span class="nv">$DIRECTORY</span><span class="s2">"</span>
<span class="k">for</span><span class="w"> </span>filename<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span>find<span class="w"> </span><span class="s2">"</span><span class="nv">$DIRECTORY</span><span class="s2">"</span><span class="w"> </span>-name<span class="w"> </span><span class="s2">"*</span><span class="nv">$EXT</span><span class="s2">"</span><span class="w"> </span>-type<span class="w"> </span>f<span class="sb">`</span><span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span>
<span class="w"> </span>ffmpeg<span class="w"> </span>-i<span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span><span class="w"> </span>-acodec<span class="w"> </span>mp3<span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span>.mp3
<span class="w"> </span>rm<span class="w"> </span><span class="s2">"</span><span class="nv">$filename</span><span class="s2">"</span>
<span class="k">done</span>
<span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<h2>Renommage des fichiers mp3</h2>
<p>Les fichiers téléchargés suivent une séquence propre au programme GNOME Podcast mais il est parfois intéressant de savoir à quelle date correspond ce fichier. C'est particulièrement utile pour suivre les sorties cinéma.</p>
<p>Pour cela, j'analyse les metadata des fichiers m2a et j'extrait la date à partir du titre ou du tag <code>date</code>. </p>
<p>Le programme suivant </p>
<ul>
<li>recherche les fichiers <code>m2a</code> dans un dossier donné</li>
<li>renomme le fichier en se basant sur la date présente dans les métadata (titre ou champs date).</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env python</span>
<span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">import</span> <span class="nn">ffmpeg</span>
<span class="kn">import</span> <span class="nn">glob</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">_get_date</span><span class="p">(</span><span class="n">input_str</span><span class="p">,</span> <span class="n">datetime_format</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Return the datetime object from the string containing the date of the media file</span>
<span class="sd"> :param input_str: string to analyze</span>
<span class="sd"> :param: regex_date: regex format of the date</span>
<span class="sd"> :return: datetime object</span>
<span class="sd"> """</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'(\d+(-|\/)\d+(-|\/)\d+)'</span><span class="p">,</span> <span class="n">input_str</span><span class="p">)</span>
<span class="n">date_str</span> <span class="o">=</span> <span class="n">match</span> <span class="ow">and</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="n">date_str</span> <span class="ow">and</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">date_str</span><span class="p">,</span> <span class="n">datetime_format</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">_rename_file</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">date_obj</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Rename media file based on datetime object</span>
<span class="sd"> :param media_file: input filename</span>
<span class="sd"> :param date_obj: datetime of the media file</span>
<span class="sd"> :return:</span>
<span class="sd"> """</span>
<span class="n">vals</span> <span class="o">=</span> <span class="n">media_file</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">if</span> <span class="n">vals</span> <span class="ow">and</span> <span class="n">date_obj</span><span class="p">:</span>
<span class="n">dst_name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">vals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">date_obj</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s1">'</span><span class="si">%d</span><span class="s1">-%m-%Y'</span><span class="p">)</span><span class="si">}</span><span class="s2">.m2a"</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Trying to rename </span><span class="si">{</span><span class="n">media_file</span><span class="si">}</span><span class="s2"> --> </span><span class="si">{</span><span class="n">dst_name</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">rename</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">dst_name</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">rename_on_title_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Rename a m2a file based on the date present in the title. Example:</span>
<span class="sd"> La Semaine des 5 Heures - TITLE OF THE DAY - 03/04/2023</span>
<span class="sd"> :param media_file: fullpath of the m2a filename</span>
<span class="sd"> :param metadata: metadata dictionary of the file</span>
<span class="sd"> :return: True if the file could be renamed. Otherwise False or None</span>
<span class="sd"> """</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'format'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tags'</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">tags</span> <span class="ow">and</span> <span class="n">tags</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'title'</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">title</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">date_obj</span> <span class="o">=</span> <span class="n">_get_date</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="s2">"</span><span class="si">%d</span><span class="s2">/%m/%Y"</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">_rename_file</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">date_obj</span><span class="p">)</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">rename_on_record_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Rename a m2a file based on the date present in the record date metadata.</span>
<span class="sd"> Example: 2023-03-23</span>
<span class="sd"> :param media_file: fullpath of the m2a filename</span>
<span class="sd"> :param metadata: metadata dictionary of the file</span>
<span class="sd"> :return: True if the file could be renamed. Otherwise False</span>
<span class="sd"> """</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'format'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tags'</span><span class="p">)</span>
<span class="n">date_str</span> <span class="o">=</span> <span class="n">tags</span> <span class="ow">and</span> <span class="n">tags</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'date'</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">date_str</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">date_obj</span> <span class="o">=</span> <span class="n">_get_date</span><span class="p">(</span><span class="n">date_str</span><span class="p">,</span> <span class="s2">"%Y-%m-</span><span class="si">%d</span><span class="s2">"</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">_rename_file</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">date_obj</span><span class="p">)</span>
<span class="k">return</span> <span class="n">res</span>
<span class="k">def</span> <span class="nf">launch</span><span class="p">(</span><span class="n">dir_name</span><span class="p">):</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">dir_name</span><span class="p">,</span> <span class="s1">'*.m2a'</span><span class="p">)</span>
<span class="n">files</span> <span class="o">=</span> <span class="n">glob</span><span class="o">.</span><span class="n">glob</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">for</span> <span class="n">media_file</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
<span class="n">metadata</span> <span class="o">=</span> <span class="n">ffmpeg</span><span class="o">.</span><span class="n">probe</span><span class="p">(</span><span class="n">media_file</span><span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">rename_on_title_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">res</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">rename_on_record_date</span><span class="p">(</span><span class="n">media_file</span><span class="p">,</span> <span class="n">metadata</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">res</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The file could not be renamed </span><span class="si">{</span><span class="n">media_file</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">UserWarning</span><span class="p">(</span><span class="s2">"You need to provide a directory name"</span><span class="p">)</span>
<span class="n">directory</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">launch</span><span class="p">(</span><span class="n">directory</span><span class="p">)</span>
</code></pre></div>Fail2Ban analysis2018-12-05T19:00:00+01:002018-12-05T21:00:00+01:00Arnaudtag:blog.agayon.be,2018-12-05:/fail2ban.html
<p>Running a server on the internet is quite an adventure but it's not always easy to keep track of log files, security and potential threads. This article describes some actions carried out on this server to mitigate <a href="https://en.wikipedia.org/wiki/Script_kiddie">script kiddies</a>.</p>
<p>Running a server on the internet is quite an adventure but it's not always easy to keep track of log files, security and potential threads. This article describes some actions carried out on this server to mitigate <a href="https://en.wikipedia.org/wiki/Script_kiddie">script kiddies</a>.</p>
<p>Among the good practices, I have </p>
<ul>
<li>a subscription to the Debian security mailing list, </li>
<li>automatic security updates (enabled by default with the Scaleway Debian images)</li>
<li>A monitoring tool (<a href="http://munin-monitoring.org/">Munin</a>) which provides useful graphs to watch the activity of the server.</li>
<li><a href="https://www.fail2ban.org/wiki/index.php/Main_Page">Fail2Ban</a></li>
<li>Backups</li>
</ul>
<p>The following paragraphs describe how I analyze the country IP banned by Fail2Ban.</p>
<p>As explained on their website, Fail2Ban scans log files and bans IPs that show the malicious signs: too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, mail, ssh, etc). </p>
<p>Recently, I had the need to check if Belgian IP were blacklisted. Most of my users are Belgian and one of my Fail2Ban rules was too strict. I decided to log the IP in a file to perform a geolocalisation analysis to detect and prevent false positives.</p>
<p>The <a href="https://github.com/bcambl/fail2ban-blacklist">fail2ban-blacklist</a> script was used to log blacklisted IP into a CSV file. The analysis is performed on another computer.</p>
<h1>Scripts</h1>
<p>The following script is called <code>generate_report.py</code>. It read a <code>CSV</code> file that has several information about bans: the date, time and IP. The country IP are discovered with the whois information thanks to a script. Finally, a barplot is generated to visualize the amount of hits per country. The whole process is launched with the <code>report.sh</code> script.</p>
<h2>generate_report.py</h2>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">matplotlib</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span>
<span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">,</span>
<span class="nb">format</span><span class="o">=</span><span class="s2">"[</span><span class="si">%(asctime)s</span><span class="s2">] </span><span class="si">%(levelname)s</span><span class="s2"> </span><span class="si">%(message)s</span><span class="s2">"</span><span class="p">,</span>
<span class="n">datefmt</span><span class="o">=</span><span class="s2">"%H:%M:%S"</span><span class="p">,</span>
<span class="n">stream</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_country</span><span class="p">(</span><span class="n">ip</span><span class="p">):</span>
<span class="n">whois</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s1">'whois'</span><span class="p">,</span> <span class="n">ip</span><span class="p">],</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
<span class="n">whois</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="c1"># Prevent problems if output is not utf8</span>
<span class="n">str_whois</span> <span class="o">=</span> <span class="n">whois</span><span class="o">.</span><span class="n">communicate</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
<span class="n">find_country</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'country:(.*)'</span><span class="p">,</span> <span class="n">str_whois</span><span class="p">)</span>
<span class="k">if</span> <span class="n">find_country</span><span class="p">:</span>
<span class="k">return</span> <span class="n">find_country</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">"NONE"</span>
<span class="k">def</span> <span class="nf">plot_graph</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="p">)</span>
<span class="n">df</span><span class="o">.</span><span class="n">country</span><span class="o">.</span><span class="n">value_counts</span><span class="p">()</span><span class="o">.</span><span class="n">plot</span><span class="o">.</span><span class="n">bar</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">'Country'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">'Counts'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">16</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s1">'Counts of blacklisted countries'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="s2">"fail2ban_report.png"</span><span class="p">,</span> <span class="n">dpi</span><span class="o">=</span><span class="mi">150</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s1">'w'</span><span class="p">,</span> <span class="n">edgecolor</span><span class="o">=</span><span class="s1">'w'</span><span class="p">,</span>
<span class="n">orientation</span><span class="o">=</span><span class="s1">'portrait'</span><span class="p">,</span> <span class="n">papertype</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">transparent</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">bbox_inches</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">pad_inches</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span>
<span class="n">frameon</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">report</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'blacklist.pkl'</span>
<span class="n">df_backup</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">df_csv</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">df</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">df_concat</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
<span class="n">df_backup</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_pickle</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="sa">r</span><span class="s1">'blacklist.csv'</span>
<span class="n">df_csv</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"UTF-8"</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span> <span class="n">engine</span><span class="o">=</span><span class="s1">'python'</span><span class="p">,</span> <span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">df_backup</span><span class="p">)</span> <span class="o"><</span> <span class="nb">len</span><span class="p">(</span><span class="n">df_csv</span><span class="p">):</span>
<span class="n">df_tmp</span> <span class="o">=</span> <span class="n">df_backup</span><span class="p">[[</span><span class="s1">'DATE'</span><span class="p">,</span> <span class="s1">'TIME'</span><span class="p">,</span> <span class="s1">'IP_ADDRESS'</span><span class="p">]]</span>
<span class="n">df_concat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">concat</span><span class="p">([</span><span class="n">df_csv</span><span class="p">,</span> <span class="n">df_tmp</span><span class="p">])</span><span class="o">.</span><span class="n">drop_duplicates</span><span class="p">(</span><span class="n">keep</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">df_concat</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">""</span>
<span class="c1"># df = df_backup.append(df_concat,sort=False)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Append </span><span class="si">{}</span><span class="s2"> lines"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">df_concat</span><span class="p">)))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No difference between backup and CSV"</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df_backup</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">friend_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1"># df_friends = pd.DataFrame(columns=list(df_backup.columns.values))</span>
<span class="n">df_friends</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_pickle</span><span class="p">(</span><span class="s2">"blacklist_friends.pkl"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">df_concat</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">for</span> <span class="n">idx</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">df_concat</span><span class="o">.</span><span class="n">iterrows</span><span class="p">():</span>
<span class="c1"># Do not process the dataframe multiples times.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">row</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Process IP : </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s1">'IP_ADDRESS'</span><span class="p">])))</span>
<span class="n">country</span> <span class="o">=</span> <span class="n">get_country</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s1">'IP_ADDRESS'</span><span class="p">]))</span>
<span class="n">df_concat</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">idx</span><span class="p">,</span> <span class="s1">'country'</span><span class="p">]</span> <span class="o">=</span> <span class="n">country</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">country</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]</span>
<span class="c1"># country == 'Be' do not work</span>
<span class="n">country</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">country</span><span class="p">)</span>
<span class="k">if</span> <span class="n">country</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'be'</span><span class="p">,</span> <span class="s1">'other_friendly_country'</span><span class="p">]:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="n">row</span><span class="p">[</span><span class="s1">'country'</span><span class="p">]</span> <span class="o">=</span> <span class="n">country</span>
<span class="n">friend_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df_backup</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">df_concat</span><span class="p">,</span> <span class="n">sort</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Append friends"</span><span class="p">)</span>
<span class="n">df_friends</span> <span class="o">=</span> <span class="n">df_friends</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">friend_list</span><span class="p">)</span>
<span class="n">df_friends</span><span class="o">.</span><span class="n">to_pickle</span><span class="p">(</span><span class="s2">"./blacklist_friends.pkl"</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">IndexError</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No friend to append"</span><span class="p">)</span>
<span class="n">df</span><span class="o">.</span><span class="n">to_pickle</span><span class="p">(</span><span class="s2">"./blacklist.pkl"</span><span class="p">)</span>
<span class="n">plot_graph</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">report</span><span class="p">()</span>
</code></pre></div>
<h2>report.sh</h2>
<p>This script copy the Fail2Ban CSV file from the server (whois requests are forbidden on my VPS), generate the data and display the bar plot with the help of <code>typop</code>, a built-in function of <a href="https://www.enlightenment.org/about-terminology">terminology</a>, a great terminal emulator for Linux/BSD/UNIX systems.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/sh</span>
scp<span class="w"> </span>user@agayon.be:/etc/fail2ban/report.py<span class="w"> </span>.
python<span class="w"> </span>generate_report.py
typop<span class="w"> </span>fail2ban_report.png
<span class="nb">exit</span><span class="w"> </span><span class="m">0</span>
</code></pre></div>
<h2>Graphs</h2>
<p>This graph was generated on 5 of december 2018. Some countries are more represented but the threat is global.</p>
<p><img alt="Fail2Ban report graph" src="https://blog.agayon.be/images/fail2ban_report.png" style="width: 2400px; height: auto; max-width: 100%;"/></p>
<p>The following graphs is generated by Munin. It display the number of ban per jail.</p>
<p><img alt="Fail2Ban graph week" src="https://blog.agayon.be/images/fail2ban-week.png" style="width: 897px; height: auto; max-width: 100%;"/></p>
<h1>Links</h1>
<ul>
<li><a href="https://www.fail2ban.org/wiki/index.php/Main_Page">Fail2Ban</a></li>
<li><a href="https://github.com/bcambl/fail2ban-blacklist">bcambl/fail2ban-blacklist</a></li>
<li><a href="http://munin-monitoring.org/">Munin</a></li>
</ul>Upgrading to Odoo 122018-11-08T11:00:00+01:002019-02-23T12:00:00+01:00Arnaudtag:blog.agayon.be,2018-11-08:/odoo_12.html
<p><a href="https://www.odoo.com/fr_FR/blog/notre-blog-5/post/odoo-12-a-mature-business-management-software-515">Odoo 12.0</a> is out since october. I am currently investigating the differences with previous versions to update the instance of the association <a href="http://www.compagnonsducep.be/">Les Compagnons du CEP</a>. A lot of changes have been made in a few years but the workflow stays about the same. This article describes my workflow, the backup policy, how a module was used and fixed to restore a missing feature. Finally, the changes in my custom product import function are presented. </p>
<p><a href="https://www.odoo.com/fr_FR/blog/notre-blog-5/post/odoo-12-a-mature-business-management-software-515">Odoo 12.0</a> is out since october. I am currently investigating the differences with previous versions to update the instance of the association <a href="http://www.compagnonsducep.be/">Les Compagnons du CEP</a>. A lot of changes have been made in a few years but the workflow stays about the same. This article describes my workflow, the backup policy, how a module was used and fixed to restore a missing feature. Finally, the changes in my custom product import function are presented. </p>
<h1>Production and debug setup</h1>
<p>Agayon.be instance of Odoo runs inside a Docker container on a small VPS. Postgresql is installed as a core package. This setup is great in production but it is difficult to debug some python code with this configuration. The first step is to run Odoo with an IDE (I personally use the excellent <a href="https://www.jetbrains.com/pycharm/">Pycharm</a>). </p>
<h2>Running and debugging Odoo</h2>
<p>The first step is the creation of the virtualenv. Once the <a href="https://wiki.archlinux.org/index.php/Odoo#Configuring_PostgreSQL_to_run_with_Odoo">Postgresql instance is ready</a>, you can build the environment.</p>
<div class="highlight"><pre><span></span><code>git clone https://www.github.com/odoo/odoo --depth 1 --branch 12.0
$ virtualenv myenv
$ source myenv/bin/activate
(myenv)$ pip install -r odoo/requirements.txt
(myenv)$ mkdir custom-addons
(myenv)$ chown odoo: odoo11-custom-addons
</code></pre></div>
<p>Edit odoo.conf and then run odoo:</p>
<div class="highlight"><pre><span></span><code>odoo/odoo-bin -c odoo.conf
</code></pre></div>
<h1>Merge purchase order</h1>
<p><strong>UPDATE: 23/02/2019: The following paragraph is not needed anymore. Odoo 12 restored the automatic merge of purchase orders.</strong> </p>
<p>Unfortunately, a major feature has disappeared in version 12.0: automatic merge of purchase order. I don't know why since it seems unrealistic to send each quotation separately to your vendor. Fortunately a <a href="https://github.com/odooaktiv/MergePurchaseOrder/">free module</a> can be used to perform the merge but it has a critical bug. Some quotation lines are merged even <a href="https://github.com/odooaktiv/MergePurchaseOrder/issues/2">if they concerns different products</a>. </p>
<h2>Fix</h2>
<p>After <a href="https://github.com/jarobase/MergePurchaseOrder">forking</a> it, I decided to start by refactoring it. The module is quite small but a lot of code is duplicated. As I try to avoid <a href="https://en.wikipedia.org/wiki/Spaghetti_code">spaghetti code</a>, it needed to be <a href="https://github.com/jarobase/MergePurchaseOrder/commit/66f70a19255a66ee659fac429315276dc7991080">refactored</a>.I think the new code may be improved but no line is duplicated. Finally, the bug has been <a href="https://github.com/jarobase/MergePurchaseOrder/commit/fe653be0f693eb319852944f52e6330bfc33c4cf">fixed</a>.</p>
<h1>Backup management</h1>
<p>Since a few version, the <code>filestore</code> is mandatory in Odoo. If it is incoherent with the database, some really <a href="https://www.odoo.com/fr_FR/forum/aide-1/question/solved-how-to-recover-deleted-attachment-files-from-filestore-folder-local-share-odoo-filestore-128453">annoying errors are raised</a> and the solution is quite tedious. My backup procedure has been updated to avoid losing any data. It is based on the article from <a href="https://zeroheure.info/how-to-restore-an-odoo-backup-quick-help/">zeroheure</a>. The backups are saved with the <a href="https://www.odoo.com/apps/modules/12.0/auto_backup/">auto_backup</a> module. Restoring the data is not possible with the web interface because the process reaches the memory limit but it can be performed with the following shell script.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">BACKUPLOCATION</span><span class="o">=</span><span class="s2">"/path/to/backup.zip"</span>
<span class="nv">DBNAME</span><span class="o">=</span><span class="s2">"db_name"</span>
<span class="nv">FILESTORE_DIR</span><span class="o">=</span><span class="s2">"/path/to/filestore"</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$FILESTORE_DIR</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$DBNAME</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="s2">"</span><span class="nv">$BACKUPLOCATION</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"verify your variables"</span>
<span class="k">else</span>
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span><span class="nv">$BACKUPLOCATION</span>
<span class="w"> </span>rm<span class="w"> </span>filestore<span class="w"> </span>
<span class="w"> </span>unzip<span class="w"> </span>-q<span class="w"> </span><span class="nv">$DBNAME</span>.zip
<span class="w"> </span>cp<span class="w"> </span>-r<span class="w"> </span>filestore<span class="w"> </span><span class="nv">$DBNAME</span>
<span class="w"> </span>sudo<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="nv">$FILESTORE_DIR</span>/<span class="nv">$DBNAME</span>
<span class="w"> </span>sudo<span class="w"> </span>mv<span class="w"> </span><span class="nv">$DBNAME</span><span class="w"> </span><span class="nv">$FILESTORE_DIR</span>
<span class="w"> </span>sudo<span class="w"> </span>chown<span class="w"> </span>-R<span class="w"> </span>odoo:odoo<span class="w"> </span><span class="nv">$FILESTORE_DIR</span>/<span class="nv">$DBNAME</span>
<span class="w"> </span>dropdb<span class="w"> </span>-U<span class="w"> </span>odoo<span class="w"> </span><span class="nv">$DBNAME</span>
<span class="w"> </span>createdb<span class="w"> </span>-U<span class="w"> </span>odoo<span class="w"> </span><span class="nv">$DBNAME</span>
<span class="w"> </span>psql<span class="w"> </span><span class="nv">$DBNAME</span><span class="w"> </span>--quiet<span class="w"> </span><<span class="w"> </span>dump.sql
<span class="k">fi</span>
</code></pre></div>
<h1>Wine import with Django website</h1>
<p>Version 12.0 needs some minor changes in the code displayed in the <a href="https://blog.agayon.be/pandas_odoo.html">previous article</a>.
These modifications includes:</p>
<ul>
<li>removing state property in the product template.</li>
<li>Adding the invoice_policy and purchase_method in the product template.[ref]See <a href="https://www.odoo.com/fr_FR/forum/aide-1/question/access-sale-config-settings-value-from-sale-order-117728">this post</a> and this <a href="https://github.com/camptocamp/odoo-dj/issues/101">issue</a> [/ref]</li>
<li>Add a reference to the standard price in the product_supplierinfo dictionary. This value is used in the orders when purchasing wines to suppliers.</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">):</span>
<span class="c1"># 1) search if default code is used?)</span>
<span class="c1"># 2) search if suppliers is in the database</span>
<span class="c1"># 3) search if the name is already used</span>
<span class="c1"># Retrieve the dataframes. This example comes from a jupyter nootebook.</span>
<span class="c1"># df_product and df_sellers are already defined.</span>
<span class="c1"># In a real case, we should use class variables.</span>
<span class="n">n_code</span><span class="p">,</span> <span class="n">df_code</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">default_code</span><span class="p">,</span>
<span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_supplier</span><span class="p">,</span> <span class="n">df_suppliers</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_suppliers</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'function'</span><span class="p">,</span>
<span class="n">search_item</span><span class="o">=</span><span class="n">supplier_code</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_name</span><span class="p">,</span> <span class="n">df_name</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'name'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">wine_name</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># ids_product = list(df_code['id'])</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">df_code</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_supplier</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'success'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Another product uses the same name with another code</span>
<span class="k">return</span> <span class="s1">'e_code_used_same_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># the code is already used</span>
<span class="k">return</span> <span class="s1">'e_code_used'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_supplier</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Cannot find the supplier</span>
<span class="k">return</span> <span class="s1">'e_missing_seller'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># A product with the same name and another code exists.</span>
<span class="k">return</span> <span class="s1">'e_code_used_different_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">def</span> <span class="nf">import2odoo</span><span class="p">():</span>
<span class="n">route_warehouse0_mto</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">route_warehouse0_manufacture</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># iterate over all rows, read the cells and assign the wine parameters to variables</span>
<span class="c1"># each row correspond to one wine</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">seller_name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">do_import</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="n">standard_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span>
<span class="n">list_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span>
<span class="n">seller_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span>
<span class="n">res_search</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span> <span class="o">=</span> <span class="n">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">seller_code</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res_search</span> <span class="o">==</span> <span class="s1">'success'</span> <span class="ow">and</span> <span class="n">do_import</span> <span class="o">==</span> <span class="s2">"1"</span><span class="p">:</span>
<span class="n">product_template</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'active'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'standard_price'</span><span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="s1">'list_price'</span><span class="p">:</span> <span class="n">list_price</span><span class="p">,</span>
<span class="s1">'description'</span><span class="p">:</span> <span class="n">comment</span> <span class="p">,</span>
<span class="s1">'default_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span>
<span class="s1">'purchase_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'sale_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_po_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'product'</span><span class="p">,</span>
<span class="s1">'cost_method'</span><span class="p">:</span> <span class="s1">'standard'</span><span class="p">,</span>
<span class="s1">'invoice_policy'</span> <span class="p">:</span> <span class="s1">'order'</span> <span class="p">,</span> <span class="c1"># ordered quantities</span>
<span class="s1">'purchase_method'</span> <span class="p">:</span> <span class="s1">'receive'</span><span class="p">,</span> <span class="c1"># control received quantities (or ordered ones, test to delivery)</span>
<span class="s1">'route_ids'</span><span class="p">:</span> <span class="p">[(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="n">route_warehouse0_mto</span><span class="p">,</span> <span class="n">route_warehouse0_manufacture</span><span class="p">])]</span>
<span class="p">}</span>
<span class="c1"># For each wine, a template must be created</span>
<span class="n">template_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.template'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span>
<span class="n">product_template</span><span class="p">)</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'product_code'</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="c1"># code for supplier</span>
<span class="s1">'product_name'</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">15</span><span class="p">],</span> <span class="c1"># name for supplier</span>
<span class="s1">'min_qty'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'delay'</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="s1">'price'</span> <span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span>
<span class="s1">'product.supplierinfo'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span> <span class="n">product_supplierinfo</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Wine </span><span class="si">{}</span><span class="s2"> : </span><span class="si">{}</span><span class="s2"> has been added"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.</span>
</code></pre></div>
<h1>Conclusions</h1>
<p>After using Odoo in production for 4 years, I still love it. My users are happy to use it daily and the improvements over the year are impressive. Like any other alive project, some changes may break the workflow but Odoo and it's community make it relatively simple to adapt with plugins. Odoo 12 is strong and I look forward to use it for the next 3 years :-).</p>
<h1>Links</h1>
<ul>
<li><a href="https://www.odoo.com/">Odoo</a></li>
<li><a href="https://zeroheure.info/how-to-restore-an-odoo-backup-quick-help/">Backup management</a></li>
<li><a href="https://zeroheure.info/how-to-restore-an-odoo-backup-quick-help/">zeroheure backup quick help</a></li>
</ul>3615 MyLife2018-07-19T10:20:00+02:002018-07-19T10:21:00+02:00Arnaudtag:blog.agayon.be,2018-07-19:/algoo.html
<p>This year, the holidays and city trip was the occasion to meet Damien Accorsi, founder of <a href="https://www.algoo.fr/">Algoo SAS</a> and his team.</p>
<p>This summer, I had the opportunity to meet Damien Accorsi in Moirans near Grenoble. He is the founder of <a href="https://www.algoo.fr/">Algoo SAS</a>, a company that provides software development services and Tracim. <a href="https://github.com/tracim/tracim">Tracim</a> is <em>a collaborative software designed to allow people to share and work on various data and document types</em>. </p>
<p>Everything started from a post on <a href="https://linuxfr.org/forums/general-hors-sujets/posts/vacances-region-de-grenoble">LinuxFR</a>. I stayed near Grenoble for a week in the beginning of July and therefore, I asked the community about nice activities to do in the region. Damien answered quite quickly and made some useful suggestions. We never talked before. He also suggested meeting in his startup in Moirans. I happily accepted and we have met in his quarter. We talked about his activities. If you speak French, I suggest his instructive <a href="https://linuxfr.org/users/lebouquetin">posts in LinuxFR</a>. We talked about our projects, we laugh and had a really good time. His employees are really nice and fun. When I left, I promised myself to make more IRL meetings in the future.</p>
<p>Yet It was not the first time I visited a software developer. Two years ago, I have met Goffi from the <a href="https://salut-a-toi.org/">Salut-à-Toi</a> (SàT) project in Prague (Czech Republic). We had nice conversations and exchange about XMPP, the link between communication tools and politics, the struggle of developers to take part in open source project during free time, building a community with limited resources, etc. From this exchange started a nice collaboration on his tool. To this day, I write the PKGBUILD (packages) of SàT for <a href="https://aur.archlinux.org/packages/?K=jnanar&SeB=m">Archlinux</a>. </p>
<p>I hope to meet other people during holidays and events like FOSDEM. It is nice to put a face on a nickname. Maybe next time it will be an inventor or an artist.</p>
<p>In the meantime, if you go to Grenoble, according to Damien and myself, you should try:</p>
<ul>
<li><a href="http://www.vertige38.com/spip.php?rubrique3">bungee jumping</a></li>
</ul>
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;" allowfullscreen="" fetchpriority="high" frameborder="0" height="315" src="https://www.youtube.com/embed/B8vpDMUac5Y?si=NgS1rlIpx7386eEW" title="YouTube video player" width="560"></iframe>
<ul>
<li><a href="https://www.hikideas.com/walk-himalayan-footbridges-of-monteynard/">hiking on the Himalayan footbridges of Monteynard</a>
<img alt="Monteynard" src="https://blog.agayon.be/images/lac_monteynard.JPG" style="width: 560px; height: auto; max-width: 100%;"/></li>
<li>Chill at the "Grand lac de Laffrey".
<img alt="Laffrey" src="https://blog.agayon.be/images/lac_laffrey.JPG" style="width: 560px; height: auto; max-width: 100%;"/></li>
<li>Watch Belgium beats Brazil at Football. \o/</li>
</ul>
Mixing Pandas with Odoo2018-03-07T21:00:00+01:002018-03-13T19:00:00+01:00Arnaudtag:blog.agayon.be,2018-03-07:/pandas_odoo.html
<p>This article describes the use of XML-RPC API provided by <a href="https://www.odoo.com/">Odoo</a>, a well-known ERP system. Upgrading to version 11.0 is the occasion to update my python scripts to reduce considerably the number of requests. The improvements were done with the help of <a href="https://pandas.pydata.org/">pandas</a>, the famous data structures and data analysis library.</p>
<p><a href="https://www.flickr.com/photos/ken_from_md/4697952954/"><img alt="Photo credit: Panda_3956, Ken_from_MD" src="https://blog.agayon.be/images/panda0.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a> <br/>
Photo credit: Panda_3956, <a href="https://www.flickr.com/photos/ken_from_md/">Ken_from_MD</a> </p>
<p>This article describes the use of XML-RPC API provided by <a href="https://www.odoo.com/">Odoo</a>, a well-known ERP system. Upgrading to version 11.0 is the occasion to update my python scripts to reduce considerably the number of requests. The improvements were done with the help of <a href="https://pandas.pydata.org/">pandas</a>, the famous data structures and data analysis library.</p>
<p><a href="https://www.flickr.com/photos/ken_from_md/4697952954/"><img alt="Photo credit: Panda_3956, Ken_from_MD" src="https://blog.agayon.be/images/panda0.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a> <br/>
Photo credit: Panda_3956, <a href="https://www.flickr.com/photos/ken_from_md/">Ken_from_MD</a> </p>
<h1>Introduction</h1>
<p>In my spare time, I help a small association, <a href="http://www.compagnonsducep.be/">Les Compagnons du CEP</a>, a joint buying organization who buys French wines for its members directly from producers. Since 2014, I set up an <a href="https://www.odoo.com/">Odoo</a> instance to manage the quotations, purchase order and the members. Odoo fulfills all their needs and we are happy to use it daily.</p>
<p>In addition to the user-friendly Web interface, the following management tasks are performed within a custom <a href="https://www.djangoproject.com/">Django</a> website.</p>
<ul>
<li>Calculate the price of wines based on the seller price, taxes, transport cost, VAT, <a href="https://www.fostplus.be/en">Fost+</a>. </li>
<li>Import wines to database. The list of sellable wines is updated two times a year in order to adapt to the seasonal dishes[ref]The covered regions are: Champagne, Alsace, Loire, Bourgogne, Beaujolais - Maconnais, Rhône, Provence, Languedoc [/ref]. Of course, the vintages are changed each year depending of the wines. The association sells approximately 600 different wines (up to 10000 bottles a year). As a result, I wrote a massive import script. This program reads large Excel files containing the several parameters (name, vintage, seller name, description, etc) and it uses the XML-RPC API offered by Odoo to create the items in the PostgreSQL database. This task is the main subject if this article.</li>
<li>Generate the price list based on the newly added wines. A price list Excel file is uploaded by the user. The file is transferred on another machine with the help of <a href="https://blog.agayon.be/errol.html">Errol</a>. A <a href="https://www.latex-project.org/">LaTeX</a> document is generated with python from the Excel file and it is compiled to PDF with <a href="https://www.tug.org/texlive/">TeX Live</a>. Afterward, the PDF is automatically copied on the Django website with Errol.</li>
</ul>
<p>The Django website is successfully used since 4 years. Unfortunately, I observed slowdown in the process since the implementation of product update from the Excel file.</p>
<p>In order to prepare the upgrade to <a href="https://www.odoo.com/fr_FR/blog/notre-blog-5/post/introducing-odoo-11-455">Odoo 11.0</a>, I decided to update the XML-RPC calls in order to reduce their number and therefore accelerate the import process.</p>
<h1>Some code to get your teeth into</h1>
<p><a href=""><img alt="Photo credit: Panda, Sue Cantan https://www.flickr.com/photos/suecan/4349221370/" src="https://blog.agayon.be/images/panda3.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a><br/>
Photo credit: Panda, <a href="https://www.flickr.com/photos/suecan/">Sue Cantan</a></p>
<p>To chose the more suitable strategy, I decided to compare the current code base to a new scenario.</p>
<h2>Description of the current code</h2>
<p>Each row of the Excel file correspond to a wine. The name, seller code[ref]The seller code is saved in the field 'function'[/ref] and default code are searched in the Odoo database in order to avoid duplicates. If no collision is found with the older products, a product template is created and the supplier information are updated. Since 2014, version 8.0 is used in production with <a href="https://www.odoo.com/documentation/user/9.0/inventory/settings/products/variants.html">product variant</a> support. Unfortunately, the variant requires to perform an additional search to update the price of the product. In version 11.0, we will get rid of the product variant. </p>
<p>The python code can be summarized as follows.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">xmlrpc.client</span> <span class="k">as</span> <span class="nn">xmlrpclib</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="n">username</span> <span class="o">=</span> <span class="s1">'user'</span>
<span class="n">pwd</span> <span class="o">=</span> <span class="s1">'password'</span>
<span class="n">dbname</span><span class="o">=</span> <span class="s1">'database_name'</span>
<span class="n">sock_common</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span><span class="s1">'http://localhost:8069/xmlrpc/common'</span><span class="p">)</span>
<span class="n">uid</span> <span class="o">=</span> <span class="n">sock_common</span><span class="o">.</span><span class="n">login</span><span class="p">(</span><span class="n">database</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">pwd</span><span class="p">)</span>
<span class="n">sock</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span><span class="s1">'http://localhost:8069/xmlrpc/object'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">):</span>
<span class="n">supplier_code</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">supplier_code</span><span class="p">))</span>
<span class="n">args_p</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">default_code</span><span class="p">)]</span>
<span class="n">args_s</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'function'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">)]</span>
<span class="n">args_n</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">)]</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">n_supplier</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">n_name</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span> <span class="n">args_p</span><span class="p">)</span>
<span class="c1"># Rather than retrieve a possibly gigantic list of records and count them, search_count()</span>
<span class="c1"># can be used to retrieve only the number of records matching the query.</span>
<span class="c1"># It takes the same domain filter as search() and no other parameter.</span>
<span class="c1"># https://www.odoo.com/documentation/8.0/api_integration.html</span>
<span class="n">n_supplier</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'res.partner'</span><span class="p">,</span> <span class="s1">'search_count'</span><span class="p">,</span> <span class="n">args_s</span><span class="p">)</span>
<span class="n">n_name</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search_count'</span><span class="p">,</span> <span class="n">args_n</span><span class="p">)</span>
<span class="n">n_supplier</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_supplier</span><span class="p">)</span>
<span class="n">n_name</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_name</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="n">start_session</span><span class="p">()</span>
<span class="k">return</span> <span class="s2">"e_initialization"</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids_product</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_supplier</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'success'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids_product</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Le code est déjà utilisé et un produit du même nom existe.</span>
<span class="k">return</span> <span class="s1">'e_code_used_same_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids_product</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># Le code est déjà utilisé</span>
<span class="k">return</span> <span class="s1">'e_code_used'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="c1"># if len(ids_product) == 0:</span>
<span class="k">if</span> <span class="n">n_supplier</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># Le fournisseur n existe pas</span>
<span class="k">return</span> <span class="s1">'e_missing_seller'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># un produit du même nom mais pas le même code existe</span>
<span class="k">return</span> <span class="s1">'e_code_used_different_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">def</span> <span class="nf">import2odoo</span><span class="p">():</span>
<span class="n">route_warehouse0_mto</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">route_warehouse0_manufacture</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># iterate over all rows, read the cells and assign the wine parameters to variables</span>
<span class="c1"># each row correspond to one wine</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">seller_name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">do_import</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="n">standard_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span>
<span class="n">list_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span>
<span class="n">seller_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span>
<span class="n">res_search</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">ids_supplier</span><span class="p">,</span> <span class="n">ids_name</span> <span class="o">=</span> <span class="n">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">seller_code</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res_search</span> <span class="o">==</span> <span class="s1">'success'</span> <span class="ow">and</span> <span class="n">do_import</span> <span class="o">==</span> <span class="s2">"1"</span><span class="p">:</span>
<span class="c1"># Success, the wine may be added</span>
<span class="n">results_dict</span><span class="p">[</span><span class="s1">'added_wines'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">default_code</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">standard_price</span><span class="p">,</span> <span class="n">list_price</span><span class="p">,</span> <span class="n">comment</span><span class="p">])</span>
<span class="n">product_template</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'active'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'state'</span><span class="p">:</span> <span class="s1">'sellable'</span><span class="p">,</span>
<span class="s1">'standard_price'</span><span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="s1">'list_price'</span><span class="p">:</span> <span class="n">list_price</span><span class="p">,</span>
<span class="s1">'description'</span><span class="p">:</span> <span class="n">comment</span> <span class="p">,</span>
<span class="s1">'purchase_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'sale_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_po_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'product'</span><span class="p">,</span>
<span class="s1">'cost_method'</span><span class="p">:</span> <span class="s1">'standard'</span><span class="p">,</span>
<span class="s1">'route_ids'</span><span class="p">:</span> <span class="p">[(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="n">route_warehouse0_mto</span><span class="p">,</span> <span class="n">route_warehouse0_manufacture</span><span class="p">])]</span>
<span class="p">}</span>
<span class="c1"># For each wine, a template must be created</span>
<span class="n">template_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.template'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span>
<span class="n">product_template</span><span class="p">)</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'product_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span> <span class="c1"># code chez le fournisseur</span>
<span class="s1">'product_name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="c1"># name chez le producteur</span>
<span class="s1">'min_qty'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'delay'</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span>
<span class="s1">'product.supplierinfo'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span> <span class="n">product_supplierinfo</span><span class="p">)</span>
<span class="c1"># The product id must be obtained to set the default code</span>
<span class="n">product_product</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="s1">'default_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'='</span><span class="p">,</span> <span class="n">name</span><span class="p">),</span> <span class="p">]</span>
<span class="n">product_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span>
<span class="n">args</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'write'</span><span class="p">,</span>
<span class="n">product_id</span><span class="p">,</span>
<span class="n">product_product</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.</span>
</code></pre></div>
<h3>Metrics</h3>
<p>If <code>N</code> is the number of wines to add, <code>x</code>, the number of milliseconds to research an item and <code>y</code> the number of milliseconds to write an item into the database, the previous code is made of:</p>
<ul>
<li>3 researches per wine : 3<code>N</code> * <code>x</code></li>
<li>templates creation: <code>N</code> <code>y</code> </li>
<li>supplierinfo creation: <code>N</code> <code>y</code></li>
<li>research products: <code>N</code> <code>x</code></li>
<li>product update: <code>N</code> <code>y</code></li>
</ul>
<p><strong>Total: <code>4N x + 3N y</code></strong></p>
<p>This code is not optimized. It runs slowly even if it achieves what we expect from it. Now, we can try to do enhance it.</p>
<h2>The new version</h2>
<p>As previously mentioned, the product variant suppression will remove one search but there is more room for improvements.</p>
<p>The API call for each row is the main bottleneck of the script. I decided to replace them by a search of all product at the beginning of the script. The list if wine is return in a list of dictionary. Fortunately, a list of dicts is the easiest object to convert to a Pandas Dataframe.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="k">def</span> <span class="nf">dataframes_generator</span><span class="p">():</span>
<span class="c1"># Use a search function with empty args to get all ids :</span>
<span class="c1"># https://www.odoo.com/fr_FR/forum/aide-1/question/is-it-possible-to-retrieve-2-fields-of-all-entry-within-a-model-thru-xml-rpc-6886</span>
<span class="c1"># 1st => search all ids 2nd => read the selected fields on the whole list of ids.</span>
<span class="c1"># Only 2 rpc-xml requests :o)</span>
<span class="c1">#1</span>
<span class="n">args_product</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="s1">'!='</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">)]</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span> <span class="n">args_product</span><span class="p">)</span>
<span class="c1"># 2</span>
<span class="n">fields_products</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">,</span> <span class="s1">'default_code'</span><span class="p">]</span>
<span class="n">recordset_products</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.product'</span><span class="p">,</span> <span class="s1">'read'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">fields_products</span><span class="p">)</span>
<span class="n">args_seller</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'!='</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">)]</span>
<span class="n">fields_seller</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">,</span> <span class="s1">'function'</span><span class="p">]</span>
<span class="n">ids_sellers</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'res.partner'</span><span class="p">,</span> <span class="s1">'search'</span><span class="p">,</span> <span class="n">args_seller</span><span class="p">)</span>
<span class="n">recordset_sellers</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'res.partner'</span><span class="p">,</span> <span class="s1">'read'</span><span class="p">,</span> <span class="n">ids_sellers</span><span class="p">,</span>
<span class="n">fields_seller</span><span class="p">)</span>
<span class="n">df_product</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">recordset_products</span><span class="p">)</span>
<span class="n">df_sellers</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">recordset_sellers</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">True</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">ids_sellers</span>
</code></pre></div>
<p>For now, the ids list is generated by searching all product with a dummy code ('foo' in the example). My trials to get rid of the <code>args_product</code> variable failed.</p>
<p>As the dataframes are generated, it is easy to search the product and sellers among their respective dataframes:</p>
<div class="highlight"><pre><span></span><code><span class="n">code_to_find</span> <span class="o">=</span> <span class="s1">'A18'</span>
<span class="n">df_product</span><span class="p">[</span><span class="n">df_product</span><span class="p">[</span><span class="s1">'default_code'</span><span class="p">]</span><span class="o">.</span><span class="n">str</span><span class="o">.</span><span class="n">contains</span><span class="p">(</span><span class="n">code_to_find</span><span class="p">)]</span>
<span class="n">default_code</span> <span class="nb">id</span> <span class="n">name</span>
<span class="mi">0</span> <span class="n">A18</span><span class="o">/</span><span class="mi">999</span> <span class="mi">21</span> <span class="n">First</span> <span class="n">wine</span>
<span class="mi">1</span> <span class="n">A18</span><span class="o">/</span><span class="mi">999</span> <span class="mi">22</span> <span class="n">Second</span> <span class="n">wine</span>
</code></pre></div>
<p>A small function <code>search_string_df</code> can be written to facilitate the future searches in dataframes. It returns the dataframe and its lenght:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="c1"># if we search an int.</span>
<span class="k">if</span> <span class="n">search_int</span><span class="p">:</span>
<span class="n">df_res</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">df</span><span class="p">[</span><span class="n">col_name</span><span class="p">]</span> <span class="o">==</span> <span class="n">search_item</span><span class="p">]</span>
<span class="c1"># if we search a string. We must ignore NaN values.</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">df_res</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="n">df</span><span class="p">[</span><span class="n">col_name</span><span class="p">]</span><span class="o">.</span><span class="n">str</span><span class="o">.</span><span class="n">contains</span><span class="p">(</span><span class="n">search_item</span><span class="p">,</span> <span class="n">na</span><span class="o">=</span><span class="kc">False</span><span class="p">)]</span>
<span class="n">n_res</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">df_res</span><span class="p">)</span>
<span class="k">return</span> <span class="n">n_res</span><span class="p">,</span> <span class="n">df_res</span>
<span class="n">n</span><span class="p">,</span> <span class="n">df</span> <span class="o">=</span> <span class="n">search_string_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="n">search_str</span><span class="o">=</span><span class="s1">'A18'</span><span class="p">)</span>
</code></pre></div>
<h3>Final cut</h3>
<p>When the research function is modified, the whole code can also be adapted:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">supplier_code</span><span class="p">,</span> <span class="n">wine_name</span><span class="p">):</span>
<span class="c1"># 1) search if default code is used?)</span>
<span class="c1"># 2) search if suppliers is in the database</span>
<span class="c1"># 3) search if the name is already used</span>
<span class="c1"># Retrieve the dataframes. This example comes from a jupyter nootebook.</span>
<span class="c1"># df_product and df_sellers are already defined.</span>
<span class="c1"># In a real case, we should use class variables.</span>
<span class="n">n_code</span><span class="p">,</span> <span class="n">df_code</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'default_code'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">default_code</span><span class="p">,</span>
<span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_supplier</span><span class="p">,</span> <span class="n">df_suppliers</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_suppliers</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'function'</span><span class="p">,</span>
<span class="n">search_item</span><span class="o">=</span><span class="n">supplier_code</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">n_name</span><span class="p">,</span> <span class="n">df_name</span> <span class="o">=</span> <span class="n">search_df</span><span class="p">(</span><span class="n">df</span><span class="o">=</span><span class="n">df_product</span><span class="p">,</span> <span class="n">col_name</span><span class="o">=</span><span class="s1">'name'</span><span class="p">,</span> <span class="n">search_item</span><span class="o">=</span><span class="n">wine_name</span><span class="p">,</span> <span class="n">search_int</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># ids_product = list(df_code['id'])</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="n">df_code</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="n">ids_product</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_supplier</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'success'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Another product uses the same name with another code</span>
<span class="k">return</span> <span class="s1">'e_code_used_same_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_code</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># the code is already used</span>
<span class="k">return</span> <span class="s1">'e_code_used'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_supplier</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># Cannot find the supplier</span>
<span class="k">return</span> <span class="s1">'e_missing_seller'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">if</span> <span class="n">n_name</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># A product with the same name and another code exists.</span>
<span class="k">return</span> <span class="s1">'e_code_used_different_name'</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span>
<span class="k">def</span> <span class="nf">import2odoo</span><span class="p">():</span>
<span class="n">route_warehouse0_mto</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">route_warehouse0_manufacture</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># iterate over all rows, read the cells and assign the wine parameters to variables</span>
<span class="c1"># each row correspond to one wine</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">seller_name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">do_import</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="n">default_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="n">standard_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span>
<span class="n">list_price</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span>
<span class="n">seller_code</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span>
<span class="n">res_search</span><span class="p">,</span> <span class="n">ids_product</span><span class="p">,</span> <span class="n">n_supplier</span><span class="p">,</span> <span class="n">n_name</span> <span class="o">=</span> <span class="n">research</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">seller_code</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res_search</span> <span class="o">==</span> <span class="s1">'success'</span> <span class="ow">and</span> <span class="n">do_import</span> <span class="o">==</span> <span class="s2">"1"</span><span class="p">:</span>
<span class="n">product_template</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'active'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="s1">'standard_price'</span><span class="p">:</span> <span class="n">standard_price</span><span class="p">,</span>
<span class="s1">'list_price'</span><span class="p">:</span> <span class="n">list_price</span><span class="p">,</span>
<span class="s1">'description'</span><span class="p">:</span> <span class="n">comment</span> <span class="p">,</span>
<span class="s1">'default_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span>
<span class="s1">'purchase_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'sale_ok'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'uom_po_id'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'type'</span><span class="p">:</span> <span class="s1">'product'</span><span class="p">,</span>
<span class="s1">'cost_method'</span><span class="p">:</span> <span class="s1">'standard'</span><span class="p">,</span>
<span class="s1">'route_ids'</span><span class="p">:</span> <span class="p">[(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[</span><span class="n">route_warehouse0_mto</span><span class="p">,</span> <span class="n">route_warehouse0_manufacture</span><span class="p">])]</span>
<span class="p">}</span>
<span class="c1"># For each wine, a template must be created</span>
<span class="n">template_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span> <span class="s1">'product.template'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span>
<span class="n">product_template</span><span class="p">)</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s1">'product_code'</span><span class="p">:</span> <span class="n">default_code</span><span class="p">,</span> <span class="c1"># code chez le fournisseur</span>
<span class="s1">'product_name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="c1"># name chez le producteur</span>
<span class="s1">'min_qty'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s1">'delay'</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="s1">'product_tmpl_id'</span><span class="p">:</span> <span class="n">template_id</span><span class="p">,</span>
<span class="p">}</span>
<span class="c1"># Create the supplier information for the wine</span>
<span class="n">product_supplierinfo_id</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">dbname</span><span class="p">,</span> <span class="n">uid</span><span class="p">,</span> <span class="n">pwd</span><span class="p">,</span>
<span class="s1">'product.supplierinfo'</span><span class="p">,</span>
<span class="s1">'create'</span><span class="p">,</span> <span class="n">product_supplierinfo</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Wine </span><span class="si">{}</span><span class="s2"> : </span><span class="si">{}</span><span class="s2"> has been added"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">default_code</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="c1"># Here we take into account the exceptions and several cases: the wine is already present, the seller is missing etc.</span>
</code></pre></div>
<h3>Metrics</h3>
<p>If we follow the same conventions than before:</p>
<p>The new code is made of:</p>
<ul>
<li>3 researches: 3 * <code>x</code></li>
<li>templates creation: <code>N</code> <code>y</code> </li>
<li>supplierinfo creation: <code>N</code> <code>y</code></li>
</ul>
<p><strong>Total: <code>3 x + 2N y</code></strong></p>
<p>This result is a huge improvement. The data search in Dataframe is really fast and therefore, the impact of the bottleneck is decreased.</p>
<h1>Conclusions</h1>
<p>After several tests, the new implementation seems solid. It is not as fast as I would have expected but I am working on it. Some helpful <a href="https://fr.slideshare.net/openobject/performance2014-35689113">resources</a> are available on the web.</p>
<p>The import campaign of spring is already finished but the new algorithm will be tested with the aim to be ready for the wines coming in autumn. Nevertheless, reducing the number of request from 3N to 3 where N is the number of wines can only have beneficial effects. </p>
<p>The next step will be to watch and analyse the <a href="https://www.odoo.com/documentation/11.0/reference/cmdline.html#logging">databases requests</a>. The option <code>log-level=debug_rpc</code> will probably be crucial.</p>
<p>The results of these investigations will be shared here.</p>
<p>Stay tuned !</p>
<p><a href="https://www.flickr.com/photos/gzlu/7708872342/"><img alt="Photo credit: Panda in China, George Lu https://www.flickr.com/photos/gzlu/7708872342/" src="https://blog.agayon.be/images/panda1.jpg" style="width: 640px; height: auto; max-width: 100%;"/></a><br/>
Photo credit: Panda in China, <a href="https://www.flickr.com/photos/gzlu/">George Lu</a> </p>
<h1>Links</h1>
<ul>
<li><a href="https://www.odoo.com/">Odoo</a></li>
<li><a href="https://pandas.pydata.org/">Pandas</a></li>
<li><a href="https://www.djangoproject.com/">Django</a></li>
<li><a href="https://fr.slideshare.net/openobject/performance2014-35689113">Improving the performance of Odoo deployments</a></li>
<li><a href="https://www.odoo.com/documentation/11.0/reference/cmdline.html">Command-line interface: odoo-bin</a></li>
</ul>Atom feeds2017-01-12T19:00:00+01:002017-01-12T19:00:00+01:00Arnaudtag:blog.agayon.be,2017-01-12:/feeds.html<p>This post shows the list of ATOM feeds for the blog.</p><p>This post shows the list of ATOM feeds for the blog.</p>
<p>There is a feed for the entire activity of the blog: <a href="https://blog.agayon.be/feeds/all.atom.xml">https://blog.agayon.be/feeds/all.atom.xml</a></p>
<p>Feeds are also available per</p>
<ul>
<li>
<p>Category</p>
<ul>
<li>XMPP: <a href="https://blog.agayon.be/feeds/xmpp.atom.xml">https://blog.agayon.be/feeds/xmpp.atom.xml</a></li>
<li>Misc <a href="https://blog.agayon.be/feeds/misc.atom.xml">https://blog.agayon.be/feeds/misc.atom.xml</a></li>
</ul>
</li>
<li>
<p>Tag</p>
<ul>
<li>XMPP: <a href="https://blog.agayon.be/feeds/tag-xmpp.atom.xml">https://blog.agayon.be/feeds/tag-xmpp.atom.xml</a></li>
<li>Python: <a href="https://blog.agayon.be/feeds/tag-python.atom.xml">https://blog.agayon.be/feeds/tag-python.atom.xml</a></li>
<li>Django <a href="https://blog.agayon.be/feeds/tag-django.atom.xml">https://blog.agayon.be/feeds/tag-django.atom.xml</a></li>
</ul>
</li>
</ul>
<p>More categories and tags will be created with the writing of new articles: <a href="https://en.wikipedia.org/wiki/Do_it_yourself">DIY</a>, Robots ,
<a href="https://www.agayon.be">Agayon</a>, ...</p>
<p>Stay tuned !</p>First Post2017-01-10T18:00:00+01:002017-01-10T18:00:00+01:00Arnaudtag:blog.agayon.be,2017-01-10:/first_post.html<p>First post of the Agayon</p><p>The blog of <a href="https://www.agayon.be">Agayon.be</a> is finally online.</p>
<p>I will post here some short articles about topics I am interested in (robotic, python, django, etc).</p>
<p>Stay tuned!</p>