ads' corner

While writing and stress-testing a Python script which uses the multiprocessing library, I ran into the problem that occasionally the script hangs at the end. Literally runs past the last line of code and then hangs.

In this script I'm using Queues and Events, so I made sure that I properly close the queues in the forked/spawned (tried both) processes and also clean out all queues in the parent. Nevertheless occasionally - like seldom, but it happens - the script hangs. Checked the process list, checked remaining threads, checked the queues, all fine. Still ...

When I hit Ctrl+C, I get the following stack trace:

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.9/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.9/multiprocessing/queues.py", line 201, in _finalize_join
    thread.join()
  File "/usr/lib/python3.9/threading.py", line 1053, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.9/threading.py", line 1073, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt

(Can't change the Python 3.9, as I don't control the target system where this is supposed to run)

There's nothing left to join, the only two remaining threads are MainThread and QueueFeederThread.

After some more debugging I had the simple idea not only to empty the queue before finishing the script, but setting the queue variable to None:

worker_queue = None

This helps. The script is no longer hanging at the end. At a later point I will see if the problem is fixed in newer versions.

When I make changes in OpenStreetMap, I often need to extract the object colour from a picture or a video, as example for a building, bench or a roof. This collides with a feature in KDE which I really like: Dim Inactive Windows.

The setting is in the KDE settings, under Desktop Effects -> Dim Inactive. As the name implies, it dims (makes them a bit more dark) all non-active windows - which makes it visually clear which window is currently active. Quite useful.

For picking a colour from a picture/video I'm using KColorChooser, which provides the colour in Hex code - exactly what I need in OSM.

However when KColorChooser is the active window, and all other windows are dimmed, it extracts the wrong colour from the picture/video - the colour which is already dimmed. Not what I need. Therefore when I do OSM edits, I temporarily disable this feature.

At first I went into the KDE settings, twice, but this is too many clicks. Then I figured out the commandline arguments to disable and enable this feature.

Disable "Dim Inactive":

kwriteconfig5 --file kwinrc --group Plugins --key diminactiveEnabled false
qdbus org.kde.KWin /KWin reconfigure

Enable "Dim Inactive":

kwriteconfig5 --file kwinrc --group Plugins --key diminactiveEnabled false
qdbus org.kde.KWin /KWin reconfigure

Now all I need to do is execute a small script which is in my personal ~/bin directory. Let the edits continue.

Note: I wish JOSM would have an integrated video player which also allows to pick colours. But so far I haven't found any such plugin.

Shortcodes in Hugo are a neat and poweful system to avoid repating the same piece of text over and over again. Let's say I have the following text:

Nunc in odio id magna molestie congue. Donec erat nulla, pulvinar eget volutpat non, molestie at nisi. Curabitur nec tristique felis. Cras imperdiet, ante et vestibulum iaculis, tellus ipsum pulvinar felis, at viverra est tellus et eros. In nec dignissim lectus, bibendum hendrerit ex. Praesent lobortis eget justo non vehicula.

Nulla et neque cursus libero tristique laoreet nec a ligula. Fusce sit “amet” scelerisque erat. Quisque lorem lectus, lobortis vitae mattis non, tincidunt sed felis. Donec sit “amet” erat nibh.

Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In et imperdiet dui. In ut enim pharetra, blandit purus vel, malesuada est. Morbi sollicitudin eget leo nec dignissim. Praesent sed leo quis purus pretium aliquet sed quis arcu. Pellentesque facilisis tellus nulla, dignissim laoreet quam maximus et.

And I want to link the "amet" word, then I can create a shortcode:

layouts/shortcodes/amet.html

And in this shortcode I place the text:

<a href="https://www.lipsum.com/feed/html">amet</a>

In the Markdown source this is:

Nulla et neque cursus libero tristique laoreet nec a ligula. Fusce sit "{{< amet >}}" scelerisque erat. Quisque lorem lectus, lobortis vitae mattis non, tincidunt sed felis. Donec sit "{{< amet >}}" erat nibh.

Pretty easy. However Hugo by default does a little bit too much: while generating the static content, it adds a line break. Which in the browser results in:

Nulla et neque cursus libero tristique laoreet nec a ligula. Fusce sit “amet ” scelerisque erat. Quisque lorem lectus, lobortis vitae mattis non, tincidunt sed felis. Donec sit “amet ” erat nibh.
                                                                            ^                                                                                                         ^

There is a nasty little space between the word and the quotation mark. The generated HTML source shows the problem:

  <p>Nunc in odio id magna molestie congue. Donec erat nulla, pulvinar eget volutpat non, molestie at nisi. Curabitur nec tristique felis. Cras imperdiet, ante et vestibulum iaculis, tellus ipsum pulvinar felis, at viverra est tellus et eros. In nec dignissim lectus, bibendum hendrerit ex. Praesent lobortis eget justo non vehicula.</p>
<p>Nulla et neque cursus libero tristique laoreet nec a ligula. Fusce sit &ldquo;<a href="https://www.lipsum.com/feed/html">amet</a>
&rdquo; scelerisque erat. Quisque lorem lectus, lobortis vitae mattis non, tincidunt sed felis. Donec sit &ldquo;<a href="https://www.lipsum.com/feed/html">amet</a>
&rdquo; erat nibh.</p>

Usually a line break in HTML is not a big deal, the browsers deal with that. But here an unnecessary space is added while rendering the content. Hugo makes it complicated to avoid this problem. There is no default setting to turn this off, however the shortcode can have an extra Hugo tag to avoid the training linebreak:

<a href="https://www.lipsum.com/feed/html">amet</a>{{- / Strip trailing newline. / -}}

By adding the {{- -}}, the training line break is not included in the final HTML output, and the quotation mark comes right after the shortcode content.

Once or twice a year I have to setup a new Redmine project, which we use to organize conferences for PostgreSQL Europe.

Every single time I'm trying to figure out how the child pages for the meeting notes work.

Steps required:

• Create a page "MeetingMinutes", doesn't need to have any content, but it must exist
• Create the first child page with meeting minutes, as parent page select "MeetingMinutes"
• On the main wiki page insert a link with: "{{child_pages(MeetingMinutes)}}"

Every time I forget step 1), because this page is only required as parent page, and serves no further purpose. And then I'm wondering why I can't create child pages.

The Huginn software is not only good for monitoring Twitter feeds, it can also be used to monitor websites for changes.

That's actually easy and needs only two agents:

• Website Agent
• Notification Agent - Email Agent, maybe

A very common use case for hooks in git is a "pre-commit" hook. This hook is used to verify the to-be-committed data before it is added to the repository.

One important note: hooks are not part of the repository itself. Everyone can install a hook on it's own checkout of a repository, but by default the hook is not there when you clone/checkout the repository. This avoids security problems by executing arbitrary code during "git commit", or any "git" operation.
Because of this implication it is common that developers install a hook from somewhere in the repository into the ".git/hooks" directory. And in addition, the server side (the repository) can run the same checks during "git push", to enforce the rules.

Hooks in git work in a simple way: whatever program or script is run as the hook has to set a return code. If the return code is "0", git proceeds. If it's not "0", git aborts the operation.

One of the things I'm using Huginn for is monitoring for Twitter keywords. Some of them might appear in pairs, as example people like to tag Tweets about PostgreSQL with both #postgresql and #postgres. When I was using IFTTT this always created two emails, one for each hashtag. With Huginn I can deduplicate the events, and only notify about the first occurrence.

I need the following agents for this scenario:

• Twitter Search Agent
• Trigger Agent
• De Duplication Agent
• Email Agent

Huginn comes with a number of Twitter agents, but before they can be used you need to authenticate them. That process requires multiple steps, and they are not well documented.

The basics are described on the "Configuring OAuth applications" Wiki page.

For the Twitter integration one must know that you are using a single Twitter app for all users, but every Huginn user can (and should) have this app authenticated against their own Twitter account. Therefore if you want to use multiple Twitter accounts in Huginn, you need multiple Huginn users.

Huginn is a great piece of software, but the documentation is ... a bit sparse. Especially when it comes to details of the agents. I'm going to blog about a couple more more examples in the future.

For another project I'm using Leaflet, a JavaScript library for rendering maps in a browser. New versions are released occasionally, and I want to know when it's time to update the project website. Huginn can do that.

After installing Telegraf and hooking up everything into InfluxDB, I was missing the status of my backups. Every system here creates encrypted backups every night, and stores them on a central NAS, and off-site. But I want to know statistics about the backups, and see if something is not working.

I'm using Restic for the backups (will blog about this another time). However Telegraf does not support Restic directly, I need a few workarounds. This blog post however is not directly about monitoring the backups, but about how to write your own plugin for Telegraf.