ads' corner

ctid and other PostgreSQL table internals

While attending a conference (yay) the discussion turned to system columns, and especially 'ctid'. This was in context of backups, and the physical position of data on disk. The question was if that can be used to determine if an incremental backup is required, and if it is enough to just copy that field where the 'ctid' points to. Not quite, but I’d like to use this question to describe the system columns more in detail.

PostgreSQL Upgrades are hard

Together with Lætitia Avrot and Nikolay Samokhvalov I was invited to participate in a Community Panel about PostgreSQL Upgradability at Postgres Vision 2022. During this panel each of us provided examples for how easy or complicated *PostgreSQL* upgrades still are.

GSoC 2021 completed

The Google Summer of Code 2021 for the PostgreSQL Project is wrapped up.

Delete directories recursive on Hetzner Storage Box

Among other external solutions, I store some data on Storage Boxes from Hetzner. The Storage Box allows you to have sub-accounts, so for every server and system storing data there, I use a separate account. For each sub-account, one can select a subdirectory where the data is stored, and the sub-account then can only see this data. The Admin account can see all data, and see all directories.

PostgreSQL Project @ GSoC 2021

Wow! The PostgreSQL Project got all 7 proposals accepted into Google Summer of Code 2021!

Restic backup

Was asked quite a few times how I do my backups with Restic.

For more than 10 years I was using “Duplicity” for backups, but in 2019 I changed to Restic. The main reason for the change was that Duplicity still can’t handle “Big Data”, as in: larger directories. In 2009 someone opened an issue on the Duplicity bugtracker, and this problem still exists as of today. For about two years I was shifting around the problem, excluding files, trying to make the sigfile smaller. But at some point I decided that it is enough and I need to change the tool.


Google Summer of Code 2020 - Intermediate status update

The three PostgreSQL projects for this year’s Google Summer of Code are on track, and making good progress. All projects expect to finish on time.

Performance Farm

The data gathering for performance farm members is completed, as well as the new implementation for the JSON data transfer. The project iteratively updated it’s goals, and adjusted for newly identified UI issues.

Current work centers around making the website more pretty and useful, as well as reducing the number of used JavaScript libraries. The next step is presenting the work to the PostgreSQL Community for broader feedback.


Monitor additional details in Telegraf with the "Exec" input filter

After installing Telegraf and hooking up everything into InfluxDB, I was missing the status of my backups. Every system here creates encrypted backups every night, and stores them on a central NAS, and off-site. But I want to know statistics about the backups, and see if something is not working.

I’m using Restic for the backups (will blog about this another time). However Telegraf does not support Restic directly, I need a few workarounds. This blog post however is not directly about monitoring the backups, but about how to write your own plugin for Telegraf.


Restic upgrade on Debian Buster

A while ago I switched backups from Duplicity to Restic. About time: I was using Duplicity for many years (I think I started using it around 2010, long before Restic became available) and it served me well. But recently I ran into more and more issues, especially with archives getting larger and larger. There is an 11 years old open bug in the Duplicity bugtracker, which describes a showstopper for backing up larger archives. And it doesn’t look like this will be solved anytime soon. Therefore it was time for something new.

Since I’m rolling out my backups with Ansible, it was relatively easy to create a set of scripts for Restic which use almost the same infrastructure as the old Duplicity backups. That works as expected on all our laptops. But the Raspberry Pi, which does the fileserver backups, seem to had a problem. Backups took way longer than before, jumped from 30-60 minutes (depending on the amount of changes) to constantly around 10 hours.


Ansible: Just mount an existing filesystem

There is no easy way to “just mount” an existing filesystem in Ansible, using the mount module. I just want to mount an already defined mount point, and don’t really care about all the configuration here.