Was asked quite a few times how I do my backups with Restic.
For more than 10 years I was using “Duplicity” for backups, but in 2019 I changed to Restic. The main reason for the change was that Duplicity still can’t handle “Big Data”, as in: larger directories. In 2009 someone opened an issue on the Duplicity bugtracker, and this problem still exists as of today. For about two years I was shifting around the problem, excluding files, trying to make the sigfile smaller. But at some point I decided that it is enough and I need to change the tool.
The three PostgreSQL projects for this year’s Google Summer of Code are on track, and making good progress. All projects expect to finish on time.
The data gathering for performance farm members is completed, as well as the new implementation for the JSON data transfer. The project iteratively updated it’s goals, and adjusted for newly identified UI issues.
Current work centers around making the website more pretty and useful, as well as reducing the number of used JavaScript libraries. The next step is presenting the work to the PostgreSQL Community for broader feedback.
After installing Telegraf and hooking up everything into InfluxDB, I was missing the status of my backups. Every system here creates encrypted backups every night, and stores them on a central NAS, and off-site. But I want to know statistics about the backups, and see if something is not working.
I’m using Restic for the backups (will blog about this another time). However Telegraf does not support Restic directly, I need a few workarounds. This blog post however is not directly about monitoring the backups, but about how to write your own plugin for Telegraf.
A while ago I switched backups from Duplicity to Restic. About time: I was using Duplicity for many years (I think I started using it around 2010, long before Restic became available) and it served me well. But recently I ran into more and more issues, especially with archives getting larger and larger. There is an 11 years old open bug in the Duplicity bugtracker, which describes a showstopper for backing up larger archives. And it doesn’t look like this will be solved anytime soon. Therefore it was time for something new.
Since I’m rolling out my backups with Ansible, it was relatively easy to create a set of scripts for Restic which use almost the same infrastructure as the old Duplicity backups. That works as expected on all our laptops. But the Raspberry Pi, which does the fileserver backups, seem to had a problem. Backups took way longer than before, jumped from 30-60 minutes (depending on the amount of changes) to constantly around 10 hours.