Skip to content

Restic backup

Was asked quite a few times how I do my backups with Restic.

For more than 10 years I was using "Duplicity" for backups, but in 2019 I changed to Restic. The main reason for the change was that Duplicity still can't handle "Big Data", as in: larger directories. In 2009 someone opened an issue on the Duplicity bugtracker, and this problem still exists as of today. For about two years I was shifting around the problem, excluding files, trying to make the sigfile smaller. But at some point I decided that it is enough and I need to change the tool.

Duplicity knows two backup modes: "full backup" and "incremental backup". Once in a while you take a full backup, and then you add incremental backups to that full backup. In order to restore a certain backup you need the full backup and the incremental backups. Therefore my go-to mode was to always have two full backups and a couple incremental backups in-between. Even if something goes wrong with the latest full backup, I can still go back to the previous full backup (of course with some changes lost, but that's still better than nothing). When taking a new full backup, the oldest one is only deleted when the new one is completed. Accordingly when a new incremental backup is created, it's a new set of files. Removing the backup removes all the files from this incremental backup. That worked well, but needed scheduling. Over time I wrote a wrapper script around Duplicity, which did schedule new full and incremental backups.

Restic works in a different way. There is no concept of "full backup" and "incremental backup". Basically every backup is a full backup, and Restic figures out which files changed, got deleted, or added. Also it does deduplication: if files are moved around, or appear multiple times, they are not added multiple times into the backup. Deduplication is something which Duplicity can't do. But because Restic can do deduplication, there is no common set of files which belong to a single snapshot. Data blobs from one backup can stay in the repository forever, removing snapshots might not remove any files at all.

Restic on the other hand needs "prune" to remove old data. A snapshot can be removed according to the policy specified, but this does not remove the data from the backup directory. A "prune" run will go over the data and remove any block which is no longer needed.

My first question - after figuring out which other backup tool to use: shall I replicate the wrapper script, or try something else? Given that the backup doesn't need complex scheduling, I decided against writing a complex wrapper. And since I am now deploying all devices with Ansible, I decided to integrate this into my Playbooks, and deploy a set of shell scripts. The goal was to have a small number of dedicated scripts doing the daily backup work, and another set of "helper" scripts which I can use to inspect the backup, modify it, or restore something.

My main goals for this: "small number of programs/scripts" (Unix style: each tool does one job), "rapid development" (don't spend weeks writing another scheduler), "rapid deployment" (re-run Playbooks and let Ansible deploy this to all devices).


Continue reading "Restic backup"

openHAB: Configuration model '....rules' has errors, therefore ignoring it: no viable alternative at input '...'

The openHAB rule system is not very helpful by pointing out when it's missing something, or when there is an error to actually point to the problem.

Was debugging an error message for a while and couldn't figure out what is wrong:

Configuration model '....rules' has errors, therefore ignoring it: [152,5]: no viable alternative at input '...'

The line number specified was somewhere else in the file, and had nothing to do with the rule in question. After removing almost everything from the rule, except a logInfo() message, the error still happened - and then it occured to me: I forgot to specify "Item".

My faulty code was:

rule "Rule Name"
    Sensor_name received update

where it should have been:

rule "Rule Name"
    Item Sensor_name received update

Error message totally not helpful ...

Advent of Code 2020: "Seating System" - Day 11

Your career as a hacker brings you more and more unreasonable tasks. Today you arrive at the ferry station, head to your next gate (no one mentioned if you even got an ice cream), and figure out where to seat. Even though you are the first person in the waiting area that does not stop you from knowing the seating habits for all other passengers, and calculate their seating patterns.

Task 1: There are some seats, and some floor tiles. Every seat is empty. Seating follows some rules: 1) if a seat is empty, and no adjacent seat is occupied, someone will absolutely sit there 2) if a seat is currently occupied, but four or more adjacent seats are also occupied, the person will instantly jump up, disconnect all charging devices, and find a new seat 3) no one sits on the floor 4) this will not occupy all seats - which seems to be a good thing in Corona-19 times. Repeat that pattern until the seats do not change between two repeats. Then count the number of occupied seats.

Task 2: Adjacent seats spawn across floor tiles, so it can be a couple steps in each of the 8 possible directions. Looks like people don't like to look at other people here. Oh, and now it's 5 or more seats which must be occupied, before someone jumps up and changes seat. Again repeat the pattern of people getting up and finding new seats all one by one, until everyone is satisfied with their current seats. And then hope no additional passenger arrives!


Continue reading "Advent of Code 2020: "Seating System" - Day 11"

Advent of Code 2020: "Adapter Array" - Day 10

After you finished hacking the plane, your laptop dies: you forgot to charge it! Rookie mistake, or the calculations do consume a lot of power. The plane has a power outlet, but somehow it's a non-standard one, you need an adapter. But first a cocktail! Then you empty your backback on the seat next to you and find out that you have dozens of different adapters, which you can stack together. None of them really work, and hopefully you don't blow another fuse, or set the plane on fire!

The adapters you have can transform an imaginary Jolts value to another Jolts value, up to 3 Jolts up. And you find out that your laptop charger jumps 3 Jolts more than the highest rated adapter you have, although right now you don't know either of these value ... And since your laptop died, you need to calculate this on paper. Here is the list of 90 adapters in your backpack. Good luck!

Task 1: If you plug all 90 adapters together (Hopefully the plane is long enough, and the flight attendant does not spot what you are doing), what is the distribution of Joltage differences between the outlet (with 0 Jolts - seriously, no juice to begin with?), all adapters, and your +3 Jolts charger? Find a set that uses all adapters, calculate the Joltage differences, and multiply the +1 with the +3 differences. Seriously?

Task 2: Someone things that 90 adapters are not enough. Figure out in how many ways the 90 adapters can be arranged. On paper, because your laptop is still dead.


Continue reading "Advent of Code 2020: "Adapter Array" - Day 10"

Advent of Code 2020: "Encoding Error" - Day 9

Uh oh, either the ice cream at the airport was bad, or the cocktails are not good for you! How else can you explain that you hack the airplane systems with some paperclips and break the XMAS encryption, while the plane is airborne? The plane's on-board systems emits some cryptic numbers and your task is it to decrypt this. Wait, did I say task? Who gave this task to you? Oh well, please be careful, if you shorten a circuit you might as well crash your own plane.

Task 1: 25 numbers, followed by more numbers. The first 25 are the initial set, in the following numbers if you add two of the last 25 numbers the result might be he current number. You have to find the first one which is not the sum of two of the previous 25 numbers.

Tast 2: Now that you found the first number which does not add up, find a contiguous set of two or more numbers which add up to the result of task 1.

Continue reading "Advent of Code 2020: "Encoding Error" - Day 9"

Advent of Code 2020: "Handheld Halting" - Day 8

You fixed the colours of the bags yesterday and security allowed you to leave the baggage section and go get your ice cream before you board your next flight. On the flight the kid next to you recognizes you as the great hacker you are, and asks you to fix a problem with the handheld game console. Go and hack the bootloader while no documentation is available because the mobile internet is off. Turns out someone coded an infinite loop into the bootloader, and missed that fact during testing. What testing, do you ask? Well, let's not jump too deep into details, you have to hack a bootloader and make a kid happy. It's Christmas soon, after all! And your cocktail is waiting!

Task 1: The assembler instructions are easy to read, to make the kid happy you just have to figure out the state of the accumulator before the bootloader repeats its set of instructions into another great infinite loop. Not sure how that fixes the problem, but ok.

Task 2: The instruction list is corrupt. But not in the way you think. The code is supposed to produce a buffer overrun, jump outside of the instruction list and stop. Modern operating systems would prevent this and abort the program, here that is "normal termination". Change exactly one "jmp" into a "nop" or vice versa, but do not change the operand. Whichever change aborts the program by jumping outside the code - the accumulator state is what we are looking for. Hopefully the number is high enough to pay for all the cocktails!


Continue reading "Advent of Code 2020: "Handheld Halting" - Day 8"

Advent of Code 2020: "Handy Haversacks" - Day 7

They really trust you a lot ... Now all flights are delayed, that's quite usual in advent times. Except it's 2020 and no one is traveling anyway. But they also impose more regulations, and now bags need to have colours, and bags need to go into other bags. And someone has to figure it out. That's right, it's you! I hope the stopover is long enough to at least grab some ice cream!

Task 1: There is a huge list of rules which colores bag can hold how many other colored bags. You have a "shiny golden" bag, but apparently you are not allowed to carry it around, it has to go into another bag. Find out which ones are allowed, and how many of them. Oh, and bags can go into bags, which means: recursion.

Task 2: As usual someone made a mistake. You are, after all, allowed to carry your "shiny golden" bag, but now you also have to have other bags in it. Calculate the possibilities, based on the rules. Yes, recursion again, and tree climbing.

And if your head is swirling: so is mine. Why on North Pole Earth does anyone even remotely care about your shiny golden bag?


Continue reading "Advent of Code 2020: "Handy Haversacks" - Day 7"

Advent of Code 2020: "Custom Customs" - Day 6

The flight is approaching a regional airport, but you still have to fill out a customs form. Since you are good with hacking things together, you walk around the plane and gather all the answers from all groups on the plane. There are 26 yes-or-no questions, marked from a to z.

Task 1: Count how many questions are answeres "yes" in each group. Multiple "yes" answers for the same question do not count. The data is provided as blocks, kind of like two days before from the passport scanners. I can re-use some of the code - which will turn out to be a major headache!

Task 2: Of course someone wrote the wrong instructions down! It's up to you to fix the mess now. You have to find out how many groups have the same question answered by anyone in the group.


Continue reading "Advent of Code 2020: "Custom Customs" - Day 6"