Skip to content

Hanging script at the end of Python multiprocessing

While writing and stress-testing a Python script which uses the multiprocessing library, I ran into the problem that occasionally the script hangs at the end. Literally runs past the last line of code and then hangs.

In this script I'm using Queues and Events, so I made sure that I properly close the queues in the forked/spawned (tried both) processes and also clean out all queues in the parent. Nevertheless occasionally - like seldom, but it happens - the script hangs. Checked the process list, checked remaining threads, checked the queues, all fine. Still ...

When I hit Ctrl+C, I get the following stack trace:

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.9/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.9/multiprocessing/queues.py", line 201, in _finalize_join
    thread.join()
  File "/usr/lib/python3.9/threading.py", line 1053, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.9/threading.py", line 1073, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt

(Can't change the Python 3.9, as I don't control the target system where this is supposed to run)

There's nothing left to join, the only two remaining threads are MainThread and QueueFeederThread.

 

After some more debugging I had the simple idea not only to empty the queue before finishing the script, but setting the queue variable to None:

worker_queue = None

This helps. The script is no longer hanging at the end. At a later point I will see if the problem is fixed in newer versions.

Advent of Code 2020: "Seating System" - Day 11

Your career as a hacker brings you more and more unreasonable tasks. Today you arrive at the ferry station, head to your next gate (no one mentioned if you even got an ice cream), and figure out where to seat. Even though you are the first person in the waiting area that does not stop you from knowing the seating habits for all other passengers, and calculate their seating patterns.

Task 1: There are some seats, and some floor tiles. Every seat is empty. Seating follows some rules: 1) if a seat is empty, and no adjacent seat is occupied, someone will absolutely sit there 2) if a seat is currently occupied, but four or more adjacent seats are also occupied, the person will instantly jump up, disconnect all charging devices, and find a new seat 3) no one sits on the floor 4) this will not occupy all seats - which seems to be a good thing in Corona-19 times. Repeat that pattern until the seats do not change between two repeats. Then count the number of occupied seats.

Task 2: Adjacent seats spawn across floor tiles, so it can be a couple steps in each of the 8 possible directions. Looks like people don't like to look at other people here. Oh, and now it's 5 or more seats which must be occupied, before someone jumps up and changes seat. Again repeat the pattern of people getting up and finding new seats all one by one, until everyone is satisfied with their current seats. And then hope no additional passenger arrives!

 

Continue reading "Advent of Code 2020: "Seating System" - Day 11"

Advent of Code 2020: "Adapter Array" - Day 10

After you finished hacking the plane, your laptop dies: you forgot to charge it! Rookie mistake, or the calculations do consume a lot of power. The plane has a power outlet, but somehow it's a non-standard one, you need an adapter. But first a cocktail! Then you empty your backback on the seat next to you and find out that you have dozens of different adapters, which you can stack together. None of them really work, and hopefully you don't blow another fuse, or set the plane on fire!

The adapters you have can transform an imaginary Jolts value to another Jolts value, up to 3 Jolts up. And you find out that your laptop charger jumps 3 Jolts more than the highest rated adapter you have, although right now you don't know either of these value ... And since your laptop died, you need to calculate this on paper. Here is the list of 90 adapters in your backpack. Good luck!

Task 1: If you plug all 90 adapters together (Hopefully the plane is long enough, and the flight attendant does not spot what you are doing), what is the distribution of Joltage differences between the outlet (with 0 Jolts - seriously, no juice to begin with?), all adapters, and your +3 Jolts charger? Find a set that uses all adapters, calculate the Joltage differences, and multiply the +1 with the +3 differences. Seriously?

Task 2: Someone things that 90 adapters are not enough. Figure out in how many ways the 90 adapters can be arranged. On paper, because your laptop is still dead.

 

Continue reading "Advent of Code 2020: "Adapter Array" - Day 10"

Advent of Code 2020: "Encoding Error" - Day 9

Uh oh, either the ice cream at the airport was bad, or the cocktails are not good for you! How else can you explain that you hack the airplane systems with some paperclips and break the XMAS encryption, while the plane is airborne? The plane's on-board systems emits some cryptic numbers and your task is it to decrypt this. Wait, did I say task? Who gave this task to you? Oh well, please be careful, if you shorten a circuit you might as well crash your own plane.

Task 1: 25 numbers, followed by more numbers. The first 25 are the initial set, in the following numbers if you add two of the last 25 numbers the result might be he current number. You have to find the first one which is not the sum of two of the previous 25 numbers.

Tast 2: Now that you found the first number which does not add up, find a contiguous set of two or more numbers which add up to the result of task 1.

Continue reading "Advent of Code 2020: "Encoding Error" - Day 9"

Advent of Code 2020: "Handheld Halting" - Day 8

You fixed the colours of the bags yesterday and security allowed you to leave the baggage section and go get your ice cream before you board your next flight. On the flight the kid next to you recognizes you as the great hacker you are, and asks you to fix a problem with the handheld game console. Go and hack the bootloader while no documentation is available because the mobile internet is off. Turns out someone coded an infinite loop into the bootloader, and missed that fact during testing. What testing, do you ask? Well, let's not jump too deep into details, you have to hack a bootloader and make a kid happy. It's Christmas soon, after all! And your cocktail is waiting!

Task 1: The assembler instructions are easy to read, to make the kid happy you just have to figure out the state of the accumulator before the bootloader repeats its set of instructions into another great infinite loop. Not sure how that fixes the problem, but ok.

Task 2: The instruction list is corrupt. But not in the way you think. The code is supposed to produce a buffer overrun, jump outside of the instruction list and stop. Modern operating systems would prevent this and abort the program, here that is "normal termination". Change exactly one "jmp" into a "nop" or vice versa, but do not change the operand. Whichever change aborts the program by jumping outside the code - the accumulator state is what we are looking for. Hopefully the number is high enough to pay for all the cocktails!

 

Continue reading "Advent of Code 2020: "Handheld Halting" - Day 8"

Advent of Code 2020: "Handy Haversacks" - Day 7

They really trust you a lot ... Now all flights are delayed, that's quite usual in advent times. Except it's 2020 and no one is traveling anyway. But they also impose more regulations, and now bags need to have colours, and bags need to go into other bags. And someone has to figure it out. That's right, it's you! I hope the stopover is long enough to at least grab some ice cream!

Task 1: There is a huge list of rules which colores bag can hold how many other colored bags. You have a "shiny golden" bag, but apparently you are not allowed to carry it around, it has to go into another bag. Find out which ones are allowed, and how many of them. Oh, and bags can go into bags, which means: recursion.

Task 2: As usual someone made a mistake. You are, after all, allowed to carry your "shiny golden" bag, but now you also have to have other bags in it. Calculate the possibilities, based on the rules. Yes, recursion again, and tree climbing.

And if your head is swirling: so is mine. Why on North Pole Earth does anyone even remotely care about your shiny golden bag?

 

Continue reading "Advent of Code 2020: "Handy Haversacks" - Day 7"

Advent of Code 2020: "Custom Customs" - Day 6

The flight is approaching a regional airport, but you still have to fill out a customs form. Since you are good with hacking things together, you walk around the plane and gather all the answers from all groups on the plane. There are 26 yes-or-no questions, marked from a to z.

Task 1: Count how many questions are answeres "yes" in each group. Multiple "yes" answers for the same question do not count. The data is provided as blocks, kind of like two days before from the passport scanners. I can re-use some of the code - which will turn out to be a major headache!

Task 2: Of course someone wrote the wrong instructions down! It's up to you to fix the mess now. You have to find out how many groups have the same question answered by anyone in the group.

 

Continue reading "Advent of Code 2020: "Custom Customs" - Day 6"

Advent of Code 2020: "Binary Boarding" - Day 5

Pretty amazing the tasks they ask you to do "on the fly": Yesterday we fixed the passport scanner, today you have to write a piece of code which scans all the boarding passes in your environment (no one said they all carry them in the hand, and show it to you), and while waiting in the gangway you also have to identify your designated seat. Because, you know, you are such a good hacker you lost your boarding pass on the way from the gate to the plane ;-)

Task 1: Scan all the boarding passes with your phone camera, extract the code, and apply binary pattern matching to identify which seats exist. The airline has a pretty complex scheme going on, which depends on binary space partitioning. Find the highest seat number on the plane.

Task 2: Use the scanned data and find your seat. Seat belts sign is already on and you are still hacking in the gangway, hurry up! The only information you have: your seat exists, it's not the first and last one, and the seats next to yours (-1, +1) exist.

 

Continue reading "Advent of Code 2020: "Binary Boarding" - Day 5"

Advent of Code 2020: "Passport Processing" - Day 4

The challenge for day 4 is more suited for databases. The input is fuzzy, but with a CTE or two I should be able to clean it up. Solved the first part already with the kid earlier, but we decided to skip part two: mainly because it involved more complex string operations (best suited for regex operations), which is something the kid hasn't learned yet.

Task 1: The airport is busy, and the passport scanners do not function correctly. Wondering why every airport seems to be the same mess, but oh well. Anyway, being someone who can do some IT support they ask you to basically hack this thing on the fly and implement passport verification. If that would be possible on a real-world airport ... In addition to all the IT problems there, some people arrive with an ID which was issued by some "North Pole" organization which is not an official government body. But because everyone is waiting in line the people in charge decide to accept this ID as well. Only problem: this ID does not have a "Country ID" code. Your task, if you accept it: check the input data and if only the cid (Country ID) is missing, then accept this non-passport as valid document.

Task 2: Lines are getting longer and longer, and there are still problems with some passports. Your task is to apply additional validation on the fuzzy data which comes from the scanners. The task does not say what happens to people with invalid ID documents, but one can only assume that they cannot board the plane.

 

Continue reading "Advent of Code 2020: "Passport Processing" - Day 4"