Fix LXC network issues in Ubuntu 22.04

Ran into a curios problem while updating the GitHub Actions Workflow for a project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Run sudo lxc exec test-container --env DEBIAN_FRONTEND=noninteractive -- apt-get -y install -y openssh-client openssh-server openssh-sftp-server
Reading package lists...
Building dependency tree...
Reading state information...
openssh-client is already the newest version (1:8.9p1-3).
The following additional packages will be installed:
  libpsl5 libwrap0 ncurses-term publicsuffix python3-distro ssh-import-id wget
Suggested packages:
  molly-guard monkeysphere ssh-askpass ufw
The following NEW packages will be installed:
  libpsl5 libwrap0 ncurses-term openssh-server openssh-sftp-server
  publicsuffix python3-distro ssh-import-id wget
0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 1371 kB of archives.
After this operation, 7679 kB of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 openssh-sftp-server amd64 1:8.9p1-3

...

Ign:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 ssh-import-id all 5.11-0ubuntu1
Err:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 openssh-sftp-server amd64 1:8.9p1-3
  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out

...

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/ssh-import-id/ssh-import-id_5.11-0ubuntu1_all.deb  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: Process completed with exit code 100.

For the tests, I’m spinning up a LXC container on the Runner, and then try to install software in it. This specific Runner is using Ubuntu 22.04 (new), and the network connection to archive.ubuntu.com is failing. Another Runner in the same workflow, using Ubuntu 20.04, is working fine. 20.04 was the old test setup, 22.04 is the new one. No other changes. But why is it suddenly failing?

Finding the issue took me a long time, and I tried many different things:

Compared the network setup on the Runner, it’s the same
Compared the routing table on the Runner, it’s the same
Checked the Runner network setup, it’s the same
Compared the network setup in the LXC container, it’s the same
Compared the routing table in the LXC container, it’s the same
Disabled the LXC firewall:

1
2
3
4
5
      - name: Firewall IPv6
        run: sudo lxc network set lxdbr0 ipv6.firewall false

      - name: Firewall IPv4
        run: sudo lxc network set lxdbr0 ipv4.firewall false

Nothing …
Disabled ufw firewall on the Runner:

1
2
3
4
5
6
7
8
      - name: Firewall 1
        run: sudo ufw allow in on lxdbr0

      - name: Firewall 2
        run: sudo ufw route allow in on lxdbr0

      - name: Firewall 3
        run: sudo ufw route allow out on lxdbr0

Nothing …
Disabled IPv6 on the Runner:

1
2
3
4
5
6
7
8
9
      # 22.04 can't reach the Ubuntu servers over IPv6
      - name: Disable IPv6 (all)
        run: sudo lxc exec test-container --env DEBIAN_FRONTEND=noninteractive -- sysctl -w net.ipv6.conf.all.disable_ipv6=1

      - name: Disable IPv6 (default)
        run: sudo lxc exec test-container --env DEBIAN_FRONTEND=noninteractive -- sysctl -w net.ipv6.conf.default.disable_ipv6=1

      - name: Disable IPv6 (lo)
        run: sudo lxc exec test-container --env DEBIAN_FRONTEND=noninteractive -- sysctl -w net.ipv6.conf.lo.disable_ipv6=1

Nothing … stop: now the IPv4 connection is failing
Checked Docker: both 20.04 and 22.04 have Docker running by default, but no containers by default
The snap daemon is not installed on 22.04 by default, maybe some obscure setting or feature requires it? Tried that, installed snap, but nothing
Modified the bridge network for LXC, to use a different template and setup, nothing

Then I found a rather obscure hint on the LXC website: Docker might block connections by default. What?

Sure enough, two action lines later:

1
2
3
4
5
      - name: Disable Docker Firewall 1
        run: sudo iptables -I DOCKER-USER -i lxdbr0 -o eth0 -j ACCEPT

      - name: Disable Docker Firewall 2
        run: sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

And everything is working, even the IPv6 connectivity. This only affects 22.04, not 20.04. Apparently either Docker or Ubuntu changed something which blocks the traffic for the bridged networks by default. Thanks for nothing.

Categories: [Software]

Tags: [Docker] [Firewall] [Lxc] [Network] [Routing] [Ubuntu]

Friends