This was even worse than the headline made it sound.

If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up.

First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host.

At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run.

And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...

Copy-pasta of Jonathan Corbet:

It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe.

https://news.ycombinator.com/item?id=31852477

If you're interested in detailed commentary on and investigations of the FOSS space, I can't recommend a subscription to LWN enough!

This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down.

It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this:

while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done

On the next reboot i was able to ssh in and change to the (then just released within the past hour) kernel that doesn't have this stupid bug. After another reboot you can move containderd back and it should be working again

affected: linux-image-5.13.0-1028 not affected: >linux-image-5.13.0-1029

I like Ubuntu, but in the last few months I have been following how things are packaged more closely.

For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption.

http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres...

I would have thought this would have been packaged earlier, as I would expect the Ubuntu + postgresql would be a common combination.

It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.

The cost of complexity showing itself.

A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

This isn't only bug.

I've recently migrated to Ubuntu 22.04 and got this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453

on HP ProLiant servers.

This seems like the kind of thing that automated testing should have been able to catch. It’s not like running Docker is a small use-case these days.

This affected my team and I directly. I had to recover a server from backup, we then postponed kernel updates when I noticed something was wrong after reading this https://www.mail-archive.com/kernel-packages@lists.launchpad...

Perhaps add “June 2022” to the title to reduce panic? Updated packages that resolve the issue were released on 2022-06-10, so this article is a post-mortem not an alert of a new problem that could affect people now.

They are just trying to convert docker users to snap enthusiasts.

ubuntu 22.04 also broke many IBM laptops. Took them 2 months to fix it, without any acknowledgement that the bug existed.

number of tickets in launchpad such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957

It makes me Q the value in my org looking into an ubuntu advantage subscription. When there are tickets that have lots of "me too" that result in unusable laptops, one should at least triage them / consolidate them into a single ticket and then be able to mark when fixed.

I'm using Oracle's ARM servers and I thought it was some weird patch they did to the kernel, the bug only disappeared when I force upgraded it to 22.04. Ubuntu/Canonical itself would be the last place I would have thought to be the source of a problem like that.

Interesting, around the same time both of my Ubuntu 20 laptops (Dell and Lenovo) started having major problems connecting to my home wifi. It only affects these 2 laptops, all my other devices have no problems. Before reading this I did think this was a result of a bad Ubuntu update. Given they switched kernel versions, my guess is that this is infact the culprit. Very annoying, it takes me anywhere from 5 to 45 minutes to establish a wifi session now.

Auto update strikes again. Really, we need to re-think this.

Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?

Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.

The more container bugs the better. I hope they can't fix it.

Does this apply for desktop also?