[05:37] <ali1234> xnox: i'm confused by openssl SECLEVEL patches in ubuntu https://launchpad.net/ubuntu/+source/openssl/1.1.1f-1ubuntu1
[05:38] <ali1234> specifically i don't understand why you: Revert "Enable system default config to enforce TLS1.2 as a minimum" & "Increase default security level from 1 to 2".
[05:38] <ali1234> but then later on re-apply the same changes
[05:39] <ali1234> it seems to me that you end up exactly where you started, with default SECLEVEL=2 and all the problems that causes
[05:41] <ali1234> also it seems that ubuntu openssl is even more strict than the version in buster, and i don't understand why
[05:42] <ali1234> and i'm not sure if it is even intentional
[06:00] <ali1234> hmm ignore that last bit, i reproduced it with debian 10 as well
[06:47] <jamespage> ddstreet: the rmq changes in groovy have been merged and uploaded to debian unstable so a re-sync should be possible
[07:52] <ackk> juliank, hi, I reworked by branch for the maas transitional package based on your input. I created a new MP since we want this in focal now, as maas is gone from the archive in groovy. Could you please take a look when you have time: https://code.launchpad.net/~ack/ubuntu/+source/maas/+git/maas/+merge/384013 ?
[07:53] <ackk> juliank, I linked the SRU bug as well, please let me know if that needs tweaking as I don't know the process well
[08:20] <juliank> ackk: ack
[09:06] <FourDollars> Could anyone help me to upload https://bugs.launchpad.net/bugs/1875797?
[10:45] <ddstreet> jamespage i'm confused about rabbitmq-server.service Wants epmd@.socket...it's not a template socket (or service)...?
[10:45] <ddstreet> i see lp #1808766 but i still don't get what it's doing
[10:45] <ddstreet> or trying to do
[10:53] <paride> sil2100, Laney, xnox, bdmurray, we have a problem with a xenial update: https://bugs.launchpad.net/ubuntu/+source/json-c/+bug/1878723
[10:54] <paride> this is taking down xenial instances with unattended-upgrades enabled
[10:59] <Laney> paride: doh, but I can't really help pull the update
[10:59] <Laney> leosilva: ubuntu-archive ^-
[10:59] <ddstreet> ouch, that's bad
[11:00]  * apw looks at the panic removal request
[11:01] <apw> paride, i assume this was not happening before the json-c update, so it is that we are looking to remove
[11:02] <paride> apw, that's my understanding too
[11:03] <Laney> if it's that bad an emergency chmod might be warranted - thoughts?
[11:04] <apw> Laney, i am unclear if that is significantly quicker to propogate other than our mirrors
[11:05] <cjwatson> It doesn't help with propagation, but lots of people use our mirrors
[11:05] <Laney> It'll help people who use the Canoncial mirrors
[11:05] <apw> cjwatson, and we would not remove it in that case ?
[11:05] <cjwatson> https://wiki.canonical.com/UbuntuEngineering/DealingWithCrisis#Limitation
[11:05] <Laney> You would, but this would mitigate immediately
[11:05] <cjwatson> ^- that
[11:05] <Laney> yes, that wiki page
[11:06] <cjwatson> That procedure has been tried and updated recently
[11:09] <wgrant> At least it doesn't break unattended-upgrades, so as long as we get the new update out before people reboot we're not too bad.
[11:09] <chrisccoulson> hi, what's the status of this right now?
[11:10] <wgrant> So it seems of high priority to push a revert with greater version.
[11:10] <apw> ok removed; and the previous version is now copied back in
[11:10] <cjwatson> apw: Thanks, as long as we know that isn't the end of the relatively urgent story (as wgrant says)
[11:10] <apw> cjwatson, yep we know that thanks
[11:10] <wgrant> Is it just xenial?
[11:11] <apw> i have only been told about xenial
[11:11] <wgrant> Until we hear otherwise, let's prep reverts to -security of everything in that USN?
[11:11] <cjwatson> It was only issued to xenial and newer, and >= bionic doesn't have upstart
[11:12] <wgrant> Ah yes, no pre-xenial indeed.
[11:12] <apw> wgrant, how do we identify the USN
[11:12] <cjwatson> https://usn.ubuntu.com/4360-1/
[11:12] <wgrant> https://usn.ubuntu.com/4360-1/
[11:12] <cjwatson> Ah, ESM
[11:12] <wgrant> Ah yes
[11:12] <cjwatson> Conceivable it affects that too ...
[11:12] <wgrant> That's annoying, technically does need security team then
[11:12] <cjwatson> Is anyone in a position to try?
[11:13] <chrisccoulson> Hi :)
[11:13] <cjwatson> Before it takes out half our DC or something
[11:13] <rodarvus> folks, I'm from AWS - joining to cooperate on the efforts
[11:13] <wgrant> I have a VM I can sacrifice.
[11:13] <chrisccoulson> cjwatson, wgrant, I'm just going to build 0.11-4ubuntu2 as 0.11-4ubuntu2.2 in the security PPA and drop these changes.
[11:14] <rodarvus> to address a comment from wgrant above - we have actually heard from customers that this issue does indeed affect unattended-upgrades.
[11:14] <cjwatson> chrisccoulson: makes sense
[11:14] <wgrant> rodarvus: I mean rather that it shouldn't break unattended-upgrades from upgrading to a fixed version, just that if someone reboots in between they're in trouble.
[11:14] <rodarvus> right.
[11:14] <chrisccoulson> I don't think it's worth trying to fix the existing patches on a friday afternoon
[11:14] <wgrant> chrisccoulson: Please do. Ping me or Colin if there's any blocker at all.
[11:14] <wgrant> Absolutely.
[11:14] <wgrant> The vulnerability is way less bad than the breakage.
[11:14] <wgrant> Straight revert.
[11:14] <apw> chrisccoulson, if you need anything ping away
[11:15] <rodarvus> chrisccoulson: amazing. that sounds like the best way forward to me also. do you have any estimates on how long until the build complets and propagates to the security PPA?
[11:20] <xnox> ali1234: Ubuntu is more strict. I should publish my blog post about it. We enforce minimum protocol versions without configs.
[11:20] <wgrant> Huh
[11:20] <wgrant> Was it published to ESM?
[11:20] <wgrant> I don't see it on my trusty systems
[11:22] <AsciiWolf> kenvandine, hi, it looks like that the Snap Store "_Permissions" string translations are not synced from Launchpad, see: https://bugs.launchpad.net/snap-store/+bug/1878672
[11:30] <ivoks> wgrant: seems bionic is impacted too
[11:30] <wgrant> ivoks: We'd like to know what impact people are seeing.
[11:31] <wgrant> Since it can't break upstart there because upstart isn't used for boot.
[11:31] <ivoks> wgrant: aws just reported that customers are complaining for bionic too now
[11:31] <xnox> wgrant: *cough*
[11:31] <wgrant> xnox: ... oh?
[11:32] <wgrant> xnox: What dirty secrets do you hide!?
[11:32] <xnox> We do not re-exec from upstart to systemd
[11:33] <wgrant> xnox: You mean during an upgrade, pre-reboot?
[11:33] <wgrant> Breaking that is deeply unfortunate, but not world-burningly critical.
[11:33] <cjwatson> Did we use upstart for user sessions in xenial?  I forget
[11:34] <xnox> wgrant:  revert libjson-c update obviously. As far as i can remember on package upgrades that upstart uses, we trigger stateful re-exec of init & all user-session upstart inits.
[11:34] <philroche> We have reports that this is affecting libjson-c3 version 0.12.1-1.3ubuntu0.1 in bionic too. See SF case # 00278097
[11:34] <wgrant> xnox: ... on bionic?
[11:34] <xnox> and clearly that's crashing at serialization, or deserialization
[11:35] <wgrant> Oh or you mean it will break the system at upgrade on xenial, not at reboot?
[11:35] <wgrant> philroche: "affecting" is very vague.
[11:35] <wgrant> We'd really really really appreciate specifics.
[11:36] <xnox> wgrant: upstart is not published in bionic; but it can be still installed (xenial version). And i bet the maintainer hooks in libjson-c3 still try to restart it.
[11:36] <xnox> wgrant:  thus i'd check that libjson-c3 stops trying to initrctl daemon-reexec or whatever it was called
[11:36] <wgrant> xnox: Right, but breaking something that isn't /sbin/init is approximately one thousand times less critical than the situation on xenial, isn't it?
[11:36] <philroche> wgrant: it is vague :( Direct quote "Ubuntu will need to block the Bionic version as well, as it is also affected 0.12.1-1.3ubuntu0.1"
[11:36] <xnox> wgrant:  if they have upstart still installed, it means they are forcing to still use it.
[11:36] <wgrant> Ah
[11:36] <xnox> as pid 1
[11:37] <xnox> wgrant:  and upstart has support to be used inside chroots
[11:37] <cjwatson> philroche: The thing to sort out is whether that's just somebody observing that it's from the same USN, or whether it causes real issues (and if so what)
[11:37] <philroche> cjwatson: ack. I will get clarificaiton
[11:38] <ddstreet> rodarvus are you able to share details about how it affects bionic?
[11:39] <xnox> wgrant:  telinit u || : is done by json-c in xenial; but not bionic.
[11:39] <wgrant> Aha
[11:39] <xnox> wgrant:  so it almost feels like "affects bionic" => where someone has xenial enabled in sources, and force installed upstart?
[11:39] <wgrant> In which case they get to keep both pieces for the next two hours?
[11:39] <ddstreet> philroche cjwatson latest report from AWS is to ignore the 'affects bionic' comment.  they are doing more testing before being sure about that.
[11:39] <wgrant> Unless anyone strenuously disagrees.
[11:40] <philroche> ddstreet: ack
[11:40] <xnox> wgrant:  should i be looking into fixing up upstart stateful re-exec here?
[11:40] <xnox> and or rebuilding upstart, to see the testsuite emit fire balls
[11:42] <rodarvus> ddstreet: we don't have access to customer instances, so unfortunately all we have is a number of customer tickets. I (personally) don't have any indication that is happened in bionic.
[11:44] <xnox> wgrant:  rodarvus: ddstreet: at most, i'd expect dist-upgrades failing.
[11:44] <xnox> xenial => bionic, where one opted to stay on upstart in xenial.
[11:44] <wgrant> xnox: right, thanks for confirming.
[11:44] <AsciiWolf> kenvandine, btw. there is another issue that was already fixed in upstream and it would be great if it could also be fixed in Snap Store (and deb gnome-software): https://bugs.launchpad.net/snap-store/+bug/1750818
[11:44] <AsciiWolf> (the fix is easy to backport)
[11:46] <wgrant> So, now that reverts are in progress, and it seems likely that it is only properly fatal to xenial...
[11:46] <wgrant> We should try to work out the various xenial scenarios
[11:46] <wgrant> We have reports of OOMkills after upgrade and before reboot
[11:46] <wgrant> We have reports of panics on reboot
[11:46] <wgrant> And we have reports of OOMkills some time after reboot
[11:47] <wgrant> Oh, I guess the second case may be the third, just soon after boot.
[11:47] <wgrant> At least that means that booting and quickly upgrading should avert doom. The system isn't entirely unbootable.
[11:53] <cjwatson> Mirror triggering is disabled for the time being to avoid accidents
[11:53] <cjwatson> (from the master)
[12:01] <philroche> Update RE bionic "So bionic doesn't get the crash on init, and no kernel panic. But I am testing that the bug is still present in Bionic, just not triggered, as there is no upstart in Bionic. I'm looking to test the bug"
[12:02] <wgrant> philroche: Thanks for confimring.