[13:28] <hallyn> hm, ejabberd in bionic.  concerns?
[15:59] <teward> right so, who pioneered USN-5089-1?
[15:59] <teward> because that has introduced regressions for cases where LetsEncrypt certificates still have the older signature because they haven't autorenewed yet
[16:01] <teward> i'm more curious why that wouldn't be pushed out ON cert chain day
[16:01] <teward> instead of early
[16:27] <teward> mdeslaur: leosilva:  ^ important because I just did a test cert from Lets Encrypt for a test domain today, and it's still signed with DST Root CA X3, not the new cert chains, which means that Ubuntu has introduced a "broken trust" problem BEFORE the expiration of the CA cert.
[16:28] <teward> i also emailed security@ Lets Encrypt to let them know of the broken trust problem.
[16:28] <teward> so, any chance someone can unfubar the regression you've introduced until the cert actually dies?
[16:32] <sdeziel> teward: normally, the chain you'd receive would include "ISRG Root X1", is that not the case?
[16:32] <mdeslaur> teward: can I see what the server is returning?
[16:32] <teward> sdeziel: correct, chains are still issued with DST root on some cases, especialyl for domains that are 'still valid' with the original cert
[16:32] <teward> mdeslaur: stdby
[16:33] <mdeslaur> teward: and what makes you think that the problem will be solved in a week?
[16:33] <teward> ('cause i had to beat the *hell* out of this thing to get the DST root temporarily added as an 'extra' cert I control to unbreak this regression)
[16:33] <sdeziel> teward: right but the fact that "ISRG Root X1" is signed by "DST Root CA X3" shouldn't prevent a client from validating the chain and stopping at X1 that is trusted, no?
[16:33] <teward> sdeziel: except when DST Root CA X3 can't be validated
[16:34] <teward> that's what was literally happening right now
[16:34] <teward> give me a few seconds to pull the data, SSH is being slow right now
[16:34] <mdeslaur> AFAIK, let's encrypt will not be changing anything in a week
[16:34] <mdeslaur> https://community.letsencrypt.org/t/production-chain-changes/150739
[16:35] <teward> mdeslaur: then something is broken somewhere
[16:35] <mdeslaur> teward: what version of ubuntu are you seeing the errors on?
[16:35] <teward> mdeslaur: 20.04 LTS
[16:36] <teward> 1 moment though stdby trying to get this to paste.u.c - the cert I jsut force-reissued for one server via acme client and the test domain cert
[16:36] <teward> (hate working off hotspots)
[16:36] <sdeziel> teward: I'd like to test if the endpoint publicly reachable, ofc?
[16:36] <teward> https://metasmoke.erwaysoftware.com/ is the current thing that got fubar'd by the USN
[16:36] <teward> (Charcoal project.group for StackExchange, run off my infra)
[16:37] <teward> i just force reissued that cert via acme client on the system it's on and it's still DST root signed
[16:37] <teward> even though i just force reissued today
[16:38] <teward> https://paste.ubuntu.com/p/8MVd8FfDyS/ has the cert and my output on curl
[16:38] <teward> and note I just pulled the cert out of debug logs of the last LE run about 30 minutes or so ago
[16:38] <teward> (which i force-renewed on)
[16:38] <teward> as for 'be solved in a week', well if they have any care of their certs not exploding they'll fix their signing chains
[16:39] <teward> i'm loading a new test domain just to verify this is still the problem but stdby getting to thigns is hard as sin on hotspot when you have only 2 bars
[16:40] <sdeziel> teward: you are apparently not sending the right chain on that server
[16:40] <teward> you sure about that?  'cause that's all automated
[16:40] <mdeslaur> yeah, that's not the right chain
[16:40] <teward> then i have to open a bug somewhere
[16:40] <mdeslaur> Certificate chain
[16:40] <mdeslaur>  0 s:CN = metasmoke.erwaysoftware.com
[16:40] <mdeslaur>    i:C = US, O = Let's Encrypt, CN = R3
[16:40] <mdeslaur>  1 s:C = US, O = Let's Encrypt, CN = R3
[16:40] <mdeslaur>    i:O = Digital Signature Trust Co., CN = DST Root CA X3
[16:41] <teward> hmmmmm
[16:41] <teward> then that means acme clients are broken
[16:41] <teward> i'll have to do some major work to fix that after the client OK's it
[16:41] <sdeziel> https://www.ssllabs.com/ssltest/analyze.html?d=metasmoke.erwaysoftware.com shows that R3 signed by X1 is missing
[16:42] <sdeziel> most browsers paper over this by fetching the missing intermediate but CLI clients are much less forgiving
[16:42] <teward> hmmmmmmm
[16:42] <teward> i'll have to file some bugs then
[16:42] <teward> and do some manual 'emergency' work on the client machine
[16:42] <teward> 'course i think they have haproxy in front of the endpoint which means haproxy is broken af
[16:43] <mdeslaur> yep, sdeziel is right, the server isn't sending the R3 intermediary cert
[16:44] <mdeslaur> you would have hit the same issue in a week
[16:46] <teward> i'm going to have to call this client, and wake them up, they had off today >.>
[16:46] <teward> i hate it when i'm wrong, but this introduced a "WTF" moment
[16:46] <mdeslaur> it is quite unfortunate
[16:47] <mdeslaur> a problem that only gets exposed because of something else
[16:47] <mdeslaur> sometimes I'm amazed that the Internet works at all
[17:00] <sdeziel> to be precise, the chain sent include the R3 intermediate but the one that was signed directly by DST Root CA X3 (https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.txt)
[17:01] <teward> sdeziel: not sure WHY that's the case, but we'll find out.
[17:01] <teward> found out that autoupdates and certs were NOT working on client's machine because they're... a little dumb and pointed at invalid repos.  Gonna do some cleanup THEN see if this works
[17:15] <teward> sdeziel: mdeslaur: ... I think there's a missing cert in the full chain that is being returned by ACME clients to LetsEncrypt
[17:17] <teward> this is the cert chain being returned by acme clients now - https://paste.ubuntu.com/p/f8BRW8H6wd/ - can you tell me if you see something missing here?
[17:17] <sdeziel> teward: I think it's a little more subtle than that. There are 2 versions of R3 and the site ships the "retired" one signed by DST Root CA X3
[17:17] <teward> sdeziel: more importantly, i'm asking if you can look at this chain and see if it's missing
[17:17] <sdeziel> checking
[17:17] <teward> because that's the CA chain returned by the acme client in this system
[17:17] <teward> or see if the chain is wrong
[17:17] <teward> if the CHAIN is wrong then it doesn't matter, because that's the chain that the acme client is gettin gback
[17:19] <sdeziel> teward: that chain in the paste looks good to me
[17:20] <teward> hmm
[17:20] <teward> ok then i think something's up and i'll have to pound this system with a nuke and rebuild it to the proper specs
[17:20] <teward> and make sure the client's haproxy is properly configured