[13:28] hm, ejabberd in bionic. concerns? [15:59] right so, who pioneered USN-5089-1? [15:59] because that has introduced regressions for cases where LetsEncrypt certificates still have the older signature because they haven't autorenewed yet [16:01] i'm more curious why that wouldn't be pushed out ON cert chain day [16:01] instead of early [16:27] mdeslaur: leosilva: ^ important because I just did a test cert from Lets Encrypt for a test domain today, and it's still signed with DST Root CA X3, not the new cert chains, which means that Ubuntu has introduced a "broken trust" problem BEFORE the expiration of the CA cert. [16:28] i also emailed security@ Lets Encrypt to let them know of the broken trust problem. [16:28] so, any chance someone can unfubar the regression you've introduced until the cert actually dies? [16:32] teward: normally, the chain you'd receive would include "ISRG Root X1", is that not the case? [16:32] teward: can I see what the server is returning? [16:32] sdeziel: correct, chains are still issued with DST root on some cases, especialyl for domains that are 'still valid' with the original cert [16:32] mdeslaur: stdby [16:33] teward: and what makes you think that the problem will be solved in a week? [16:33] ('cause i had to beat the *hell* out of this thing to get the DST root temporarily added as an 'extra' cert I control to unbreak this regression) [16:33] teward: right but the fact that "ISRG Root X1" is signed by "DST Root CA X3" shouldn't prevent a client from validating the chain and stopping at X1 that is trusted, no? [16:33] sdeziel: except when DST Root CA X3 can't be validated [16:34] that's what was literally happening right now [16:34] give me a few seconds to pull the data, SSH is being slow right now [16:34] AFAIK, let's encrypt will not be changing anything in a week [16:34] https://community.letsencrypt.org/t/production-chain-changes/150739 [16:35] mdeslaur: then something is broken somewhere [16:35] teward: what version of ubuntu are you seeing the errors on? [16:35] mdeslaur: 20.04 LTS [16:36] 1 moment though stdby trying to get this to paste.u.c - the cert I jsut force-reissued for one server via acme client and the test domain cert [16:36] (hate working off hotspots) [16:36] teward: I'd like to test if the endpoint publicly reachable, ofc? [16:36] https://metasmoke.erwaysoftware.com/ is the current thing that got fubar'd by the USN [16:36] (Charcoal project.group for StackExchange, run off my infra) [16:37] i just force reissued that cert via acme client on the system it's on and it's still DST root signed [16:37] even though i just force reissued today [16:38] https://paste.ubuntu.com/p/8MVd8FfDyS/ has the cert and my output on curl [16:38] and note I just pulled the cert out of debug logs of the last LE run about 30 minutes or so ago [16:38] (which i force-renewed on) [16:38] as for 'be solved in a week', well if they have any care of their certs not exploding they'll fix their signing chains [16:39] i'm loading a new test domain just to verify this is still the problem but stdby getting to thigns is hard as sin on hotspot when you have only 2 bars [16:40] teward: you are apparently not sending the right chain on that server [16:40] you sure about that? 'cause that's all automated [16:40] yeah, that's not the right chain [16:40] then i have to open a bug somewhere [16:40] Certificate chain [16:40] 0 s:CN = metasmoke.erwaysoftware.com [16:40] i:C = US, O = Let's Encrypt, CN = R3 [16:40] 1 s:C = US, O = Let's Encrypt, CN = R3 [16:40] i:O = Digital Signature Trust Co., CN = DST Root CA X3 [16:41] hmmmmm [16:41] then that means acme clients are broken [16:41] i'll have to do some major work to fix that after the client OK's it [16:41] https://www.ssllabs.com/ssltest/analyze.html?d=metasmoke.erwaysoftware.com shows that R3 signed by X1 is missing [16:42] most browsers paper over this by fetching the missing intermediate but CLI clients are much less forgiving [16:42] hmmmmmmm [16:42] i'll have to file some bugs then [16:42] and do some manual 'emergency' work on the client machine [16:42] 'course i think they have haproxy in front of the endpoint which means haproxy is broken af [16:43] yep, sdeziel is right, the server isn't sending the R3 intermediary cert [16:44] you would have hit the same issue in a week [16:46] i'm going to have to call this client, and wake them up, they had off today >.> [16:46] i hate it when i'm wrong, but this introduced a "WTF" moment [16:46] it is quite unfortunate [16:47] a problem that only gets exposed because of something else [16:47] sometimes I'm amazed that the Internet works at all [17:00] to be precise, the chain sent include the R3 intermediate but the one that was signed directly by DST Root CA X3 (https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.txt) [17:01] sdeziel: not sure WHY that's the case, but we'll find out. [17:01] found out that autoupdates and certs were NOT working on client's machine because they're... a little dumb and pointed at invalid repos. Gonna do some cleanup THEN see if this works [17:15] sdeziel: mdeslaur: ... I think there's a missing cert in the full chain that is being returned by ACME clients to LetsEncrypt [17:17] this is the cert chain being returned by acme clients now - https://paste.ubuntu.com/p/f8BRW8H6wd/ - can you tell me if you see something missing here? [17:17] teward: I think it's a little more subtle than that. There are 2 versions of R3 and the site ships the "retired" one signed by DST Root CA X3 [17:17] sdeziel: more importantly, i'm asking if you can look at this chain and see if it's missing [17:17] checking [17:17] because that's the CA chain returned by the acme client in this system [17:17] or see if the chain is wrong [17:17] if the CHAIN is wrong then it doesn't matter, because that's the chain that the acme client is gettin gback [17:19] teward: that chain in the paste looks good to me [17:20] hmm [17:20] ok then i think something's up and i'll have to pound this system with a nuke and rebuild it to the proper specs [17:20] and make sure the client's haproxy is properly configured