| hallyn | hm, ejabberd in bionic. concerns? | 13:28 |
|---|---|---|
| teward | right so, who pioneered USN-5089-1? | 15:59 |
| teward | because that has introduced regressions for cases where LetsEncrypt certificates still have the older signature because they haven't autorenewed yet | 15:59 |
| teward | i'm more curious why that wouldn't be pushed out ON cert chain day | 16:01 |
| teward | instead of early | 16:01 |
| teward | mdeslaur: leosilva: ^ important because I just did a test cert from Lets Encrypt for a test domain today, and it's still signed with DST Root CA X3, not the new cert chains, which means that Ubuntu has introduced a "broken trust" problem BEFORE the expiration of the CA cert. | 16:27 |
| teward | i also emailed security@ Lets Encrypt to let them know of the broken trust problem. | 16:28 |
| teward | so, any chance someone can unfubar the regression you've introduced until the cert actually dies? | 16:28 |
| sdeziel | teward: normally, the chain you'd receive would include "ISRG Root X1", is that not the case? | 16:32 |
| mdeslaur | teward: can I see what the server is returning? | 16:32 |
| teward | sdeziel: correct, chains are still issued with DST root on some cases, especialyl for domains that are 'still valid' with the original cert | 16:32 |
| teward | mdeslaur: stdby | 16:32 |
| mdeslaur | teward: and what makes you think that the problem will be solved in a week? | 16:33 |
| teward | ('cause i had to beat the *hell* out of this thing to get the DST root temporarily added as an 'extra' cert I control to unbreak this regression) | 16:33 |
| sdeziel | teward: right but the fact that "ISRG Root X1" is signed by "DST Root CA X3" shouldn't prevent a client from validating the chain and stopping at X1 that is trusted, no? | 16:33 |
| teward | sdeziel: except when DST Root CA X3 can't be validated | 16:33 |
| teward | that's what was literally happening right now | 16:34 |
| teward | give me a few seconds to pull the data, SSH is being slow right now | 16:34 |
| mdeslaur | AFAIK, let's encrypt will not be changing anything in a week | 16:34 |
| mdeslaur | https://community.letsencrypt.org/t/production-chain-changes/150739 | 16:34 |
| teward | mdeslaur: then something is broken somewhere | 16:35 |
| mdeslaur | teward: what version of ubuntu are you seeing the errors on? | 16:35 |
| teward | mdeslaur: 20.04 LTS | 16:35 |
| teward | 1 moment though stdby trying to get this to paste.u.c - the cert I jsut force-reissued for one server via acme client and the test domain cert | 16:36 |
| teward | (hate working off hotspots) | 16:36 |
| sdeziel | teward: I'd like to test if the endpoint publicly reachable, ofc? | 16:36 |
| teward | https://metasmoke.erwaysoftware.com/ is the current thing that got fubar'd by the USN | 16:36 |
| teward | (Charcoal project.group for StackExchange, run off my infra) | 16:36 |
| teward | i just force reissued that cert via acme client on the system it's on and it's still DST root signed | 16:37 |
| teward | even though i just force reissued today | 16:37 |
| teward | https://paste.ubuntu.com/p/8MVd8FfDyS/ has the cert and my output on curl | 16:38 |
| teward | and note I just pulled the cert out of debug logs of the last LE run about 30 minutes or so ago | 16:38 |
| teward | (which i force-renewed on) | 16:38 |
| teward | as for 'be solved in a week', well if they have any care of their certs not exploding they'll fix their signing chains | 16:38 |
| teward | i'm loading a new test domain just to verify this is still the problem but stdby getting to thigns is hard as sin on hotspot when you have only 2 bars | 16:39 |
| sdeziel | teward: you are apparently not sending the right chain on that server | 16:40 |
| teward | you sure about that? 'cause that's all automated | 16:40 |
| mdeslaur | yeah, that's not the right chain | 16:40 |
| teward | then i have to open a bug somewhere | 16:40 |
| mdeslaur | Certificate chain | 16:40 |
| mdeslaur | 0 s:CN = metasmoke.erwaysoftware.com | 16:40 |
| mdeslaur | i:C = US, O = Let's Encrypt, CN = R3 | 16:40 |
| mdeslaur | 1 s:C = US, O = Let's Encrypt, CN = R3 | 16:40 |
| mdeslaur | i:O = Digital Signature Trust Co., CN = DST Root CA X3 | 16:40 |
| teward | hmmmmm | 16:41 |
| teward | then that means acme clients are broken | 16:41 |
| teward | i'll have to do some major work to fix that after the client OK's it | 16:41 |
| sdeziel | https://www.ssllabs.com/ssltest/analyze.html?d=metasmoke.erwaysoftware.com shows that R3 signed by X1 is missing | 16:41 |
| sdeziel | most browsers paper over this by fetching the missing intermediate but CLI clients are much less forgiving | 16:42 |
| teward | hmmmmmmm | 16:42 |
| teward | i'll have to file some bugs then | 16:42 |
| teward | and do some manual 'emergency' work on the client machine | 16:42 |
| teward | 'course i think they have haproxy in front of the endpoint which means haproxy is broken af | 16:42 |
| mdeslaur | yep, sdeziel is right, the server isn't sending the R3 intermediary cert | 16:43 |
| mdeslaur | you would have hit the same issue in a week | 16:44 |
| teward | i'm going to have to call this client, and wake them up, they had off today >.> | 16:46 |
| teward | i hate it when i'm wrong, but this introduced a "WTF" moment | 16:46 |
| mdeslaur | it is quite unfortunate | 16:46 |
| mdeslaur | a problem that only gets exposed because of something else | 16:47 |
| mdeslaur | sometimes I'm amazed that the Internet works at all | 16:47 |
| sdeziel | to be precise, the chain sent include the R3 intermediate but the one that was signed directly by DST Root CA X3 (https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.txt) | 17:00 |
| teward | sdeziel: not sure WHY that's the case, but we'll find out. | 17:01 |
| teward | found out that autoupdates and certs were NOT working on client's machine because they're... a little dumb and pointed at invalid repos. Gonna do some cleanup THEN see if this works | 17:01 |
| teward | sdeziel: mdeslaur: ... I think there's a missing cert in the full chain that is being returned by ACME clients to LetsEncrypt | 17:15 |
| teward | this is the cert chain being returned by acme clients now - https://paste.ubuntu.com/p/f8BRW8H6wd/ - can you tell me if you see something missing here? | 17:17 |
| sdeziel | teward: I think it's a little more subtle than that. There are 2 versions of R3 and the site ships the "retired" one signed by DST Root CA X3 | 17:17 |
| teward | sdeziel: more importantly, i'm asking if you can look at this chain and see if it's missing | 17:17 |
| sdeziel | checking | 17:17 |
| teward | because that's the CA chain returned by the acme client in this system | 17:17 |
| teward | or see if the chain is wrong | 17:17 |
| teward | if the CHAIN is wrong then it doesn't matter, because that's the chain that the acme client is gettin gback | 17:17 |
| sdeziel | teward: that chain in the paste looks good to me | 17:19 |
| teward | hmm | 17:20 |
| teward | ok then i think something's up and i'll have to pound this system with a nuke and rebuild it to the proper specs | 17:20 |
| teward | and make sure the client's haproxy is properly configured | 17:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!