=== axino` is now known as axino === kostkon_ is now known as kostkon [11:06] it seems like nginx 1.18 is incompatible with OpenSSL 3, yet Ubuntu 22.04 ships nginx 1.18 and OpenSSL 3 [11:06] what makes you think that ? [11:07] what is the depends on the nginx package in terms of ssl [11:08] nginx don't work for me on 22.04, and threads online are suggesting that nginx 1.18 with OpenSSL 3 breaks in exactly the way I'm seeing [11:08] what's the package depends on nginx [11:09] I don't understand the question [11:09] the nginx package, will have dependencies and optional dependencies [11:09] what are the dependencies on SSL [11:11] it seems to depend on libssl >=3 [11:12] I was just looking https://packages.ubuntu.com/jammy/nginx-core [11:12] so it needs ssl3 alpha 1 according to that depend [11:12] I'm assuming you're using the alpha 1 package [11:13] I'm using whatever Ubuntu installed, which includes whatever nginx depends on [11:13] do an update and make sure it's the current alpha 1 package, [11:15] I seem to be on openssl 3.0.2-0ubuntu1.6 [11:15] all packages are up to date [11:19] if it's still broke - seems like you should check if a bug has been has aready been raised, if so, add your details to the bug [11:19] if not raise a bu [11:19] bug [11:21] but that's what's so weird, this isn't just a normal bug where some small thing is broken in some edge case [11:21] this is "nginx doesn't work in Ubuntu LTS" [11:21] yes, I heard you [11:21] so there should already be a bug [11:21] if nginx cannot serve ssl in 22.04 [11:21] no, it should not have been released [11:22] that's a perfectly valid view [11:22] however, checking if there is a bug and reading the bug may give context [11:22] that's what's making me think I must be doing something wrong, nobody in their right mind would release an Ubuntu server LTS which can't serve web pages [11:22] may give insight [11:22] well, I'd certainly be checking for bug reports, as it could be a known bug that was released on purpose while waiting for an update etc [11:23] I will check out bug reports [11:24] there seems to be some nginx conf hack to work around the issue, it might be that for example the nginx package comes with that config option set but I didn't let it overwrite my conf or something [11:24] I'll investigate [11:45] I'm unable to find any bug reports for nginx [11:46] I simply don't understand this [11:46] the config patch I talked about was an openssl config, and doesn't work [12:10] it seems unlikely (not impossible) that such a critical failure would not have been caught in testing, and even if it wasn't, no-one has logged a bug against it [12:16] I don't have anything related to nginx in my openssl 3.0 bugs :/ === lucasmoura__ is now known as lucasmoura === arif-ali_ is now known as arif-ali [16:14] i know for an absolute fact it works fine with openssl 3.0 and serves SSL fine for all currently supported TLS versions [16:15] and I put a TON of effort into testing nginx every cycle [16:16] mort: can you pastebin your config you were attempting to use initially and I will run tests on a fresh lts environment/vm [16:17] (fyi I am nginx comaintainer in Debian and a huge contributor here in Ubuntu for nginx) [16:20] teward[m]: so, what happens seems to be: nginx works but logs critical ssl errors, *and also* something else was broken with my php config which caused nginx not to serve the site it's supposed to [16:20] (and php-fpm's log file or journald log or anything else didn't contain the output which I'm sure php has somehow printed somewhere) [16:21] it's weird, it was simply a matter of the php code throwing an exception, it should've ended up in some log [16:21] are you using fpm? use fastcgi_intercept_errors on; in the config [16:22] otherwise the php errors sometimew just disappear [16:22] amazing [16:22] nginx then will throw the error message into error.log typically then [16:22] it didn't, fwiw [16:23] unless you put that config bit i said then no it wont [16:23] I don't understand how these things hang together, nor why everything uses error.log files rather than printing to stderr so that it ends up in journald [16:23] the php error log is for fpm errors [16:23] for normal errors, since your using fastcgi, they go over the fastcgi socket [16:24] so your webserver would have to log them === arif-ali_ is now known as arif-ali [16:24] and nginx promptyl ignores them [16:24] awesome [16:24] that's exactly the right default [16:24] and are logged to nginx's log files not stderr [16:24] nope [16:24] mort no [16:24] add this to your php block: [16:24] fastcgi_intercept_errors on; [16:24] yes otherwise it ignores errors [16:24] that's what I'm calling exactly the right default [16:25] So now you're just whining then [16:25] yes, it's a terrible choice which cost me a whole lot of hours and frustration [16:25] a system which ignores errors unless explicitly configured to not ignore errors is broken [16:25] there was a default to make php just work in nginx? [16:26] I always added that myself, so ya, no point in there being a default as it would be custom anyways [16:26] I'm not sure I understand the question [16:26] so file a bug against the nginx package suggesting we uodate default configs to fix this out of the box. or complain to nginx upstream. [16:26] if you have to explicitly ask nginx to not swallow errors, the default is to swallow errors [16:26] but if the default is, no fastcgi configured, why care? [16:26] you have to adjust it anyways manually [16:27] ^^ that [16:27] because swallowing errors is a terrible default? [16:27] like [16:27] ubuntu desktop doesn't run the terminal emulator by default [16:27] mort: then complain to nginx upstream [16:27] but if you start the terminal emulator, it has a reasonable set of things like keybinds and color choices [16:27] they chose the defaults [16:27] what is a terminal emulator? [16:27] I didn't say ubuntu picked terrible defaults [16:28] if the default profile for gnome-terminal had black text on a black background, that would have been a terrible default, even though ubuntu doesn't run gnome-terminal by default [16:28] again, fastcgi isn't a default [16:28] neither is running gnome-terminal, you have to ask for it by running the application [16:28] heh? [16:28] what's confusing? [16:28] ya [16:28] normally when you install gnome-terminal, you expect to read it [16:29] if you install nginx you expect a webserver, fastcgi isn't expected and isn't enable [16:29] normally when you enable fpm, you expect errors to not be swallowed [16:29] and the package enabled php? [16:29] or did you? [16:29] just because fpm isn't enabled by default doesn't mean it can't have a terrible default [16:29] if you screwed up the custom edits of the config file, why is that the packages fault? [16:29] I'm not sure you're trying to argue [16:30] swallowing logs by default is a bad default, that's all I'm saying [16:30] I'm just saying just cause you feel that way, doesn't mean it is right [16:30] so you're saying [16:30] just cause php uses fastcgi errors [16:30] there are people out there who argue that swallowing errors by default is good [16:30] most things I know of don't, except php [16:30] who are these people who think errors should be swallowed by default [16:30] you could just configure php to not send errors via fastcgi [16:30] that is also an acceptable solution [16:31] this has literally nothing to do with anything I have said ever [16:31] who are these people who think errors should be swallowed by default [16:31] dunno what your talking about [16:31] I am talking about nginx swallowing errors by default [16:31] I think that is a bad default [16:31] there are people that would configure that a bad thing, as taking errors from fastcgi pollutes your nginx logs with non-nginx things [16:31] you said that's just my opinion, that it's not necessarily right that nginx shouldn't swallow errors by default [16:31] and php should log to it's own file [16:31] who thinks that swallowing errors by default is a good idea [16:32] I just stated it [16:32] it points out a configuration error [16:32] why do you think that errors should be swallowed by default [16:32] HOW you resolve that error depends on you [16:32] who does it help that errors are swallowed by default [16:32] cause it was a config issue [16:32] it helps me? [16:32] the default is to swallow errors [16:32] I just said why twice [16:33] do you not understand? [16:33] I do not understand, no, why errors should be swallowed by default [16:33] now I understand that maybe you think php-fpm should log errors somewhere else than via the fpm socket, that it should put it in a log file or print it to its stderr or something [16:33] that's a perfectly fine config [16:33] but in that case, nginx wouldn't receive errors over the fpm socket, so it wouldn't hurt if nginx logged errors it receives over the fpm socket [16:34] it would hurt [16:34] how [16:34] cause a *dev* would just log to nginx fastcgi [16:34] would see the logs, and go, all is fine [16:34] but now the logs are crossed, if your doing a large scale logging system [16:34] and don't want php logs labeled with nginx [16:35] wouldn't you want to notice that errors end up going over the fpm socket? [16:35] no, that is part of testing, no logs? well something is screwed [16:35] is it better that the developer doesn't see any errors at all and thinks everything is fine? [16:35] but if there is logs, well, testing might pass [16:35] "no logs" usually means things are ok [16:35] no [16:35] yes [16:35] when you log something and don't see it [16:35] you know it's wrong [16:36] but when you don't log something and you don't see it, you don't [16:36] yes, but then that isn't a test is it? [16:36] if you have code which logs warnings if something is wrong, and you don't see warnings, you think everything is fine [16:36] do you know how to make any production systems? [16:36] if your to the point of logging warnings and errors and you didn't test, you already failed [16:36] what kind of question is that [16:37] you think it's better if warnings or errors are swallowed than them going to the wrong place [16:37] that's literally retarded [16:37] you never test your logging systems? [16:37] I'm saying it doesn't matter [16:37] no, you're saying you think it's better if warnings or errors are swallowed than them going to the wrong place [16:37] you have to customize the config anyways, you have to test anyways so [16:37] why [16:37] are you so [16:37] dead set on ignoring errors [16:38] why can't you just accept that a system which swallows errors is worse than a system which doesn't swallow errors [16:38] who said I want to ignore them? [16:38] you did [16:38] your dead set on that [16:38] no [16:38] you think the right default is to ignore errors [16:38] I disagree with that [16:38] I never said that [16:38] yea you did [16:38] you think the right default is to ignore errors, simple as that [16:38] I said, the default doesnt matter cause there are usecases both ways, and fastcgi isn't a default and has to be configured [16:38] so fastcgi logging *SHOULD* also be configured [16:39] you're going in circles [16:39] and just cause you feel different doesn't mean the whole world should change [16:39] just because fastcgi has to be enabled, doesn't mean it doesn't have defaults if you do enable it [16:39] it has the default that errors are swallowebd [16:39] your saying there is NEVER a point to disable it [16:39] but it is disabled, and a reason why [16:39] unless you can overcome that reason with an good statement, it won't change [16:39] what the hell are you talking about [16:39] so stop bitching and make the case for it [16:39] I never said there's never a reason to disable it [16:40] I know [16:40] your ignorant that there is a case [16:40] literally all I'm saying is that swallowing errors is a terrible default [16:40] and you won't change my mind [16:40] I'm as close to objectively correct as you can get on this [16:41] I don't understand why you're *this* defensive over... the idea that swallowing errors by default is a bad default [16:41] patdk-lap? [16:42] ... has probably noticed that i opped up. lets drop this argument and move on [16:42] eh ban me if you want, I'm right about this [16:42] didnt say you are right or wrong [16:43] didn't notice [16:43] but this isnt the place for this argument [16:43] just there is no point [16:43] patdk-lap: may I suggest therapy [16:43] and you can both go into timeout now. [17:32] teward: so, these SSL errors nginx prints (or may print?) on 22.04 were previously reported. maybe by the same user, not sure (the one i talked to then also seemed a bit boneheaded, not wanting to reort a bug). unfortunately i no longer have the actual message. last time it was reported i searched that error message, google web search only returned two or three web pages where it was mentioned to be an issue with nginx on ubuntu 22.04 but [17:32] not with the upstream (nginx.com) packages. i'll see if i can find out more, there was something related on the upstream nginx changelog. [17:33] ah, here we go: SSL_read() failed (SSL: error:0A000126:SSL routines::unexpected eof while reading) while keepalive, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:443 [17:34] https://help.nextcloud.com/t/nginx-letsencrypt-openssl-error-on-new-ubuntu-22-04-unexpected-eof-while-reading/138609 [17:40] https://github.com/openssl/openssl/issues/18866 [17:40] Issue 18866 in openssl/openssl "SSL_read() failed (SSL: error:0A000126:SSL routines::unexpected eof while reading)" [Open] [17:42] i assume (similar error message) this can be similar to https://github.com/openssl/openssl/issues/11381 where kroeckx goes into detail of the situation, environment, SSL handshakes causing this. [17:42] Issue 11381 in openssl/openssl "ssl3_read_n:unexpected eof while reading while keepalive" [Closed] [17:44] i do *not* claim that this erro occurs to nginx on 22.04 in general. it's probably just certain proxy_pass configuration, possibly with streams, causing it. [17:44] (or this users' configuration.) it was also not clear whether they were just unhappy with the error messages or anything actually broke. [17:47] hey, does anybody know how to make systemd forget about a templated systemd unit (eg: ceph-osd@3.service)? I can't seem to get it to stop trying to act on that unit in the more generic targets === ajfriesen69 is now known as ajfriesen6 [18:04] icey[m]: can you disable it? [18:05] and check if there isn't another systemd unit that might be starting it, like a *.socket, or *.timer [18:06] ahasenack: yeah but, for some other independent reasons, things will start it again by doing things like `systemctl list-units --full --all --no-pager -t service` to identify what's on the machine, and poke at it directly from there [18:07] then mask it? [18:07] or find it in /etc/systemd and rm it [18:07] ahasenack: incidentally, it seems like long enough wait after doing a `systemctl stop ceph-osd@3; systemctl disable ceph-osd@3; systemctl reset-failed; systemctl daemon-reload` managed to get systemd to forget about it :-/ [18:07] I think these templated units don't exist until someone actually enables them, with the right aprameter [18:07] there's no file in /etc/systemd for the unit :-/ [18:07] the other place is /run/systemd [18:08] AH HA!!1! thanks ahasenack ! `/run/systemd/system/ceph-osd.target.wants/ceph-osd@3.service` [18:10] "ah, here we go: SSL_read..." <- tomreyn that's what happens when the handshake fails with a backend or a client. Usually the result of a terminated connection or connecting to HTTPS withouut SSL. Those are typically red herrings last I checked though. [18:11] teward[m]: that's what i was thinking, too. [18:13] i'd assume a CRIT level issue if it actually broke NGINX and the libssl implementation [18:13] and also, i see this when talking with the wrong ciphers. [18:13] which is of course dependent on the properly configured OpenSSL settings in nginx and in libssl === SuperL4g is now known as SuperLag [18:21] what they said was that they upgraded from a working 20.04 LTS to a "broken" (but not clear how) 22.04 LTS nginx + php, if I recall correctly. [18:22] but it won't help to try to pull lost info from my brains now. they had better reported a bug if it was one. [18:26] we need a bug report, or, at the very least if they don't want to create a LP account, a mailing list post, or discourse post, with the details [18:29] quod erat dictum [18:44] I may be late to the party, but regarding the EOF ssl error, we do have https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457 [18:44] Launchpad bug 1981457 in nginx (Ubuntu Jammy) "Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option." [Undecided, Triaged] [18:44] it may have been concealed from LP searches earlier today since the kinetic bug is marked as fixed [18:45] hmpf, I searched for that in https://bugs.launchpad.net/ubuntu/+source/nginx [18:45] yeah! [18:45] stupi^W n/m [18:45] also, we did patch the same ssl3 issue earlier this month for jammy's php [18:45] LP search can be very frustrating [18:46] "empty" pinging bryceh who may be interested in the discussion :) [18:50] o/ [18:53] the earlier chat i referred to above was with in #ubuntu with Guest9844, logged at https://irclogs.ubuntu.com/2022/08/21/%23ubuntu.html#t08:07 [19:00] from the last bug comment, looks like we have a reproducer [19:01] at the very least, I counted at least two other persons who said the ppa fixed the problem [19:01] so we can rely on community testing if it comes to that [19:39] which ppa might i ask [19:40] teward: https://launchpad.net/~bryce/+archive/ubuntu/nginx-fix-lp1981457 linked from https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457/comments/4 [19:40] Launchpad bug 1981457 in nginx (Ubuntu Jammy) "Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option." [Undecided, Triaged] [19:40] oh [19:40] the backport [19:41] sarnold: i'm stealing you for security hat stuff now [19:41] see above [19:41] bryceh: see above as well there - if there's an OK from Security, even preliminary, I'd suggest we implement [19:41] unrelated: holy shit i'm losing power [19:41] teward, works for me [19:42] teward: find a power socket, pronto! [19:42] ahasenack: lol. battery backups for critical infra [19:42] so power flickers don't affect my critical infra [19:42] unrelated it's 'torrential downpour' outside. and i was going to go walk to get more coffee. [19:42] oh well guess i'm staying indoors now [19:43] I was already wondering if we should proceed to SRU with user-validation. The patch is pretty straightforward, just reproduction is a challenge. [19:43] bryceh: i would say JFDI [19:43] it's a sensible fix in my opinion [19:44] ooof, no coffee *and* no power?? [19:44] sarnold: well power's flickery [19:44] coffee generates power [19:44] as for coffee i have 'coffee' but not hte good fresh brewed espresso from coffee shops heh [19:44] I see a correlation [19:44] thoug hthey probably lost power so i mean :p [19:44] so, drag over one of your batteries? [19:44] sarnold: already done :P [19:45] :D [19:45] my primary driver's a laptop so i have power if there's flickers [19:45] core infra and network gear connected to the UPSes as well [19:45] so :p [19:45] my netflix got disrupted when my desktop almost died but eh [19:46] sarnold: i just want security's blessing on https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457 [19:46] Launchpad bug 1981457 in nginx (Ubuntu Jammy) "Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option." [Undecided, Triaged] [19:46] just because i' mpicky ;) [19:46] this is a good thing to backport though I think - removes excess cruft from the logs [19:46] (which are red herrings) [19:46] but because it touches SSL opts i wanted an extra opinion :0 [19:47] teward, jealous of your rain, where are you located? [19:48] PA. and this isn't rain, this is a storm [19:48] yeah, I don't want *torrential downpour* but a sprinkle would be nice.. [19:48] aha that's where Oregon's weather got to [19:48] 'rain' is a steady not severe rain that won't explode things or cause power outages or flods [19:48] "hell" is when it is tstorms that futz with power or rains taht make even normal roads floody in spots [19:48] ok so we'll go halvsies [19:49] :D [19:52] bryceh: my apologies for being persistent and annoying looping in Security but any time SSL options get touched I Get 'concerned' heh [19:56] no prob, I'd have uploaded this already but I generally prefer being able to reproduce the issue myself before sending srus off. Too often seen sru bugs languishing waiting due to lack of testers [19:57] but doesn't look like we short be short on testers for this one so maybe fine [19:57] "short be short" *sigh* now who needs coffee [19:58] heheh time to get some caffeine bryceh xD [19:59] bryceh: i'd also like to modify the default config(s) we ship for nginx to add a line to the fastcgi/php default sections to put an option that makes sense in the examples [20:00] ... i'd also like to restructure some of the package but i must clear *that* restructuring with my Debian comaintainers first [20:00] I like sdeziel's reproducer in there; that's way easier than my first thought of "try to curl something real big and kill -9 the thing before it's done" [20:01] teward, is that something that can wait for l-series? Else just send a debdiff or git-ubuntu MP and one of us will pick it up as normal [20:01] yep L series is my target [20:01] i'm not touching Kinetic at the moment [20:01] (err: keys) [20:02] *nod* [20:02] I'm really happy with how cleaned up the nginx ubuntu delta is now for git ubuntu. Hopefully future merges will be very smooth now [20:04] next thing for nginx maybe, I noticed we lack a page for nginx in the server guide... https://ubuntu.com/server/docs [20:05] documenting how to get https up, enable modules, and other common tasks, seems logical to include in the guide. Might be a bit of a project though. === genii_ is now known as genii