[11:06] <mort> it seems like nginx 1.18 is incompatible with OpenSSL 3, yet Ubuntu 22.04 ships nginx 1.18 and OpenSSL 3
[11:06] <ikonia> what makes you think that ?
[11:07] <ikonia> what is the depends on the nginx package in terms of ssl 
[11:08] <mort> nginx don't work for me on 22.04, and threads online are suggesting that nginx 1.18 with OpenSSL 3 breaks in exactly the way I'm seeing
[11:08] <ikonia> what's the package depends on nginx 
[11:09] <mort> I don't understand the question
[11:09] <ikonia> the nginx package, will have dependencies and optional dependencies
[11:09] <ikonia> what are the dependencies on SSL 
[11:11] <mort> it seems to depend on libssl >=3
[11:12] <ikonia> I was just looking https://packages.ubuntu.com/jammy/nginx-core
[11:12] <ikonia> so it needs ssl3 alpha 1 according to that depend
[11:12] <ikonia> I'm assuming you're using the alpha 1 package
[11:13] <mort> I'm using whatever Ubuntu installed, which includes whatever nginx depends on
[11:13] <ikonia> do an update and make sure it's the current alpha 1 package, 
[11:15] <mort> I seem to be on openssl 3.0.2-0ubuntu1.6
[11:15] <mort> all packages are up to date
[11:19] <ikonia> if it's still broke - seems like you should check if a bug has been has aready been raised, if so, add your details to the bug
[11:19] <ikonia> if not raise a bu
[11:19] <ikonia> bug
[11:21] <mort> but that's what's so weird, this isn't just a normal bug where some small thing is broken in some edge case
[11:21] <mort> this is "nginx doesn't work in Ubuntu LTS"
[11:21] <ikonia> yes, I heard you
[11:21] <ikonia> so there should already be a bug
[11:21] <ikonia> if nginx cannot serve ssl in 22.04
[11:21] <mort> no, it should not have been released
[11:22] <ikonia> that's a perfectly valid view
[11:22] <ikonia> however, checking if there is a bug and reading the bug may give context
[11:22] <mort> that's what's making me think I must be doing something wrong, nobody in their right mind would release an Ubuntu server LTS which can't serve web pages
[11:22] <ikonia> may give insight
[11:22] <ikonia> well, I'd certainly be checking for bug reports, as it could be a known bug that was released on purpose while waiting for an update etc
[11:23] <mort> I will check out bug reports
[11:24] <mort> there seems to be some nginx conf hack to work around the issue, it might be that for example the nginx package comes with that config option set but I didn't let it overwrite my conf or something
[11:24] <mort> I'll investigate
[11:45] <mort> I'm unable to find any bug reports for nginx
[11:46] <mort> I simply don't understand this
[11:46] <mort> the config patch I talked about was an openssl config, and doesn't work
[12:10] <ikonia> it seems unlikely (not impossible) that such a critical failure would not have been caught in testing, and even if it wasn't, no-one has logged a bug against it
[12:16] <schopin> I don't have anything related to nginx in my openssl 3.0 bugs :/
[16:14] <teward[m]> i know for an absolute fact it works fine with openssl 3.0 and serves SSL fine for all currently supported TLS versions
[16:15] <teward[m]> and I put a TON of effort into testing nginx every cycle
[16:16] <teward[m]> mort: can you pastebin your config you were attempting to use initially and I will run tests on a fresh lts environment/vm
[16:17] <teward[m]> (fyi I am nginx comaintainer in Debian and a huge contributor here in Ubuntu for nginx)
[16:20] <mort> teward[m]: so, what happens seems to be: nginx works but logs critical ssl errors, *and also* something else was broken with my php config which caused nginx not to serve the site it's supposed to
[16:20] <mort> (and php-fpm's log file or journald log or anything else didn't contain the output which I'm sure php has somehow printed somewhere)
[16:21] <mort> it's weird, it was simply a matter of the php code throwing an exception, it should've ended up in some log
[16:21] <teward[m]> are you using fpm? use fastcgi_intercept_errors on;  in the config
[16:22] <teward[m]> otherwise the php errors sometimew just disappear
[16:22] <mort> amazing
[16:22] <teward[m]> nginx then will throw the error message into error.log typically then
[16:22] <mort> it didn't, fwiw
[16:23] <teward[m]> unless you put that config bit i said then no it wont
[16:23] <mort> I don't understand how these things hang together, nor why everything uses error.log files rather than printing to stderr so that it ends up in journald
[16:23] <patdk-lap> the php error log is for fpm errors
[16:23] <patdk-lap> for normal errors, since your using fastcgi, they go over the fastcgi socket
[16:24] <patdk-lap> so your webserver would have to log them
[16:24] <mort> and nginx promptyl ignores them
[16:24] <mort> awesome
[16:24] <mort> that's exactly the right default
[16:24] <teward[m]> and are logged to nginx's log files not stderr
[16:24] <mort> nope
[16:24] <teward[m]> mort no
[16:24] <teward[m]> add this to your php block:
[16:24] <teward[m]> fastcgi_intercept_errors on;
[16:24] <mort> yes otherwise it ignores errors
[16:24] <mort> that's what I'm calling exactly the right default
[16:25] <teward[m]> So now you're just whining then
[16:25] <mort> yes, it's a terrible choice which cost me a whole lot of hours and frustration
[16:25] <mort> a system which ignores errors unless explicitly configured to not ignore errors is broken
[16:25] <patdk-lap> there was a default to make php just work in nginx?
[16:26] <patdk-lap> I always added that myself, so ya, no point in there being a default as it would be custom anyways
[16:26] <mort> I'm not sure I understand the question
[16:26] <teward> so file a bug against the nginx package suggesting we uodate default configs to fix this out of the box.  or complain to nginx upstream.
[16:26] <mort> if you have to explicitly ask nginx to not swallow errors, the default is to swallow errors
[16:26] <patdk-lap> but if the default is, no fastcgi configured, why care?
[16:26] <patdk-lap> you have to adjust it anyways manually
[16:27] <teward> ^^ that
[16:27] <mort> because swallowing errors is a terrible default?
[16:27] <mort> like
[16:27] <mort> ubuntu desktop doesn't run the terminal emulator by default
[16:27] <teward> mort: then complain to nginx upstream
[16:27] <mort> but if you start the terminal emulator, it has a reasonable set of things like keybinds and color choices
[16:27] <teward> they chose the defaults
[16:27] <patdk-lap> what is a terminal emulator?
[16:27] <mort> I didn't say ubuntu picked terrible defaults
[16:28] <mort> if the default profile for gnome-terminal had black text on a black background, that would have been a terrible default, even though ubuntu doesn't run gnome-terminal by default
[16:28] <patdk-lap> again, fastcgi isn't a default
[16:28] <mort> neither is running gnome-terminal, you have to ask for it by running the application
[16:28] <patdk-lap> heh?
[16:28] <mort> what's confusing?
[16:28] <patdk-lap> ya
[16:28] <patdk-lap> normally when you install gnome-terminal, you expect to read it
[16:29] <patdk-lap> if you install nginx you expect a webserver, fastcgi isn't expected and isn't enable
[16:29] <mort> normally when you enable fpm, you expect errors to not be swallowed
[16:29] <patdk-lap> and the package enabled php?
[16:29] <patdk-lap> or did you?
[16:29] <mort> just because fpm isn't enabled by default doesn't mean it can't have a terrible default
[16:29] <patdk-lap> if you screwed up the custom edits of the config file, why is that the packages fault?
[16:29] <mort> I'm not sure you're trying to argue
[16:30] <mort> swallowing logs by default is a bad default, that's all I'm saying
[16:30] <patdk-lap> I'm just saying just cause you feel that way, doesn't mean it is right
[16:30] <mort> so you're saying
[16:30] <patdk-lap> just cause php uses fastcgi errors
[16:30] <mort> there are people out there who argue that swallowing errors by default is good
[16:30] <patdk-lap> most things I know of don't, except php
[16:30] <mort> who are these people who think errors should be swallowed by default
[16:30] <patdk-lap> you could just configure php to not send errors via fastcgi
[16:30] <patdk-lap> that is also an acceptable solution
[16:31] <mort> this has literally nothing to do with anything I have said ever
[16:31] <mort> who are these people who think errors should be swallowed by default
[16:31] <patdk-lap> dunno what your talking about
[16:31] <mort> I am talking about nginx swallowing errors by default
[16:31] <mort> I think that is a bad default
[16:31] <patdk-lap> there are people that would configure that a bad thing, as taking errors from fastcgi pollutes your nginx logs with non-nginx things
[16:31] <mort> you said that's just my opinion, that it's not necessarily right that nginx shouldn't swallow errors by default
[16:31] <patdk-lap> and php should log to it's own file
[16:31] <mort> who thinks that swallowing errors by default is a good idea
[16:32] <patdk-lap> I just stated it
[16:32] <patdk-lap> it points out a configuration error
[16:32] <mort> why do you think that errors should be swallowed by default
[16:32] <patdk-lap> HOW you resolve that error depends on you
[16:32] <mort> who does it help that errors are swallowed by default
[16:32] <patdk-lap> cause it was a config issue
[16:32] <patdk-lap> it helps me?
[16:32] <mort> the default is to swallow errors
[16:32] <patdk-lap> I just said why twice
[16:33] <patdk-lap> do you not understand?
[16:33] <mort> I do not understand, no, why errors should be swallowed by default
[16:33] <mort> now I understand that maybe you think php-fpm should log errors somewhere else than via the fpm socket, that it should put it in a log file or print it to its stderr or something
[16:33] <mort> that's a perfectly fine config
[16:33] <mort> but in that case, nginx wouldn't receive errors over the fpm socket, so it wouldn't hurt if nginx logged errors it receives over the fpm socket
[16:34] <patdk-lap> it would hurt
[16:34] <mort> how
[16:34] <patdk-lap> cause a *dev* would just log to nginx fastcgi
[16:34] <patdk-lap> would see the logs, and go, all is fine
[16:34] <patdk-lap> but now the logs are crossed, if your doing a large scale logging system
[16:34] <patdk-lap> and don't want php logs labeled with nginx
[16:35] <mort> wouldn't you want to notice that errors end up going over the fpm socket?
[16:35] <patdk-lap> no, that is part of testing, no logs? well something is screwed
[16:35] <mort> is it better that the developer doesn't see any errors at all and thinks everything is fine?
[16:35] <patdk-lap> but if there is logs, well, testing might pass
[16:35] <mort> "no logs" usually means things are ok
[16:35] <patdk-lap> no
[16:35] <mort> yes
[16:35] <patdk-lap> when you log something and don't see it
[16:35] <patdk-lap> you know it's wrong
[16:36] <mort> but when you don't log something and you don't see it, you don't
[16:36] <patdk-lap> yes, but then that isn't a test is it?
[16:36] <mort> if you have code which logs warnings if something is wrong, and you don't see warnings, you think everything is fine
[16:36] <patdk-lap> do you know how to make any production systems?
[16:36] <patdk-lap> if your to the point of logging warnings and errors and you didn't test, you already failed
[16:36] <mort> what kind of question is that
[16:37] <mort> you think it's better if warnings or errors are swallowed than them going to the wrong place
[16:37] <mort> that's literally retarded
[16:37] <patdk-lap> you never test your logging systems?
[16:37] <patdk-lap> I'm saying it doesn't matter
[16:37] <mort> no, you're saying you think it's better if warnings or errors are swallowed than them going to the wrong place
[16:37] <patdk-lap> you have to customize the config anyways, you have to test anyways so
[16:37] <mort> why
[16:37] <mort> are you so
[16:37] <mort> dead set on ignoring errors
[16:38] <mort> why can't you just accept that a system which swallows errors is worse than a system which doesn't swallow errors
[16:38] <patdk-lap> who said I want to ignore them?
[16:38] <mort> you did
[16:38] <patdk-lap> your dead set on that
[16:38] <patdk-lap> no
[16:38] <mort> you think the right default is to ignore errors
[16:38] <mort> I disagree with that
[16:38] <patdk-lap> I never said that
[16:38] <mort> yea you did
[16:38] <mort> you think the right default is to ignore errors, simple as that
[16:38] <patdk-lap> I said, the default doesnt matter cause there are usecases both ways, and fastcgi isn't a default and has to be configured
[16:38] <patdk-lap> so fastcgi logging *SHOULD* also be configured
[16:39] <mort> you're going in circles
[16:39] <patdk-lap> and just cause you feel different doesn't mean the whole world should change
[16:39] <mort> just because fastcgi has to be enabled, doesn't mean it doesn't have defaults if you do enable it
[16:39] <mort> it has the default that errors are swallowebd
[16:39] <patdk-lap> your saying there is NEVER a point to disable it
[16:39] <patdk-lap> but it is disabled, and a reason why
[16:39] <patdk-lap> unless you can overcome that reason with an good statement, it won't change
[16:39] <mort> what the hell are you talking about
[16:39] <patdk-lap> so stop bitching and make the case for it
[16:39] <mort> I never said there's never a reason to disable it
[16:40] <patdk-lap> I know
[16:40] <patdk-lap> your ignorant that there is a case
[16:40] <mort> literally all I'm saying is that swallowing errors is a terrible default
[16:40] <mort> and you won't change my mind
[16:40] <mort> I'm as close to objectively correct as you can get on this
[16:41] <mort> I don't understand why you're *this* defensive over... the idea that swallowing errors by default is a bad default
[16:41] <mort> patdk-lap?
[16:42] <teward> ... has probably noticed that i opped up.  lets drop this argument and move on
[16:42] <mort> eh ban me if you want, I'm right about this
[16:42] <teward> didnt say you are right or wrong
[16:43] <patdk-lap> didn't notice
[16:43] <teward> but this isnt the place for this argument
[16:43] <patdk-lap> just there is no point
[16:43] <mort> patdk-lap: may I suggest therapy
[16:43] <teward> and you can both go into timeout now.
[17:32] <tomreyn> teward: so, these SSL errors nginx prints (or may print?) on 22.04 were previously reported. maybe by the same user, not sure (the one i talked to then also seemed a bit boneheaded, not wanting to reort a bug). unfortunately i no longer have the actual message. last time it was reported i searched that error message, google web search only returned two or three web pages where it was mentioned to be an issue with nginx on ubuntu 22.04 but 
[17:32] <tomreyn> not with the upstream (nginx.com) packages. i'll see if i can find out more, there was something related on the upstream nginx changelog.
[17:33] <tomreyn> ah, here we go: SSL_read() failed (SSL: error:0A000126:SSL routines::unexpected eof while reading) while keepalive, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:443
[17:34] <tomreyn> https://help.nextcloud.com/t/nginx-letsencrypt-openssl-error-on-new-ubuntu-22-04-unexpected-eof-while-reading/138609
[17:40] <tomreyn> https://github.com/openssl/openssl/issues/18866
[17:42] <tomreyn> i assume (similar error message) this can be similar to https://github.com/openssl/openssl/issues/11381 where kroeckx goes into detail of the situation, environment, SSL handshakes causing this.
[17:44] <tomreyn> i do *not* claim that this erro occurs to nginx on 22.04 in general. it's probably just certain proxy_pass configuration, possibly with streams, causing it.
[17:44] <tomreyn> (or this users' configuration.) it was also not clear whether they were just unhappy with the error messages or anything actually broke.
[17:47] <icey[m]> hey, does anybody know how to make systemd forget about a templated systemd unit (eg: ceph-osd@3.service)? I can't seem to get it to stop trying to act on that unit in the more generic targets
[18:04] <ahasenack> icey[m]: can you disable it?
[18:05] <ahasenack> and check if there isn't another systemd unit that might be starting it, like a *.socket, or *.timer
[18:06] <icey[m]> ahasenack: yeah but, for some other independent reasons, things will start it again by doing things like `systemctl list-units --full --all --no-pager -t service` to identify what's on the machine, and poke at it directly from there
[18:07] <ahasenack> then mask it?
[18:07] <ahasenack> or find it in /etc/systemd and rm it
[18:07] <icey[m]> ahasenack: incidentally, it seems like long enough wait after doing a `systemctl stop ceph-osd@3; systemctl disable ceph-osd@3; systemctl reset-failed; systemctl daemon-reload` managed to get systemd to forget about it :-/
[18:07] <ahasenack> I think these templated units don't exist until someone actually enables them, with the right aprameter
[18:07] <icey[m]> there's no file in /etc/systemd for the unit :-/
[18:07] <ahasenack> the other place is /run/systemd
[18:08] <icey[m]> AH HA!!1! thanks ahasenack ! `/run/systemd/system/ceph-osd.target.wants/ceph-osd@3.service`
 "ah, here we go: SSL_read..." <- tomreyn that's what happens when the handshake fails with a backend or a client.  Usually the result of a terminated connection or connecting to HTTPS withouut SSL.  Those are typically red herrings last I checked though.
[18:11] <tomreyn> teward[m]: that's what i was thinking, too.
[18:13] <teward[m]> i'd assume a CRIT level issue if it actually broke NGINX and the libssl implementation
[18:13] <teward[m]> and also, i see this when talking with the wrong ciphers.
[18:13] <teward[m]> which is of course dependent on the properly configured OpenSSL settings in nginx and in libssl
[18:21] <tomreyn> what they said was that they upgraded from a working 20.04 LTS to a "broken" (but not clear how) 22.04 LTS nginx + php, if I recall correctly.
[18:22] <tomreyn> but it won't help to try to pull lost info from my brains now. they had better reported a bug if it was one.
[18:26] <ahasenack> we need a bug report, or, at the very least if they don't want to create a LP account, a mailing list post, or discourse post, with the details
[18:29] <tomreyn> quod erat dictum
[18:44] <athos> I may be late to the party, but regarding the EOF ssl error, we do have https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457
[18:44] <athos> it may have been concealed from LP searches earlier today since the kinetic bug is marked as fixed
[18:45] <ahasenack> hmpf, I searched for that in https://bugs.launchpad.net/ubuntu/+source/nginx
[18:45] <ahasenack> yeah!
[18:45] <ahasenack> stupi^W n/m
[18:45] <athos> also, we did patch the same ssl3 issue earlier this month for jammy's php
[18:45] <ahasenack> LP search can be very frustrating
[18:46] <athos> "empty" pinging bryceh who may be interested in the discussion :)
[18:50] <bryceh> o/
[18:53] <tomreyn> the earlier chat i referred to above was with in #ubuntu with Guest9844, logged at https://irclogs.ubuntu.com/2022/08/21/%23ubuntu.html#t08:07
[19:00] <ahasenack> from the last bug comment, looks like we have a reproducer
[19:01] <ahasenack> at the very least, I counted at least two other persons who said the ppa fixed the problem
[19:01] <ahasenack> so we can rely on community testing if it comes to that
[19:39] <teward> which ppa might i ask
[19:40] <ahasenack> teward: https://launchpad.net/~bryce/+archive/ubuntu/nginx-fix-lp1981457 linked from https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457/comments/4
[19:40] <teward> oh
[19:40] <teward> the backport
[19:41] <teward> sarnold: i'm stealing you for security hat stuff now
[19:41] <teward> see above
[19:41] <teward> bryceh: see above as well there - if there's an OK from Security, even preliminary, I'd suggest we implement
[19:41] <teward> unrelated: holy shit i'm losing power
[19:41] <bryceh> teward, works for me
 teward: find a power socket, pronto!
[19:42] <teward> ahasenack: lol.  battery backups for critical infra
[19:42] <teward> so power flickers don't affect my critical infra
[19:42] <teward> unrelated it's 'torrential downpour' outside.  and i was going to go walk to get more coffee.
[19:42] <teward> oh well guess i'm staying indoors now
[19:43] <bryceh> I was already wondering if we should proceed to SRU with user-validation.  The patch is pretty straightforward, just reproduction is a challenge.
[19:43] <teward> bryceh: i would say JFDI
[19:43] <teward> it's a sensible fix in my opinion
[19:44] <sarnold> ooof, no coffee *and* no power??
[19:44] <teward> sarnold: well power's flickery
[19:44] <ahasenack> coffee generates power
[19:44] <teward> as for coffee i have 'coffee' but not hte good fresh brewed espresso from coffee shops heh
[19:44] <ahasenack> I see a correlation
[19:44] <teward> thoug hthey probably lost power so i mean :p
[19:44] <sarnold> so, drag over one of your batteries?
[19:44] <teward> sarnold: already done :P
[19:45] <sarnold> :D
[19:45] <teward> my primary driver's a laptop so i have power if there's flickers
[19:45] <teward> core infra and network gear connected to the UPSes as well
[19:45] <teward> so :p
[19:45] <teward> my netflix got disrupted when my desktop almost died but eh
[19:46] <teward> sarnold: i just want security's blessing on https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457
[19:46] <teward> just because i' mpicky ;)
[19:46] <teward> this is a good thing to backport though I think - removes excess cruft from the logs
[19:46] <teward> (which are red herrings)
[19:46] <teward> but because it touches SSL opts i wanted an extra opinion :0
[19:47] <bryceh> teward, jealous of your rain, where are you located?
[19:48] <teward> PA.  and this isn't rain, this is a storm
[19:48] <sarnold> yeah, I don't want *torrential downpour* but a sprinkle would be nice..
[19:48] <bryceh> aha that's where Oregon's weather got to
[19:48] <teward> 'rain' is a steady not severe rain that won't explode things or cause power outages or flods
[19:48] <teward> "hell" is when it is tstorms that futz with power or rains taht make even normal roads floody in spots
[19:48] <bryceh> ok so we'll go halvsies
[19:49] <sarnold> :D
[19:52] <teward> bryceh: my apologies for being persistent and annoying looping in Security but any time SSL options get touched I Get 'concerned' heh
[19:56] <bryceh> no prob, I'd have uploaded this already but I generally prefer being able to reproduce the issue myself before sending srus off.  Too often seen sru bugs languishing waiting due to lack of testers
[19:57] <bryceh> but doesn't look like we short be short on testers for this one so maybe fine
[19:57] <bryceh> "short be short" *sigh* now who needs coffee
[19:58] <teward> heheh time to get some caffeine bryceh xD
[19:59] <teward> bryceh: i'd also like to modify the default config(s) we ship for nginx to add a line to the fastcgi/php default sections to put an option that makes sense in the examples
[20:00] <teward> ... i'd also like to restructure some of the package but i must clear *that* restructuring with my Debian comaintainers first
[20:00] <sarnold> I like sdeziel's reproducer in there; that's way easier than my first thought of "try to curl something real big and kill -9 the thing before it's done"
[20:01] <bryceh> teward, is that something that can wait for l-series?  Else just send a debdiff or git-ubuntu MP and one of us will pick it up as normal
[20:01] <teward> yep L series is my target
[20:01] <teward> i'm not touching Kinetic at the moment
[20:01] <teward> (err: keys)
[20:02] <bryceh> *nod*
[20:02] <bryceh> I'm really happy with how cleaned up the nginx ubuntu delta is now for git ubuntu.  Hopefully future merges will be very smooth now
[20:04] <bryceh> next thing for nginx maybe, I noticed we lack a page for nginx in the server guide...  https://ubuntu.com/server/docs
[20:05] <bryceh> documenting how to get https up, enable modules, and other common tasks, seems logical to include in the guide.  Might be a bit of a project though.