/srv/irclogs.ubuntu.com/2022/08/29/#ubuntu-server.txt

=== axino` is now known as axino
=== kostkon_ is now known as kostkon
mortit seems like nginx 1.18 is incompatible with OpenSSL 3, yet Ubuntu 22.04 ships nginx 1.18 and OpenSSL 311:06
ikoniawhat makes you think that ?11:06
ikoniawhat is the depends on the nginx package in terms of ssl 11:07
mortnginx don't work for me on 22.04, and threads online are suggesting that nginx 1.18 with OpenSSL 3 breaks in exactly the way I'm seeing11:08
ikoniawhat's the package depends on nginx 11:08
mortI don't understand the question11:09
ikoniathe nginx package, will have dependencies and optional dependencies11:09
ikoniawhat are the dependencies on SSL 11:09
mortit seems to depend on libssl >=311:11
ikoniaI was just looking https://packages.ubuntu.com/jammy/nginx-core11:12
ikoniaso it needs ssl3 alpha 1 according to that depend11:12
ikoniaI'm assuming you're using the alpha 1 package11:12
mortI'm using whatever Ubuntu installed, which includes whatever nginx depends on11:13
ikoniado an update and make sure it's the current alpha 1 package, 11:13
mortI seem to be on openssl 3.0.2-0ubuntu1.611:15
mortall packages are up to date11:15
ikoniaif it's still broke - seems like you should check if a bug has been has aready been raised, if so, add your details to the bug11:19
ikoniaif not raise a bu11:19
ikoniabug11:19
mortbut that's what's so weird, this isn't just a normal bug where some small thing is broken in some edge case11:21
mortthis is "nginx doesn't work in Ubuntu LTS"11:21
ikoniayes, I heard you11:21
ikoniaso there should already be a bug11:21
ikoniaif nginx cannot serve ssl in 22.0411:21
mortno, it should not have been released11:21
ikoniathat's a perfectly valid view11:22
ikoniahowever, checking if there is a bug and reading the bug may give context11:22
mortthat's what's making me think I must be doing something wrong, nobody in their right mind would release an Ubuntu server LTS which can't serve web pages11:22
ikoniamay give insight11:22
ikoniawell, I'd certainly be checking for bug reports, as it could be a known bug that was released on purpose while waiting for an update etc11:22
mortI will check out bug reports11:23
mortthere seems to be some nginx conf hack to work around the issue, it might be that for example the nginx package comes with that config option set but I didn't let it overwrite my conf or something11:24
mortI'll investigate11:24
mortI'm unable to find any bug reports for nginx11:45
mortI simply don't understand this11:46
mortthe config patch I talked about was an openssl config, and doesn't work11:46
ikoniait seems unlikely (not impossible) that such a critical failure would not have been caught in testing, and even if it wasn't, no-one has logged a bug against it12:10
schopinI don't have anything related to nginx in my openssl 3.0 bugs :/12:16
=== lucasmoura__ is now known as lucasmoura
=== arif-ali_ is now known as arif-ali
teward[m]i know for an absolute fact it works fine with openssl 3.0 and serves SSL fine for all currently supported TLS versions16:14
teward[m]and I put a TON of effort into testing nginx every cycle16:15
teward[m]mort: can you pastebin your config you were attempting to use initially and I will run tests on a fresh lts environment/vm16:16
teward[m](fyi I am nginx comaintainer in Debian and a huge contributor here in Ubuntu for nginx)16:17
mortteward[m]: so, what happens seems to be: nginx works but logs critical ssl errors, *and also* something else was broken with my php config which caused nginx not to serve the site it's supposed to16:20
mort(and php-fpm's log file or journald log or anything else didn't contain the output which I'm sure php has somehow printed somewhere)16:20
mortit's weird, it was simply a matter of the php code throwing an exception, it should've ended up in some log16:21
teward[m]are you using fpm? use fastcgi_intercept_errors on;  in the config16:21
teward[m]otherwise the php errors sometimew just disappear16:22
mortamazing16:22
teward[m]nginx then will throw the error message into error.log typically then16:22
mortit didn't, fwiw16:22
teward[m]unless you put that config bit i said then no it wont16:23
mortI don't understand how these things hang together, nor why everything uses error.log files rather than printing to stderr so that it ends up in journald16:23
patdk-lapthe php error log is for fpm errors16:23
patdk-lapfor normal errors, since your using fastcgi, they go over the fastcgi socket16:23
patdk-lapso your webserver would have to log them16:24
=== arif-ali_ is now known as arif-ali
mortand nginx promptyl ignores them16:24
mortawesome16:24
mortthat's exactly the right default16:24
teward[m]and are logged to nginx's log files not stderr16:24
mortnope16:24
teward[m]mort no16:24
teward[m]add this to your php block:16:24
teward[m]fastcgi_intercept_errors on;16:24
mortyes otherwise it ignores errors16:24
mortthat's what I'm calling exactly the right default16:24
teward[m]So now you're just whining then16:25
mortyes, it's a terrible choice which cost me a whole lot of hours and frustration16:25
morta system which ignores errors unless explicitly configured to not ignore errors is broken16:25
patdk-lapthere was a default to make php just work in nginx?16:25
patdk-lapI always added that myself, so ya, no point in there being a default as it would be custom anyways16:26
mortI'm not sure I understand the question16:26
tewardso file a bug against the nginx package suggesting we uodate default configs to fix this out of the box.  or complain to nginx upstream.16:26
mortif you have to explicitly ask nginx to not swallow errors, the default is to swallow errors16:26
patdk-lapbut if the default is, no fastcgi configured, why care?16:26
patdk-lapyou have to adjust it anyways manually16:26
teward^^ that16:27
mortbecause swallowing errors is a terrible default?16:27
mortlike16:27
mortubuntu desktop doesn't run the terminal emulator by default16:27
tewardmort: then complain to nginx upstream16:27
mortbut if you start the terminal emulator, it has a reasonable set of things like keybinds and color choices16:27
tewardthey chose the defaults16:27
patdk-lapwhat is a terminal emulator?16:27
mortI didn't say ubuntu picked terrible defaults16:27
mortif the default profile for gnome-terminal had black text on a black background, that would have been a terrible default, even though ubuntu doesn't run gnome-terminal by default16:28
patdk-lapagain, fastcgi isn't a default16:28
mortneither is running gnome-terminal, you have to ask for it by running the application16:28
patdk-lapheh?16:28
mortwhat's confusing?16:28
patdk-lapya16:28
patdk-lapnormally when you install gnome-terminal, you expect to read it16:28
patdk-lapif you install nginx you expect a webserver, fastcgi isn't expected and isn't enable16:29
mortnormally when you enable fpm, you expect errors to not be swallowed16:29
patdk-lapand the package enabled php?16:29
patdk-lapor did you?16:29
mortjust because fpm isn't enabled by default doesn't mean it can't have a terrible default16:29
patdk-lapif you screwed up the custom edits of the config file, why is that the packages fault?16:29
mortI'm not sure you're trying to argue16:29
mortswallowing logs by default is a bad default, that's all I'm saying16:30
patdk-lapI'm just saying just cause you feel that way, doesn't mean it is right16:30
mortso you're saying16:30
patdk-lapjust cause php uses fastcgi errors16:30
mortthere are people out there who argue that swallowing errors by default is good16:30
patdk-lapmost things I know of don't, except php16:30
mortwho are these people who think errors should be swallowed by default16:30
patdk-lapyou could just configure php to not send errors via fastcgi16:30
patdk-lapthat is also an acceptable solution16:30
mortthis has literally nothing to do with anything I have said ever16:31
mortwho are these people who think errors should be swallowed by default16:31
patdk-lapdunno what your talking about16:31
mortI am talking about nginx swallowing errors by default16:31
mortI think that is a bad default16:31
patdk-lapthere are people that would configure that a bad thing, as taking errors from fastcgi pollutes your nginx logs with non-nginx things16:31
mortyou said that's just my opinion, that it's not necessarily right that nginx shouldn't swallow errors by default16:31
patdk-lapand php should log to it's own file16:31
mortwho thinks that swallowing errors by default is a good idea16:31
patdk-lapI just stated it16:32
patdk-lapit points out a configuration error16:32
mortwhy do you think that errors should be swallowed by default16:32
patdk-lapHOW you resolve that error depends on you16:32
mortwho does it help that errors are swallowed by default16:32
patdk-lapcause it was a config issue16:32
patdk-lapit helps me?16:32
mortthe default is to swallow errors16:32
patdk-lapI just said why twice16:32
patdk-lapdo you not understand?16:33
mortI do not understand, no, why errors should be swallowed by default16:33
mortnow I understand that maybe you think php-fpm should log errors somewhere else than via the fpm socket, that it should put it in a log file or print it to its stderr or something16:33
mortthat's a perfectly fine config16:33
mortbut in that case, nginx wouldn't receive errors over the fpm socket, so it wouldn't hurt if nginx logged errors it receives over the fpm socket16:33
patdk-lapit would hurt16:34
morthow16:34
patdk-lapcause a *dev* would just log to nginx fastcgi16:34
patdk-lapwould see the logs, and go, all is fine16:34
patdk-lapbut now the logs are crossed, if your doing a large scale logging system16:34
patdk-lapand don't want php logs labeled with nginx16:34
mortwouldn't you want to notice that errors end up going over the fpm socket?16:35
patdk-lapno, that is part of testing, no logs? well something is screwed16:35
mortis it better that the developer doesn't see any errors at all and thinks everything is fine?16:35
patdk-lapbut if there is logs, well, testing might pass16:35
mort"no logs" usually means things are ok16:35
patdk-lapno16:35
mortyes16:35
patdk-lapwhen you log something and don't see it16:35
patdk-lapyou know it's wrong16:35
mortbut when you don't log something and you don't see it, you don't16:36
patdk-lapyes, but then that isn't a test is it?16:36
mortif you have code which logs warnings if something is wrong, and you don't see warnings, you think everything is fine16:36
patdk-lapdo you know how to make any production systems?16:36
patdk-lapif your to the point of logging warnings and errors and you didn't test, you already failed16:36
mortwhat kind of question is that16:36
mortyou think it's better if warnings or errors are swallowed than them going to the wrong place16:37
mortthat's literally retarded16:37
patdk-lapyou never test your logging systems?16:37
patdk-lapI'm saying it doesn't matter16:37
mortno, you're saying you think it's better if warnings or errors are swallowed than them going to the wrong place16:37
patdk-lapyou have to customize the config anyways, you have to test anyways so16:37
mortwhy16:37
mortare you so16:37
mortdead set on ignoring errors16:37
mortwhy can't you just accept that a system which swallows errors is worse than a system which doesn't swallow errors16:38
patdk-lapwho said I want to ignore them?16:38
mortyou did16:38
patdk-lapyour dead set on that16:38
patdk-lapno16:38
mortyou think the right default is to ignore errors16:38
mortI disagree with that16:38
patdk-lapI never said that16:38
mortyea you did16:38
mortyou think the right default is to ignore errors, simple as that16:38
patdk-lapI said, the default doesnt matter cause there are usecases both ways, and fastcgi isn't a default and has to be configured16:38
patdk-lapso fastcgi logging *SHOULD* also be configured16:38
mortyou're going in circles16:39
patdk-lapand just cause you feel different doesn't mean the whole world should change16:39
mortjust because fastcgi has to be enabled, doesn't mean it doesn't have defaults if you do enable it16:39
mortit has the default that errors are swallowebd16:39
patdk-lapyour saying there is NEVER a point to disable it16:39
patdk-lapbut it is disabled, and a reason why16:39
patdk-lapunless you can overcome that reason with an good statement, it won't change16:39
mortwhat the hell are you talking about16:39
patdk-lapso stop bitching and make the case for it16:39
mortI never said there's never a reason to disable it16:39
patdk-lapI know16:40
patdk-lapyour ignorant that there is a case16:40
mortliterally all I'm saying is that swallowing errors is a terrible default16:40
mortand you won't change my mind16:40
mortI'm as close to objectively correct as you can get on this16:40
mortI don't understand why you're *this* defensive over... the idea that swallowing errors by default is a bad default16:41
mortpatdk-lap?16:41
teward... has probably noticed that i opped up.  lets drop this argument and move on16:42
morteh ban me if you want, I'm right about this16:42
tewarddidnt say you are right or wrong16:42
patdk-lapdidn't notice16:43
tewardbut this isnt the place for this argument16:43
patdk-lapjust there is no point16:43
mortpatdk-lap: may I suggest therapy16:43
tewardand you can both go into timeout now.16:43
tomreynteward: so, these SSL errors nginx prints (or may print?) on 22.04 were previously reported. maybe by the same user, not sure (the one i talked to then also seemed a bit boneheaded, not wanting to reort a bug). unfortunately i no longer have the actual message. last time it was reported i searched that error message, google web search only returned two or three web pages where it was mentioned to be an issue with nginx on ubuntu 22.04 but 17:32
tomreynnot with the upstream (nginx.com) packages. i'll see if i can find out more, there was something related on the upstream nginx changelog.17:32
tomreynah, here we go: SSL_read() failed (SSL: error:0A000126:SSL routines::unexpected eof while reading) while keepalive, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:44317:33
tomreynhttps://help.nextcloud.com/t/nginx-letsencrypt-openssl-error-on-new-ubuntu-22-04-unexpected-eof-while-reading/13860917:34
tomreynhttps://github.com/openssl/openssl/issues/1886617:40
ubottuIssue 18866 in openssl/openssl "SSL_read() failed (SSL: error:0A000126:SSL routines::unexpected eof while reading)" [Open]17:40
tomreyni assume (similar error message) this can be similar to https://github.com/openssl/openssl/issues/11381 where kroeckx goes into detail of the situation, environment, SSL handshakes causing this.17:42
ubottuIssue 11381 in openssl/openssl "ssl3_read_n:unexpected eof while reading while keepalive" [Closed]17:42
tomreyni do *not* claim that this erro occurs to nginx on 22.04 in general. it's probably just certain proxy_pass configuration, possibly with streams, causing it.17:44
tomreyn(or this users' configuration.) it was also not clear whether they were just unhappy with the error messages or anything actually broke.17:44
icey[m]hey, does anybody know how to make systemd forget about a templated systemd unit (eg: ceph-osd@3.service)? I can't seem to get it to stop trying to act on that unit in the more generic targets17:47
=== ajfriesen69 is now known as ajfriesen6
ahasenackicey[m]: can you disable it?18:04
ahasenackand check if there isn't another systemd unit that might be starting it, like a *.socket, or *.timer18:05
icey[m]ahasenack: yeah but, for some other independent reasons, things will start it again by doing things like `systemctl list-units --full --all --no-pager -t service` to identify what's on the machine, and poke at it directly from there18:06
ahasenackthen mask it?18:07
ahasenackor find it in /etc/systemd and rm it18:07
icey[m]ahasenack: incidentally, it seems like long enough wait after doing a `systemctl stop ceph-osd@3; systemctl disable ceph-osd@3; systemctl reset-failed; systemctl daemon-reload` managed to get systemd to forget about it :-/18:07
ahasenackI think these templated units don't exist until someone actually enables them, with the right aprameter18:07
icey[m]there's no file in /etc/systemd for the unit :-/18:07
ahasenackthe other place is /run/systemd18:07
icey[m]AH HA!!1! thanks ahasenack ! `/run/systemd/system/ceph-osd.target.wants/ceph-osd@3.service`18:08
teward[m]<tomreyn> "ah, here we go: SSL_read..." <- tomreyn that's what happens when the handshake fails with a backend or a client.  Usually the result of a terminated connection or connecting to HTTPS withouut SSL.  Those are typically red herrings last I checked though.18:10
tomreynteward[m]: that's what i was thinking, too.18:11
teward[m]i'd assume a CRIT level issue if it actually broke NGINX and the libssl implementation18:13
teward[m]and also, i see this when talking with the wrong ciphers.18:13
teward[m]which is of course dependent on the properly configured OpenSSL settings in nginx and in libssl18:13
=== SuperL4g is now known as SuperLag
tomreynwhat they said was that they upgraded from a working 20.04 LTS to a "broken" (but not clear how) 22.04 LTS nginx + php, if I recall correctly.18:21
tomreynbut it won't help to try to pull lost info from my brains now. they had better reported a bug if it was one.18:22
ahasenackwe need a bug report, or, at the very least if they don't want to create a LP account, a mailing list post, or discourse post, with the details18:26
tomreynquod erat dictum18:29
athosI may be late to the party, but regarding the EOF ssl error, we do have https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/198145718:44
ubottuLaunchpad bug 1981457 in nginx (Ubuntu Jammy) "Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option." [Undecided, Triaged]18:44
athosit may have been concealed from LP searches earlier today since the kinetic bug is marked as fixed18:44
ahasenackhmpf, I searched for that in https://bugs.launchpad.net/ubuntu/+source/nginx18:45
ahasenackyeah!18:45
ahasenackstupi^W n/m18:45
athosalso, we did patch the same ssl3 issue earlier this month for jammy's php18:45
ahasenackLP search can be very frustrating18:45
athos"empty" pinging bryceh who may be interested in the discussion :)18:46
bryceho/18:50
tomreynthe earlier chat i referred to above was with in #ubuntu with Guest9844, logged at https://irclogs.ubuntu.com/2022/08/21/%23ubuntu.html#t08:0718:53
ahasenackfrom the last bug comment, looks like we have a reproducer19:00
ahasenackat the very least, I counted at least two other persons who said the ppa fixed the problem19:01
ahasenackso we can rely on community testing if it comes to that19:01
tewardwhich ppa might i ask19:39
ahasenackteward: https://launchpad.net/~bryce/+archive/ubuntu/nginx-fix-lp1981457 linked from https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1981457/comments/419:40
ubottuLaunchpad bug 1981457 in nginx (Ubuntu Jammy) "Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option." [Undecided, Triaged]19:40
tewardoh19:40
tewardthe backport19:40
tewardsarnold: i'm stealing you for security hat stuff now19:41
tewardsee above19:41
tewardbryceh: see above as well there - if there's an OK from Security, even preliminary, I'd suggest we implement19:41
tewardunrelated: holy shit i'm losing power19:41
brycehteward, works for me19:41
ahasenack<unrelated> teward: find a power socket, pronto!19:42
tewardahasenack: lol.  battery backups for critical infra19:42
tewardso power flickers don't affect my critical infra19:42
tewardunrelated it's 'torrential downpour' outside.  and i was going to go walk to get more coffee.19:42
tewardoh well guess i'm staying indoors now19:42
brycehI was already wondering if we should proceed to SRU with user-validation.  The patch is pretty straightforward, just reproduction is a challenge.19:43
tewardbryceh: i would say JFDI19:43
tewardit's a sensible fix in my opinion19:43
sarnoldooof, no coffee *and* no power??19:44
tewardsarnold: well power's flickery19:44
ahasenackcoffee generates power19:44
tewardas for coffee i have 'coffee' but not hte good fresh brewed espresso from coffee shops heh19:44
ahasenackI see a correlation19:44
tewardthoug hthey probably lost power so i mean :p19:44
sarnoldso, drag over one of your batteries?19:44
tewardsarnold: already done :P19:44
sarnold:D19:45
tewardmy primary driver's a laptop so i have power if there's flickers19:45
tewardcore infra and network gear connected to the UPSes as well19:45
tewardso :p19:45
tewardmy netflix got disrupted when my desktop almost died but eh19:45
tewardsarnold: i just want security's blessing on https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/198145719:46
ubottuLaunchpad bug 1981457 in nginx (Ubuntu Jammy) "Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option." [Undecided, Triaged]19:46
tewardjust because i' mpicky ;)19:46
tewardthis is a good thing to backport though I think - removes excess cruft from the logs19:46
teward(which are red herrings)19:46
tewardbut because it touches SSL opts i wanted an extra opinion :019:46
brycehteward, jealous of your rain, where are you located?19:47
tewardPA.  and this isn't rain, this is a storm19:48
sarnoldyeah, I don't want *torrential downpour* but a sprinkle would be nice..19:48
brycehaha that's where Oregon's weather got to19:48
teward'rain' is a steady not severe rain that won't explode things or cause power outages or flods19:48
teward"hell" is when it is tstorms that futz with power or rains taht make even normal roads floody in spots19:48
brycehok so we'll go halvsies19:48
sarnold:D19:49
tewardbryceh: my apologies for being persistent and annoying looping in Security but any time SSL options get touched I Get 'concerned' heh19:52
brycehno prob, I'd have uploaded this already but I generally prefer being able to reproduce the issue myself before sending srus off.  Too often seen sru bugs languishing waiting due to lack of testers19:56
brycehbut doesn't look like we short be short on testers for this one so maybe fine19:57
bryceh"short be short" *sigh* now who needs coffee19:57
tewardheheh time to get some caffeine bryceh xD19:58
tewardbryceh: i'd also like to modify the default config(s) we ship for nginx to add a line to the fastcgi/php default sections to put an option that makes sense in the examples19:59
teward... i'd also like to restructure some of the package but i must clear *that* restructuring with my Debian comaintainers first20:00
sarnoldI like sdeziel's reproducer in there; that's way easier than my first thought of "try to curl something real big and kill -9 the thing before it's done"20:00
brycehteward, is that something that can wait for l-series?  Else just send a debdiff or git-ubuntu MP and one of us will pick it up as normal20:01
tewardyep L series is my target20:01
tewardi'm not touching Kinetic at the moment20:01
teward(err: keys)20:01
bryceh*nod*20:02
brycehI'm really happy with how cleaned up the nginx ubuntu delta is now for git ubuntu.  Hopefully future merges will be very smooth now20:02
brycehnext thing for nginx maybe, I noticed we lack a page for nginx in the server guide...  https://ubuntu.com/server/docs20:04
brycehdocumenting how to get https up, enable modules, and other common tasks, seems logical to include in the guide.  Might be a bit of a project though.20:05
=== genii_ is now known as genii

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!