=== thegoose is now known as el === JanC_ is now known as JanC === HeadlessHorseman is now known as Unit193 [05:51] While I might dispute the validity of the NMU, someone that is gtk2-rm crazy could sync gparted from experimental. === pieq_ is now known as pieq [08:34] doko: There's a bug in MySQL 8 which seems to have been the same or re-surfaced as I have found an exact same bug un launchpad for 5.5 from 3 years ago. I've had a poke round launchpad but cannot find where to file an updated report or submit a patch. I would like to do it if someone can point me in the right direction. [08:35] Bug is actually with the logrotate script, so probably just a packaging issue and not an issue with MySQL 8.0 itself. (https://bugs.launchpad.net/ubuntu/+source/mysql-5.5/+bug/1558471) [08:35] Launchpad bug 1558471 in mysql-5.5 (Ubuntu) "logrotate config mysql-server fails with error "error: skipping "/var/log/mysql.log"" [Undecided,New] === alan_g is now known as alan_g_ [09:41] I've greated a fresh patch from the code here:- https://code.launchpad.net/~usd-import-team/ubuntu/+source/mysql-8.0/+git/mysql-8.0 and I've tried to look upstream, but debian is even more confusing as they have dropped mysql from stable. :/ Now I'm lost! [09:41] s/greated/created [10:25] Gargoyle: o/ [10:25] Gargoyle: you can send a merge proposal against https://salsa.debian.org/mariadb-team/mysql/tree/mysql-8.0/ubuntu/develif you like [10:25] Or file a bug against https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+filebug [10:49] Unit193: oh I didn't realize that was an NMU. Have you tried the new gparted yet? [10:52] Thanks rbasak. Done (https://salsa.debian.org/mariadb-team/mysql/merge_requests/28) :-) [11:08] Gargoyle: thanks! Just looking at this now. [11:09] Gargoyle: I can reproduce on 19.10 by running "logrotate -f /etc/logrotate.d/mysql-server" but not "logrotate -f /etc/logrotate.conf" which I find puzzling. [11:10] It makes sense to take your merge as it's obviously correct, but in terms of importance for backporting to 19.10, how does it affect users in practice? [11:12] Maybe the base config already does something like this? [11:13] What I'm wondering is that since in the default configuration /var/log/mysql.log isn't created (I think?), then in the default configuration logrotate would never warn like this? [11:13] If so, then we should take the fix for future releases but perhaps there's no need to backport the fix? [11:14] «su root syslog» is in /etc/logrotate.conf [11:19] Aha [11:19] So this isn't a bug after all? [11:19] And yeah, the package default is /var/log/mysql/error.log [11:21] I don't really know much about how logrotate works :) [11:21] Hm, wait, I was looking on a 18.04 box [11:21] In 19.10 it seems to be «su root adm» [12:26] cyphermox: can unity-autopilot be got rid of? it depends src:autopilot, which depends on parts of src:python-qt4 which are dropped in -proposed package [12:33] rbasak: It stops the logrotate service from starting. [12:33] https://paste.ubuntu.com/p/SCf58hcgYM/ [12:34] ^ Relevant lines from "journalctl -b 0" [12:34] Gargoyle: could you give me full steps to reproduce please, on 19.10 presumably? [12:35] Yup. 1 sec === TalkingMuffin is now known as Pici [12:51] rbasak: Hmm. Nope. Apparently I cannot! With my changes removed I can still run "sudo /usr/sbin/logrotate -d /etc/logrotate.conf" without errors. [12:52] rbasak: This would make sense because "syslog" is actually a member of "adm". :-/ [13:04] Gargoyle: OK, so no bug? [13:05] "U2F support in OpenSSH HEAD" https://marc.info/?l=openssh-unix-dev&m=157259802529972&w=2 [13:06] This would be nice for Focal, but I suspect there won't be an upstream release in time. I assume cherry-picking is going to be frowned upon? [13:07] There's a bug somewhere, rbasak. It seems like it's intermittent though:- https://paste.ubuntu.com/p/PX5Zpy4nCp/ [13:07] I'm going to do a bit more poking. Looks like it might be an issue with the post rotate and "sudo systemctl status logrotate.service" was showing me a red herring. [13:10] OK [13:10] rbasak: I wouldn't be comfortable with that because (a) it's a large and complicated set of commits to CP, (b) it'd be easy to make a security-critical mistake, and (c) it wouldn't be unheard of for upstream to catch security-critical errors of their own in this sort of thing before release [13:11] I agree there's no particular assurance that there'll be an upstream release in time; the release schedule isn't regular, but somewhere around every 4-6 months is typical (though some have been a lot quicker). [13:12] cjwatson: I thought as much, thanks [13:13] Interesting that they went for a new key type, which means all servers have to update too. I guess that's another thing to add to my Twisted patch queue [13:13] (i.e. it'll be a while before we can support it in LP) [13:15] I've had basically the full patch set necessary to support Ed25519 in LP ready for a while, but the Twisted patch review cycle grinds pretty slowly :( [13:18] rbasak: It's a race condition. [13:25] rbasak: Not very easy to visualise, but here's a full boot log:- https://termbin.com/zn3c [13:28] rbasak: If you search for "Starting Rotate log files", then then scroll down, "Starting MySQL Community Server" although it's still in the same second is a page and a half later. My guess would be that when mysql's logs do actually need rotating, the post rotate script tries to run "sudo /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs" but mysql is not yet up and running. [13:29] Ah, I see [13:29] So we need to add a check before the mysqladmin call to no-op if mysqld isn't running perhaps? [13:29] Skuggen: ^? [13:30] Seems like something that flush-logs should have an option for :) [13:30] tjaalton: ping. would u mind approving sg3-utils SRU for disco ? it has already being checked/verified/tested in bug 1833618. [13:31] bug 1833618 in sg3-utils (Ubuntu Disco) "MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken (USB pendrive in this case)." [High,In progress] https://launchpad.net/bugs/1833618 [13:31] I would still like to try and create reliable steps to reproduce, but logrotate somehow knows my log files don't need rotating - even if I decompress the last rotated one [13:36] rbasak: Hm, the script runs a mysqladmin ping command just before to verify the server is up [13:36] https://salsa.debian.org/mariadb-team/mysql/blob/mysql-8.0/ubuntu/devel/debian/mysql-server-8.0.mysql-server.logrotate#L16 [13:37] Actually, what exactly does that if check? That the mysqladmin command doesn't output anything to stderr? [13:40] Skuggen: It outputs "mysql is alive". If mysql is alive - and nothing on error. [13:42] Ah, of course, that if is for the failure case. Then the flush-logs command should only be run if the server is up and ready [13:42] So if mysql is not alive, we enter that block and test the outcome of "killall -q -s0 -umysql mysqld;" [13:43] Yeah [13:43] Which has an exit status of 1 when mysql is not running. [13:44] What is signal 0 ? [13:44] rafaeldtinoco: no release on a friday, sorry [13:44] ah ok. was just following what is written in the sru wiki [13:45] maybe we should update it the [13:45] So if mysql isn't running it should return success [13:45] Signal 0 is sort of a dry-run thing, if I understand it correctly [13:45] RikMills: I suppose unity-autopilot could be dropped (at a glance, it looks like only unity wants it); but this is really a question for the desktop team [13:45] i.e. it won't actually kill mysql, but will check if it's there [13:46] RikMills: as long as the tests are still runnable somehow as they were, it would probably be fine to stop shipping them as a package [13:46] So if mysqld is in the startup phase, process running but not ready yet, it makes sense that the logrotate script will fail, I think [13:46] Skuggen: Thats what I was just typing! :D [13:47] So some test to detect that state is what we need :) [13:48] Which is basically what rbasak said :D [13:58] Turned that postrotate script into a stand alone bash script and ran it with a while true and started mysql. Conformed that it hits the "exit 1" during mysql startup. [14:06] Crude but effective! XD (https://paste.ubuntu.com/p/dCj5ZsvxXH/) [14:07] I ran "service mysql start" in another term when that was running ^^ [14:10] hi. are you aware of a problem connecting to DFS shares on bionic that was introduced between 2:4.7.6+dfsg~ubuntu-0ubuntu2.11 and 2:4.7.6+dfsg~ubuntu-0ubuntu2.13? That version has been out for at least a week now (I installed it on oct 24) and I can't see anything on the bug tracker about it. It worked for me on oct 22 and I noticed it only today (hadn't tried mounting that share between oct 22 and today) and going directly to the (only) share [14:10] behind it works. The person responsible for all the file servers says nothing has changed server-side since oct 22, so I'm quite confident that this was introduced with the update. [14:10] So, if any samba maintainers here are unaware of the issue, I'll write up a bug report about it. [14:11] TomyWork: that's not related tohttps://lists.ubuntu.com/archives/ubuntu-server/2019-October/007967.html is it? [14:12] temporally correlated for sure :) [14:12] i don't recognize any error messages [14:13] Please do file a bug, and if you're reporting a regression in stable release, tag it regression-update [14:13] btw is there any way I can downgrade to .11 so i can confirm this issue? [14:13] ...as a regression? [14:18] Yes [14:19] Launchpad will have the old binaries [14:19] Via https://launchpad.net/ubuntu/+source/samba/+publishinghistory [14:19] https://launchpad.net/ubuntu/+source/samba/2:4.7.6+dfsg~ubuntu-0ubuntu2.11 I think? [14:19] found it, thanks [14:19] Then click on amd64 (I assume) [14:20] You will need to work out which packages from that source you have installed (and thus need to downgrade) [14:20] yeah i'll just go through the list on packages.ubuntu.com for the source package [14:22] dpkg -l ctdb libnss-winbind libpam-winbind libparse-pidl-perl libsmbclient libsmbclient-dev libwbclient-dev libwbclient0 python-samba registry-tools samba samba-common samba-common-bin samba-dbg samba-dev samba-dsdb-modules samba-libs samba-testsuite samba-vfs-modules smbclient winbind 2>/dev/null | grep ^ii [14:22] that works :) [14:28] ok, rebooting to be sure it's live [14:34] Skuggen: rbasak: I think I can re-write that postrotate script to check for all 3 possible outcomes (mysql not running, bad debian-sys-maint user, running and connecting ok) with a single check and eliminate the need for the second "if killall.... etc" check which is causing the race condition. [14:36] Gargoyle: that sounds good! I'm working on something else right now so haven't confirmed any of the detail, but I really do appreicate any fixes you want to give us :) [14:36] Should I open a new bug? [14:37] Good question [14:37] I don't mind if you do [14:37] If you just want to send a Salsa merge request that's fine too, though if discussion gets complicated then that would probably best be done in a bug [14:38] And then there's the question of whether the fix needs to be backported to stable Ubuntu releases [14:38] OK. I'll get the patch/MR ready and see how well I can explain it! :-) [14:38] If the conclusion is that it does, then we'll need a bug to exist to track the status of that [14:38] Thanks! [14:40] Gargoyle: If you'd make a pr for that it would be great! [14:42] rafaeldtinoco: oh, you mean approve to -proposed and not -updates? yeah, that can be done [14:42] tjaalton: #) yep. thx ! [14:42] Skuggen: so it turns out my s390x problem is when mysql is built with openssl after all...I properly enabled openssl in 5.7.27 and I'm hitting the same issue [14:43] Skuggen: I'm kind of stuck now [14:43] mdeslaur: is it because openssl tries to use CPU hw acceleration and it's not available on your LPAR / VM? [14:44] mdeslaur: cause that's "new" in 1.1.1 partially [14:44] xnox: how would I find out? [14:45] invoke jfh [14:46] there are command line things to check if all the right CPU functionality is enabled and available, but i don't remember them. [14:46] jfh? [14:48] rbasak, regression confirmed. Mounting works with .11 [14:49] actually let me upgrade again first, to see whether it was just a reboot that was needed :D [14:49] (unlikely since everything else worked) [14:52] rbasak, ok forget about the whole story, I'm back on .13 and the issue isn't there anymore. I dont know what it was. [14:52] it might be an issue, but it's definitely not a regression [14:56] TomyWork: OK. Thank you for investigating anyway - negative results are also good results, and now you know what to do if you do ever spot a regression :) [15:02] yeah, and how to downgrade if one is already reported but not fixed yet :) [15:02] mdeslaur: It's during the build, right? I could try to diagnose it in a ppa [15:03] Skuggen: take the current bionic package, switch to -DWITH_SSL=system, add libssl-dev to build-depends, and it will hang during build as soon as the server is started for the tests [15:04] Skuggen: last thing in the log is "Installing system database..." [15:18] In the current script, it has: "test -x /usr/bin/mysqladmin || exit 0" followed by a comment "# If this fails, check debian.conf" what does that mean? It it trying to say the installation is broken because there is a rote script for a package that is not installed? [15:19] I'm not sure about the comment [15:19] If you're asking about the -x test, that's pretty common in Debian [15:19] It's because a package removal doesn't remove its configuration files [15:20] So that later reinstallation will maintain the previous configuration [15:20] Ahh. ok. [15:20] So configuration that is actually code must check for that condition and handle it gracefully === grumboo is now known as grumble [18:42] vorlon: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1789118 [18:42] Launchpad bug 1789118 in linux (Ubuntu Bionic) "Fails to boot under Xen PV: BUG: unable to handle kernel paging request at edc21fd9" [High,Fix released] [18:42] This bug appears fixed in distro, but (as per comment #20) installer images are broken, so unable to install Ubuntu in xen. [18:47] popey: http://gb.archive.ubuntu.com/ubuntu/dists/bionic/main/installer-i386/current/images/ doesn't have the fix because it's the release pocket. They want http://gb.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-i386/current/images/ [18:47] well, I mean, they shouldn't want anything i386 [18:47] but generally [18:48] Hi rbasak, I'd like to discuss LP #1847924 when you have some minutes [18:48] Launchpad bug 1847924 in mdadm (Ubuntu Focal) "Introduce broken state parsing to mdadm" [Medium,In progress] https://launchpad.net/bugs/1847924 [18:59] Ah. Thank you [20:14] Skuggen: rbasak: This is what I have been able to do with the best of my knowledge as it stands at the moment. Bug = https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1850980, MR = https://salsa.debian.org/mariadb-team/mysql/merge_requests/29 [20:14] Launchpad bug 1850980 in mysql-8.0 (Ubuntu) "mysql-server causes logrotate.service to fail at boot" [Undecided,New] [20:15] I decided to create the bug anyway because there's still an un-addressed edge case.