[00:01] -queuebot:#ubuntu-release- New source: u-boot-nezha (jammy-proposed/primary) [20210911-0ubuntu1] [01:03] -queuebot:#ubuntu-release- New source: golang-1.18 (jammy-proposed/primary) [1.18~beta1] [02:09] jawn-smith: version number should be 1.18~beta1-0ubuntu1 [02:13] mwhudson: thanks for the heads up [02:20] -queuebot:#ubuntu-release- New source: golang-1.18 (jammy-proposed/primary) [1.18~beta1-0ubuntu1] [03:55] chiluk: i did, will reply today [03:56] @tjaalton Cool thanks no rush. It really looks like we should just resync with debian, but I wanted to make sure you didn't know something I didn't. [04:01] chiluk: the server still has issues, races setting it up and some bind segfaults with the config, to start. i'll make sid client only so that it's backportable to bullseye(?), and keep server in exp for now [04:05] oh that's not what I wanted to hear. [04:05] what about syncing the client with debian then for jammy? [04:05] iow I wouldn't sync the current version [04:06] yeah, once it's done [04:07] Cool sounds good. I'd really like to get the server and client updated for jammy, as we have intentions of using it "eventually". If you can point me at some bug reports, I might find some actual "work" time to put some effort in. [04:15] i did file one on debian just to prevent it from getting to testing and giving false hopes [04:15] setup fails at some point thinking pki-ca isn't up [04:15] but not always [04:16] tjaalton: https://www.freeipa.org/page/Downloads freeipa seems to think it's functional on debian as well. Perhaps we should file some tickets there as well. [04:17] hm right, it links to the server pkg [04:22] i've not filed upstream bugs unless there's a known portability issue with a patch [04:22] but for the website..sure [04:24] oh and client setup fails on sid, with sasl2 bullt on sid ("no mechs left"). using and older deb it works... [04:26] that's 1000152 [05:11] -queuebot:#ubuntu-release- New: rejected golang-1.18 [source] (jammy-proposed) [1.18~beta1] [05:20] -queuebot:#ubuntu-release- New binary: libayatana-common [amd64] (jammy-proposed/universe) [0.9.6-1] (no packageset) [05:21] -queuebot:#ubuntu-release- New: accepted libayatana-common [amd64] (jammy-proposed) [0.9.6-1] [05:21] -queuebot:#ubuntu-release- New binary: libayatana-common [s390x] (jammy-proposed/universe) [0.9.6-1] (no packageset) [05:21] -queuebot:#ubuntu-release- New binary: libayatana-common [ppc64el] (jammy-proposed/universe) [0.9.6-1] (no packageset) [05:24] -queuebot:#ubuntu-release- New binary: libayatana-common [arm64] (jammy-proposed/universe) [0.9.6-1] (no packageset) [05:24] -queuebot:#ubuntu-release- New binary: libayatana-common [armhf] (jammy-proposed/universe) [0.9.6-1] (no packageset) [05:25] -queuebot:#ubuntu-release- New: accepted libayatana-common [arm64] (jammy-proposed) [0.9.6-1] [05:25] -queuebot:#ubuntu-release- New: accepted libayatana-common [s390x] (jammy-proposed) [0.9.6-1] [05:25] -queuebot:#ubuntu-release- New: accepted libayatana-common [armhf] (jammy-proposed) [0.9.6-1] [05:25] -queuebot:#ubuntu-release- New: accepted libayatana-common [ppc64el] (jammy-proposed) [0.9.6-1] === WrathOfAchilles is now known as Kuraokami [08:59] vorlon: Sorry, so one of the remotes certainly did not respond to SSH even. I rebooted all lxd remotes, the one not reachable via SSH needed a hard reset from openstack. And rebooting the armhf worker/controller, but it hang in systemd-tmpfiles-setup during boot, might trash it and replace it [09:13] Bunch of IRQ exceptions prevented 3 nodes from coming back up again, hard reset them [09:43] please nobody run dnspython [09:44] Thing fails its test suite but it get retried because autopkgtest thinks it's a temporary failure [09:45] how many people retried this on arm64, despite the initial test request still being in the queue? [09:45] * armhf [09:46] it's just killing workers left and right :D [09:49] I think we should probably drop 2.1.0 from proposed, it seems borked and unable to pass tests and holds up everything else [09:51] juliank: dnspython 2.1.0 seems fine on debianci [09:51] doesn't matter [09:51] it has internet there [09:54] looks like we need to revisit LP: #1892832 [09:54] Launchpad bug 1892832 in dnspython (Ubuntu) "dnspython 2.0.0-1 fails IPv6 autopkgtests" [High, Fix Released] https://launchpad.net/bugs/1892832 [09:54] coreycb: ^ you sync'd that [09:54] ginggs: here you can see the log that does not make it out as it gets retried https://paste.ubuntu.com/p/8ccthByCtp/ [09:56] I'm trying to make this a failure, but I don't seem to have success, or it did not restart the worker after pulling the version with the fix attempt [09:58] is it the string 'Temporary failure in name resolution' that causes it to retry? i recall something about that [09:59] ginggs: yup [09:59] ginggs: So I try https://git.launchpad.net/autopkgtest-cloud/commit/?id=6b4b78d2ff780f64cabea1f1c467239b568676b5 but I'm not sure it helps [09:59] Commit 6b4b78d in autopkgtest-cloud "dns.exception.Timeout is a real failure for dnspython autopkgtest-cloud-worker-126" [10:05] juliank: shouldn't dnspython go in OK_PKG_STRINGS, like systemd? line 136 [10:05] ginggs: I don't know, systemd is in both of them [10:05] it's odd [10:08] well OK_PKG_STRINGS are exceptions to TEMPORARY_TEST_FAIL_STRINGS, so i think we should try [10:08] 'dnspython': ['Temporary failure in name resolution'] [10:08] in OK_PKG_STRINGS [10:08] sounds right-ish to me [10:09] o/ laney! [10:09] heya ginggs [10:10] ginggs: done in charm revision 127 [10:11] juliank: thanks! [10:17] ginggs: I think both yours and coreycb retry failed hard in a different place now, so hooray, it's gone from the qeue again. [10:20] The whole exit with 10 if told to restart business is annoying me [10:20] I wonder if it would be nicer to just exec ourselves again [10:20] It kills the workers, and then restarts them after like a 5 min timeout which is a bad way to apply code updates [10:23] Move the control loop out of the workers, just have each worker live for one test request and then exit [10:25] oh no, I did that thing, s/just // [10:26] laney: doesn't help if you need to change the control loop, or the inner loop where it holds on to the queue item [10:26] the while ...: run; if broken: sleep(5 minutes) [10:27] it decouples the two parts and you can change them independently of each other [10:32] i guess i'm describing an operator aren't i [11:18] laney: it will get weird to monitor [11:19] laney: One thing I want to do is move the amqp queue length reporting to run on the rabbitmq server itself/use the admin interface, this way, it doesn't go down to 0 if the server gets under load and blocks consumers :D [11:19] arguably I should have both metrics, and if they disagree strongly, it should alert about rabbitmq failure :D [11:29] -queuebot:#ubuntu-release- Unapproved: rejected ubuntu-dev-tools [source] (hirsute-backports) [0.185~bpo21.04.1] [11:41] iirc it relies on that weird locking thing we did, if you just ask rmq it might be wrong sometimes [11:45] -queuebot:#ubuntu-release- Unapproved: accepted ubuntu-dev-tools [source] (impish-backports) [0.187~bpo21.10.1] [11:46] -queuebot:#ubuntu-release- Unapproved: accepted ubuntu-dev-tools [source] (hirsute-backports) [0.187~bpo21.04.1] [12:27] -queuebot:#ubuntu-release- Unapproved: rejected ubuntu-dev-tools [source] (focal-backports) [0.187~bpo11+1] [13:39] rbasak: you rejected #1942699 but there have been several updates since, and the packages have also been re-uploaded, however no updates came. now telegram-desktop is quite broken in focal… can we ask for a new review? [13:40] LP#1942699 [13:40] Launchpad bug 1942699 in telegram-desktop (Ubuntu Focal) "SRU: Update Telegram Desktop to 3.1.1" [High, In Progress] https://launchpad.net/bugs/1942699 [14:11] -queuebot:#ubuntu-release- Unapproved: accepted debhelper [source] (focal-backports) [13.5.2ubuntu1~bpo20.04.1] [14:54] -queuebot:#ubuntu-release- Unapproved: accepted dwz [source] (bionic-backports) [0.13-5~bpo18.04.1] [15:24] ginggs: juliank: I'll look into disabling the online tests for dnspython [15:26] coreycb: thanks! [15:28] -queuebot:#ubuntu-release- Unapproved: accepted debhelper [source] (bionic-backports) [13.5.2ubuntu1~bpo18.04.1] [15:30] -queuebot:#ubuntu-release- New binary: debhelper [amd64] (bionic-backports/main) [13.5.2ubuntu1~bpo18.04.1] (core) [15:46] removed grub 2.06 block-proposed, should migrate with next britney run [16:02] -queuebot:#ubuntu-release- New: accepted debhelper [amd64] (bionic-backports) [13.5.2ubuntu1~bpo18.04.1] [16:10] -queuebot:#ubuntu-release- Unapproved: ubuntu-dev-tools (focal-backports/universe) [0.185~bpo20.04.1 => 0.187~bpo20.04.1] (no packageset) [17:33] -queuebot:#ubuntu-release- Unapproved: accepted ubuntu-dev-tools [source] (focal-backports) [0.187~bpo20.04.1] [17:35] -queuebot:#ubuntu-release- Unapproved: accepted ubuntu-dev-tools [source] (bionic-backports) [0.187~bpo18.04.1] [20:04] -queuebot:#ubuntu-release- Unapproved: dpkg (impish-proposed/main) [1.20.9ubuntu2 => 1.20.9ubuntu2.1] (core, i386-whitelist) [20:06] -queuebot:#ubuntu-release- Unapproved: dpkg (focal-proposed/main) [1.19.7ubuntu3 => 1.19.7ubuntu3.1] (core, i386-whitelist) [21:27] vorlon: seems like the rebooted armhf workers are super stable, I wish throughput was a tad higher at the moment, but I guess it's running long tests right now [21:28] vorlon: We should add a metric for "average job duration" or "average time run so far" for running jobs [21:28] or max + median [21:28] and yes, currently running so we know if tasks get stuck [21:29] well [21:29] need to scale time running so far by the maximum time allotted [21:30] From the web worker, we can figure out moving average of past run times, and then do analysis of current run time vs expected runtime [21:31] then we can add a metric "tests outside 10% of expected runtime" [21:31] or test runtime outside [median, median+stddev] [21:32] probably +2stddev or +3stddev, otherwise it'll alert for something like 29% of test runs.. [21:32] yeah [21:34] sarnold: I was never good at stochastics [21:34] statistics? [21:34] I don't know [21:36] juliank: heh, yeah, statistics was some of my most frustrating courses; it felt, more than most, that it was just a pile of formulas handed down "that seemed to work okay". I think I've taken three stats courses, read several stats textbooks, and *still* don't feel like I've got a decent grasp of it all, hehe [21:37] sarnold: i had an intro course I passed [21:37] sarnold: It was better than my algebra course which I failed in the first test and then barely passed in a 2nd attempt [21:38] I do love algebra, but boy, I guess I just couldn't compete with the math students as a CS one [21:39] all other math courses were um special ones for stupid cs and physics people [21:39] heh, I skipped a math course my first semester of college and it was just so *weird* to not have math, that I added a math major along the way [21:41] i did an entire maths degree and managed to mostly avoid stats entirely [21:41] well done [21:56] * LocutusOfBorg remembers some scary moments in his CS degree [21:58] I did the first year of a maths degree before switching to pure CS, and avoided stats as hard as I could [21:58] partly because the probability supervisor was terrifying, mind [21:59] saw her serving behind the bar at a beer festival like 20 years later and instinctively went to a different bar in case she could give me a withering lecture about not having done all the problems on the sheet [22:01] juliank: thanks for getting it sorted! do you know why there were no useful logs in the journal when the units were stopping? [22:01] wow :) I'm glad to say that was not my experience :) hehe [22:01] cjwatson: lol [22:01] vorlon: Unfortunately not, I did not see anything so just followed my turning it off and back on again script :D [22:02] vorlon: I know 1 node was completely overworked and not reachable anymore, and 3 failed to reboot due to IRQ exceptions (needed hard reset), so not sure what happened there [22:03] But the logging generally is not always the most decent one :) [22:03] Arguably by the time I looked all those worker nodes were disabled already, so I guess I was not seeing any more failing [22:06] I wonder how full /tmp was before reboot and if that was playing a part [22:08] btw this remembers me: "There are three kinds of lies: lies, damned lies, and statistics'" [22:16] how about a lie-down? [22:18] cjwatson: probability is not stats though? ;-p [22:18] the probability lecturer was reasonably good i think, i can't remember who my supervisor was though [22:19] (cjwatson and i were in the same year at the same uni) [22:28] fun :D [22:43] I can't believe this is the first time I've heard that [22:49] i don't know if colin knew who i was, i only knew who he was because he was always late to lectures :) [22:52] :D [23:18] -queuebot:#ubuntu-release- New binary: ddcutil [amd64] (jammy-proposed/universe) [1.2.1-1] (no packageset) [23:19] -queuebot:#ubuntu-release- New binary: gr-fosphor [amd64] (jammy-proposed/universe) [3.9~0.974ab2f-1] (no packageset) [23:19] -queuebot:#ubuntu-release- New binary: vdr-plugin-xineliboutput [amd64] (jammy-proposed/universe) [2.2.0+git20211212-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: ddcutil [ppc64el] (jammy-proposed/universe) [1.2.1-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-aagcloudwatcher-ng [amd64] (jammy-proposed/universe) [1.6-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-armadillo-platypus [amd64] (jammy-proposed/universe) [1.0-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: ddcutil [s390x] (jammy-proposed/universe) [1.2.1-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-avalon [amd64] (jammy-proposed/universe) [1.12-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-aok [amd64] (jammy-proposed/universe) [1.1-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-astrolink4 [amd64] (jammy-proposed/universe) [0.1-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: libsbig [amd64] (jammy-proposed/multiverse) [4.9.9-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-astromechfoc [amd64] (jammy-proposed/universe) [0.2-1] (no packageset) [23:21] -queuebot:#ubuntu-release- New binary: indi-beefocus [amd64] (jammy-proposed/universe) [0.1-1] (no packageset) [23:22] -queuebot:#ubuntu-release- New binary: libfilezilla [amd64] (jammy-proposed/universe) [0.34.2-2] (no packageset) [23:22] -queuebot:#ubuntu-release- New binary: gr-satellites [amd64] (jammy-proposed/universe) [4.4.0-1] (no packageset) [23:22] -queuebot:#ubuntu-release- New binary: libayatana-appindicator [amd64] (jammy-proposed/main) [0.5.90-1] (desktop-core) [23:22] -queuebot:#ubuntu-release- New binary: ddcutil [armhf] (jammy-proposed/universe) [1.2.1-1] (no packageset) [23:24] -queuebot:#ubuntu-release- New binary: gr-fosphor [s390x] (jammy-proposed/universe) [3.9~0.974ab2f-1] (no packageset) [23:24] -queuebot:#ubuntu-release- New binary: vdr-plugin-xineliboutput [ppc64el] (jammy-proposed/universe) [2.2.0+git20211212-1] (no packageset) [23:24] -queuebot:#ubuntu-release- New binary: ddcutil [arm64] (jammy-proposed/universe) [1.2.1-1] (no packageset) [23:24] -queuebot:#ubuntu-release- New binary: indi-aagcloudwatcher-ng [ppc64el] (jammy-proposed/universe) [1.6-1] (no packageset) [23:24] -queuebot:#ubuntu-release- New binary: indi-armadillo-platypus [ppc64el] (jammy-proposed/universe) [1.0-1] (no packageset) [23:24] -queuebot:#ubuntu-release- New binary: gr-fosphor [ppc64el] (jammy-proposed/universe) [3.9~0.974ab2f-1] (no packageset) [23:25] -queuebot:#ubuntu-release- New binary: indi-aok [ppc64el] (jammy-proposed/universe) [1.1-1] (no packageset) [23:25] -queuebot:#ubuntu-release- New binary: gr-satellites [ppc64el] (jammy-proposed/universe) [4.4.0-1] (no packageset) [23:25] -queuebot:#ubuntu-release- New binary: indi-astromechfoc [ppc64el] (jammy-proposed/universe) [0.2-1] (no packageset) [23:25] -queuebot:#ubuntu-release- New binary: indi-beefocus [ppc64el] (jammy-proposed/universe) [0.1-1] (no packageset) [23:25] -queuebot:#ubuntu-release- New binary: indi-astrolink4 [ppc64el] (jammy-proposed/universe) [0.1-1] (no packageset) [23:26] -queuebot:#ubuntu-release- New binary: libsbig [ppc64el] (jammy-proposed/multiverse) [4.9.9-1] (no packageset) [23:26] -queuebot:#ubuntu-release- New binary: indi-avalon [ppc64el] (jammy-proposed/universe) [1.12-1] (no packageset) [23:26] -queuebot:#ubuntu-release- New binary: gmsh [amd64] (jammy-proposed/universe) [4.8.4+ds2-1] (no packageset) [23:26] -queuebot:#ubuntu-release- New binary: gr-fosphor [armhf] (jammy-proposed/universe) [3.9~0.974ab2f-1] (no packageset) [23:27] -queuebot:#ubuntu-release- New binary: indi-aagcloudwatcher-ng [s390x] (jammy-proposed/universe) [1.6-1] (no packageset) [23:28] -queuebot:#ubuntu-release- New binary: vdr-plugin-xineliboutput [s390x] (jammy-proposed/universe) [2.2.0+git20211212-1] (no packageset) [23:28] -queuebot:#ubuntu-release- New binary: indi-aok [s390x] (jammy-proposed/universe) [1.1-1] (no packageset) [23:29] -queuebot:#ubuntu-release- New binary: gr-fosphor [arm64] (jammy-proposed/universe) [3.9~0.974ab2f-1] (no packageset) [23:30] -queuebot:#ubuntu-release- New binary: indi-armadillo-platypus [s390x] (jammy-proposed/universe) [1.0-1] (no packageset) [23:30] -queuebot:#ubuntu-release- New binary: gr-satellites [arm64] (jammy-proposed/universe) [4.4.0-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: indi-astrolink4 [s390x] (jammy-proposed/universe) [0.1-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: indi-avalon [s390x] (jammy-proposed/universe) [1.12-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: libsbig [s390x] (jammy-proposed/multiverse) [4.9.9-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: gr-satellites [armhf] (jammy-proposed/universe) [4.4.0-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: indi-beefocus [s390x] (jammy-proposed/universe) [0.1-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: indi-astromechfoc [s390x] (jammy-proposed/universe) [0.2-1] (no packageset) [23:31] -queuebot:#ubuntu-release- New binary: meshlab [amd64] (jammy-proposed/universe) [2020.09+dfsg1-2] (no packageset) [23:36] -queuebot:#ubuntu-release- New binary: gmsh [s390x] (jammy-proposed/universe) [4.8.4+ds2-1] (no packageset) [23:36] -queuebot:#ubuntu-release- New binary: indi-beefocus [riscv64] (jammy-proposed/universe) [0.1-1] (no packageset) [23:36] -queuebot:#ubuntu-release- New binary: indi-astromechfoc [riscv64] (jammy-proposed/universe) [0.2-1] (no packageset) [23:37] -queuebot:#ubuntu-release- New binary: indi-aagcloudwatcher-ng [arm64] (jammy-proposed/universe) [1.6-1] (no packageset) [23:37] -queuebot:#ubuntu-release- New binary: indi-armadillo-platypus [riscv64] (jammy-proposed/universe) [1.0-1] (no packageset) [23:37] -queuebot:#ubuntu-release- New binary: indi-aok [armhf] (jammy-proposed/universe) [1.1-1] (no packageset) [23:38] -queuebot:#ubuntu-release- New binary: indi-aagcloudwatcher-ng [armhf] (jammy-proposed/universe) [1.6-1] (no packageset) [23:38] -queuebot:#ubuntu-release- New binary: vdr-plugin-xineliboutput [armhf] (jammy-proposed/universe) [2.2.0+git20211212-1] (no packageset) [23:38] -queuebot:#ubuntu-release- New binary: indi-aok [arm64] (jammy-proposed/universe) [1.1-1] (no packageset) [23:38] -queuebot:#ubuntu-release- New binary: vdr-plugin-xineliboutput [arm64] (jammy-proposed/universe) [2.2.0+git20211212-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: indi-armadillo-platypus [arm64] (jammy-proposed/universe) [1.0-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: indi-astrolink4 [arm64] (jammy-proposed/universe) [0.1-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: indi-astromechfoc [arm64] (jammy-proposed/universe) [0.2-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: libsbig [riscv64] (jammy-proposed/multiverse) [4.9.9-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: indi-armadillo-platypus [armhf] (jammy-proposed/universe) [1.0-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: indi-astromechfoc [armhf] (jammy-proposed/universe) [0.2-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: indi-astrolink4 [armhf] (jammy-proposed/universe) [0.1-1] (no packageset) [23:39] -queuebot:#ubuntu-release- New binary: meshlab [s390x] (jammy-proposed/universe) [2020.09+dfsg1-2] (no packageset) [23:40] -queuebot:#ubuntu-release- New binary: indi-aagcloudwatcher-ng [riscv64] (jammy-proposed/universe) [1.6-1] (no packageset) [23:40] -queuebot:#ubuntu-release- New binary: indi-avalon [arm64] (jammy-proposed/universe) [1.12-1] (no packageset) [23:40] -queuebot:#ubuntu-release- New binary: indi-avalon [riscv64] (jammy-proposed/universe) [1.12-1] (no packageset) [23:40] -queuebot:#ubuntu-release- New binary: indi-aok [riscv64] (jammy-proposed/universe) [1.1-1] (no packageset) [23:40] -queuebot:#ubuntu-release- New binary: indi-avalon [armhf] (jammy-proposed/universe) [1.12-1] (no packageset) [23:41] -queuebot:#ubuntu-release- New binary: ddcutil [riscv64] (jammy-proposed/universe) [1.2.1-1] (no packageset) [23:41] -queuebot:#ubuntu-release- New binary: indi-beefocus [arm64] (jammy-proposed/universe) [0.1-1] (no packageset) [23:41] -queuebot:#ubuntu-release- New binary: libsbig [armhf] (jammy-proposed/multiverse) [4.9.9-1] (no packageset) [23:41] -queuebot:#ubuntu-release- New binary: indi-astrolink4 [riscv64] (jammy-proposed/universe) [0.1-1] (no packageset) [23:41] -queuebot:#ubuntu-release- New binary: indi-beefocus [armhf] (jammy-proposed/universe) [0.1-1] (no packageset) [23:53] -queuebot:#ubuntu-release- New binary: vdr-plugin-xineliboutput [riscv64] (jammy-proposed/universe) [2.2.0+git20211212-1] (no packageset)