[05:58] doko: hey, wie gehts? [06:10] maxb, #ubuntu-release seems more appropriate [07:58] cyphermox: hey, remember my network-manager/modem-manager issue? I updated this bug report with some more details: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1716034 [07:58] Launchpad bug 1716034 in network-manager (Ubuntu) "Network manager stops managing Ethernet links after upgrade" [Undecided,Confirmed] [07:59] pitti: hey, since you wrote netplan initially, can you perhaps have a look as well? [08:00] pitti: tl;dr; version is that xenial -> zesty breaks networking because of a debian patch you made there [08:00] pitti: and perhaps you know how this was supposed to be changed on updates [08:10] xnox: Not sure if you noticed, but apt-helper wait-online (a dumb version just running nm-online/systemd-networkd-wait-online when their services are active) landed in artful. Catching up with automatic updates should be more reliable at resume now. SRUs will come in a few weeks if that works out well :) [08:11] the timeout is 30 seconds right now, we can still change that if we want to wait longer for network [08:11] (if you use multiple network management services, the time outs add up, though) [08:14] We might want to patch network manager or docker.io to make this reliable on systems with docker, as described in https://bugzilla.gnome.org/show_bug.cgi?id=787487 [08:14] Gnome bug 787487 in general "Please add "docker[0-9]*" to 85-nm-unmanaged.rules" [Normal,New] [08:14] * more reliable [08:58] juliank, do you really need multiple wait onlines? as the networked-wait-online currently waits for any one interface to become routable. [08:58] juliank, however i think that this thus may be buggy [08:58] juliank, and there are bug reports in artful now that wait-online stuff causes boot hangs and delays as if it is not working correctly. [08:59] xnox: need is a strong word, I just figured it's the best choice. I also moved the network-online dep from the timer to the target, that might improve boot performance. [08:59] * to the service [08:59] this way basic.target does not pull it in anymore [09:00] And we only run systemd-networkd-wait-online if networkd is running; or nm-online if NetworkManager is running [09:00] because it just hung before [09:02] s/before/without the service running/ [09:03] in theory it should already not be impacting boot performance, as basic.target has a mere Wants on timers.target, and no After= [09:03] But if it did for some reason, it should be better now [09:44] zyga-ubuntu: so that's different to bug 1690992 ? I thought that already avoided the disabling on all upgrades [09:45] bug 1690992 in network-manager (Ubuntu Zesty) "fix for bug #1569649 left NetworkManager-wait-online disabled on some systems" [Undecided,Fix released] https://launchpad.net/bugs/1690992 [09:47] pitti: that feels different [09:47] pitti: essentially N-M was igoring all but wifi [09:48] pitti: ethernet was disabled [09:48] pitti: as was my wwan (called "gsm" here) connection [09:48] pitti: this also affects my desktop running artful [09:48] pitti: (both are upgrades from xenial) [09:57] zyga-ubuntu: right, sorry, mistyped the bug #; I followed up with the right one in your's [09:58] zyga-ubuntu: I mean bug 1676547 [09:58] bug 1676547 in network-manager (Ubuntu) "No network connectivity after upgrade from 16.04 to 16.10" [Critical,In progress] https://launchpad.net/bugs/1676547 [10:00] pitti: aha [10:01] pitti: how was it supposed to work? [10:01] pitti: the crippling file is in /usr/lib by default [10:01] pitti: what was to ensure that the counterweight is in /etc on non-server systems? [10:01] zyga-ubuntu: NM should continue to work as-is on updates, and not manage eth on new installs [10:01] zyga-ubuntu: the empty conf file (shadowing the one in /usr) was supposed to be created by the postinst [10:01] but apparently there was some logic error [10:02] pitti: why should NM not manage ethernet? [10:02] pitti: note *on desktops* [10:02] pitti: I don't mind servers [10:02] zyga-ubuntu: because on new installs netplan will do that [10:02] oh, on desktops it should manage everything, yes [10:03] but that's done by a netplan conf snippet which (re-)enables eth for NM [10:03] But amusingly, to remove nplan which is unused, you remove ubuntu-standard too. :P [10:03] pitti: but on desktop netplan is not doing anything for me [10:03] pitti: no default config or anything, how was it supposed to be triggered? [10:04] there should be a default config [10:04] where? [10:04] I don't see one in the package [10:04] the installer should plant an /etc/netplan/ snippet which assingns all ethernets to NM [10:04] /etc/netplan [10:04] pitti: that's on new installs, what about updates? [10:04] mind you, if that is an upgrade, none of this will/should happen [10:04] pitti: is this just a lot of people doing dist-upgrade the old way? [10:04] and then running into a problem? [10:05] on upgrades, NM is simply meant to create the empty "shadowing" of the /usr/lib/ NM config file via postinst [10:05] dist-upgrade should be fine, the logic is in postinst, not update-manager [10:05] the postinst logic was simply not sufficient [10:05] aha [10:05] I see, thank you [10:05] are you still interested in fixing this or ENOTIME anymore? [10:06] I thought netplan was just generating NM config files? [10:06] zyga-ubuntu: cyphermox has already fixed this, or is at it [10:06] juliank: yes, that or networkd config files, depending on the config [10:06] pitti: I see, thank you! [10:06] if dpkg --compare-versions "$2" le-nl "1.2.2-0ubuntu4"; then [10:06] mkdir -p /etc/NetworkManager/conf.d || true [10:06] # for old versions, override the global config with a null config [10:06] touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf [10:06] fi [10:06] juliank: It's kind of backwards though, network-manager ships the config file that disables nm by default, whereas that snippet really should be in nplan package. :P [10:07] yeah, that would work, too - feel free to toss it around :) [10:07] I did, the bug was closed. [10:07] pitti: that's in the postins file, for whatever reason it didn't run [10:07] pitti: thank you again! [10:08] zyga-ubuntu: the reason is that 1.2.6 was uploaded to xenial-updates [10:08] zyga-ubuntu: so the version comparison did not match any more [10:09] so I think it should just be bumped to the version that introduces the file [10:45] xnox: I think boot speed might have improved with apt 1.5~rc2 indeed. The past time it always spent 10s in userspace now it's at 5 seconds [10:46] curiously, total time went from 35s down to 20s [10:46] despite the other times being the same [10:46] A: Startup finished in 1.138s (kernel) + 9.957s (initrd) + 10.063s (userspace) = 35.269s [10:46] B: Startup finished in 1.156s (kernel) + 8.399s (initrd) + 4.632s (userspace) = 20.473s. [10:47] Although I must say that the timer or the service never was in the critical chain [10:48] Let's downgrade and reboot [10:50] and yup, it's back at 10s [10:51] But that's mostly a reporting problem I guess [10:52] If the timer depends on network-online, the boot is not considered complete. [10:52] If you move the dep to the service, the boot is considered complete earlier [10:52] But well, perception matters to [10:52] o [10:52] boot plots: https://people.debian.org/~jak/boot-rc1.svg https://people.debian.org/~jak/boot-rc2.svg [10:55] I mean, unless the timer has elapsed we don't even pull in wait-online stuff during boot anymore === freyes__ is now known as freyes === mnepton is now known as mneptok [14:55] jbicha: hey! I see [14:55] Argh, copy-paste error [14:56] jbicha: hey! I see the evolution-data-server you recently uploaded enables the installed-tests autopkgtests, but those seem to be horribly flaky [14:56] jbicha: they were disabled with the previous versions, now they seem to be failing in artful-proposed for some archs [14:57] When running on my chroot I see they're indeed very unreliable, getting different failures with every run [15:05] jbicha: maybe we should disable them again? [15:13] doko, I tried to debug potemkin-clojure failure, and I failed, do you have some debug/progress to share? [15:14] no [15:15] maybe I'll ask to debian-java if you didn't already [15:29] doko, could you please include https://launchpad.net/ubuntu/+source/automake-1.15/1:1.15.1-2.1ubuntu1 if you have chances to do a new test rebuild? [15:31] So, how do the servers boot with networking? I'm currently dropping the Wants=network-online.target from apt as that has weird implications (and we have our own wait-online helper for network-manager, systemd-networkd, and connman). We still have an After=network.target, so stuff like ifupdown will still run before the service. [15:31] The question is if that's enough, or if there is something else to look out for [15:31] xenial, zesty, and artful [15:35] Well I guess it seems fine [15:41] I'm currently waiting 30s per service, so if someone has connman, NM, and systemd-networkd active, but no connection, it would wait 90s. [15:41] kind of weird [15:42] (but easy) [15:42] LocutusOfBorg: the next test rebuild will be the one once glibc migrates [15:43] so will you include that upload? [15:43] juliank, that might be a question for xnox [15:43] it is not part of artful [15:44] jbicha, are you located in florida? [15:45] juliank, servers default to networkd in artful and up [15:46] juliank, prior to that they defaulted to ifupdown [15:46] Right [15:46] The question was mostly solved anyway because ifupdown's wait-online helper did not exist back then [15:47] cyphermox: ping nplan failures [15:57] jamespage: please could you have a look at http://autopkgtest.ubuntu.com/packages/o/openvswitch/artful/i386 ? [15:57] or is this temporary? [15:58] doko, unping cyphermox [15:58] doko, nplan fails due to bad base cloud-image [15:58] ahh, ok [15:59] doko, and we need to get a new cloud-image which is currently broken due to multiple reasons, all of which are in various stages of being fixed. [15:59] doko: that looks like the transient issues with had some some architectures when I did some previous uploads - the tests just timeout [15:59] ok, retrying [16:01] doko: Earlier today I retried everything; openvswitch is currently running [16:02] ohh, ok. so better continue tomorrow with the analysis [16:02] as you wish - some results will be in already - but I wouldn't retry things again [16:02] colord looks crazy [16:03] valgrind in an autopkgtest? [16:03] ... which fails because of stderr output [16:03] heh [16:05] fitspng is caused by https://sourceware.org/bugzilla/show_bug.cgi?id=16640 [16:05] sourceware.org bug 16640 in string "string/strtok.c: undefined behaviour inconsistent between x86 and other generic code" [Minor,Resolved: fixed] [16:20] cyphermox: https://paste.ubuntu.com/25515376/ looks like the ISO tests are still getting asked for the keyboard-config, can you take another look? === JanC_ is now known as JanC [17:09] tyhicks: please could you have a look at the autopkg test failures triggered by file? [17:12] doko: I saw the emails come in over the weekend but haven't had a chance to look yet this morning [17:12] sbeattie: ^ [17:14] sbeattie: fyi, I restarted a test that was preventing bzr from migrating and now it looks to be ready to migrate [17:15] chrisccoulson: what is our firefox story? I noticed that there were no artful uploads at all [17:15] ricotz: ^^^ [17:18] doko, tyhicks: yeah, I was looking at the file one, debian has a newer version of diffoscope, but I haven't confirmed it addresses the failure. [17:19] doko: I understand that artful firefox is blocked behind N different rustc update failures, a different one for each arch [17:19] sarnold: https://launchpad.net/ubuntu/+source/rustc/1.18.0+dfsg1-4ubuntu1 looks fine === nacc_ is now known as nacc [17:19] it kinda sucks that we care about firefox on more platforms than are tier1 platforms for the rustc team [17:20] doko: that's two releases too old [17:20] ahh, ok [17:20] sarnold, no, it is enough [17:20] doko, hi [17:20] ricotz: oh? I understood 1.19 was needed for the newest firefox, and 1.20 will be needed right quick.. [17:20] anyway, you need to fix the glibc-2.26 triggered build failures [17:21] sarnold, ff57 is a long way to go [17:21] doko, I just need to upload it, but it doesn't work at all on arm at the moment [17:21] ff56 is good with 1.17.0 [17:22] doko, yeah, confince infinity to let is pass from proposed despite the failure [17:22] ? [17:22] it ftbfs [17:23] because upstream only care about tier1 archs [17:23] e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1391126 [17:23] Mozilla bug 1391126 in General "toolkit/components/backgroundhangmonitor/HangStack.cpp:2:10: fatal error: shared-libraries.h: No such file or directory" [Normal,New] [17:25] that's not even the main problem. If you get it to build it doesn't run, because of what looks like a miscompilation [17:28] chrisccoulson, did you forward the rustc fix for llvm4.0 to artful? [17:28] which fix? [17:28] add debian/patches/fix-computeKnownBits-for-ARMISD::CMOV.patch [17:29] is that not there already? [17:29] I am asking you [17:29] doko, ^? [17:30] oh, it's probably not. It's not in debian either [17:30] I just saw that it builds on debian s390x, but not on Ubuntu [17:31] chrisccoulson, same for llvm3.9, I guess [17:32] chrisccoulson, doko, also, could be that ff55 is not happy with gcc7 [17:33] doko, can I get https://github.com/llvm-mirror/llvm/commit/cdc303e5ed4d3110e6f70931775a70bb1de44ed6 in llvm-toolchain-4.0 in artful please? (it's required for rust) [17:33] well, it's required for a future version of rust, that I'm in the process of preparing [17:33] it's already in the 3.9 package [17:34] doko, what package is that specifically that builds in debian, but not in ubuntu? i'm failing to get context from above [17:37] xnox: firefox [17:37] phh [17:37] chrisccoulson, I don't think it is in llvm3.9/artful [17:38] ricotz, it is [17:38] chrisccoulson: if you want to use 3.9, please fix the build. or use 4.0 / 5.0 [17:38] chrisccoulson, ah ok, missed it [17:38] doko, but debian/fedora/etc do not use piles of embedded libraries to do builds like we do. Hence it builds :-( [17:39] especialy nss/nspr [17:39] xnox: so maybe it's time to change that, at least for the architectures we are not interested in? [17:40] doko, i'm not sure anybody cares though =/ cause firefox is out of scope on s390x [17:41] ohh, but then you can't build libreoffice [17:41] * doko hides [17:41] can't you run libreoffice in server mode these days? [17:43] doko, i'm sure you can boot to libreoffice, cause it embeds systemd =) [17:46] powersj: I think that will be fixed in a couple of hours? [17:46] cyphermox: ok thx, I'll sit tight and check results tomorrow morning [17:47] powersj: or it could be that I never uploaded the fix, I got distracted, oops [17:48] powersj: ok, the fix is in [17:48] it's just waiting for the d-i build to finish [17:48] ok thanks for checking! [17:48] once it's done building I'll take the mini.iso to make sure things are good :) [18:13] LocutusOfBorg: yes (in FL) but I'm ok [18:18] sil2100: yes I intend to disable them again since they're still flaky even with a few changes === led_ir23 is now known as led_ir22 [18:42] jbicha: thanks! === ubott2 is now known as ubottu === tdaitx_ is now known as tdaitx === dkessel_ is now known as dkessel === andyrock_ is now known as andyrock === stokachu_ is now known as stokachu === LocutusO- is now known as LocutusOfBorg [23:25] cjwatson, yeah i couldn't get start-stop-daemon to do something sensible under systemd either. i think stuff interferes because there is init.d and systemd unit but i gave up. It should just work, but i'm not sure how wide spread the problem is. And I'm sure the real answer here is to do use "systemd-run" which is equivalent of "run a one-off managed service" [23:26] hm, i wonder if start-stop-daemon should really actually use systemd-run on systemd machines. [23:28] oh god the abstractions upon abstractions .. [23:28] sarnold, it's not over until one embeds src:linux src:systemd src:network-manager! [23:29] * sarnold cries quietly in the corner [23:31] sarnold, at least one no longer needs to embed openssl, as src:linux does that now. [23:31] hahahaha