[03:21] @find ubuntu 13 [03:21] ojeq: Error: "find" is not a valid command. [05:48] Good morning [07:20] morning [07:49] morning! [07:51] hey larsu! [07:51] hi didrocks! ça va? [07:52] ça va bien, et toi ? [07:53] bien aussi, merci [07:53] * larsu dives into css again today [07:54] good luck! [07:55] * didrocks dives into why machine-id isn't recreated on boot by systemd [07:58] it can't even do simple things like this? We're doomed. systemd will be the end of linux. And choice! [07:59] ahah :) [07:59] larsu: seriously, I wonder why it fails, / is mounted rw, the perms on /etc is fine… [08:00] and if you run a simple wrapper later on, it will create the file without any issue [08:00] * didrocks takes the open() parameters one after another to see why fd < 0 [08:01] weird. well, poettering is this ---> way [08:01] (at least in my irc client :P) [08:02] larsu: I want to debug a little bit more beforehand [08:02] larsu: but if you can get him comment on pitti and I's thread on the upstream ML, that would be appreciated :) [08:02] bonjour didrocks et larsu ! [08:02] hi pitti! [08:03] bonjour pitti! [08:03] didrocks: I can't do more than you, ping him :) [08:03] didrocks: he usually comes online later, though [08:03] larsu: where is the German connection? :) [08:03] simple deal -- send one beer, get one answer :) [08:04] didrocks: ah, having fun sprinkling lots of debug statements into this code and rebooting your VM? [08:05] one of the few cases where there's *such* a good excuse for printf() debugging :) [08:05] pitti: indeed, started yesterday night, so I know why it doesn't create it in the code, and the tmpfs behavior is expected [08:05] printf debugging is the best, I don't know why people complain about it all the time [08:06] pitti: however, the all case is that open("/etc/machine-id", O_RDWR|O_CREAT|O_CLOEXEC|O_NOCTTY, 0444); return a negative fd [08:06] cause* [08:06] didrocks: what's errno? [08:06] pitti: EROFS [08:06] I opened /proc/mounts just before, / is rw [08:06] oh, heh [08:07] /etc is populated with the right perms [08:07] and if you use the wrapper script later on, all is fine, the file is created as expected [08:07] didrocks: but indeed that makes little sense -- it can apparently update the file if it exists as a 0-byte file [08:07] (after the boot proceeded) [08:07] (hence our current workaround of truncating instead of rm) [08:07] pitti: right, there is a fallback [08:07] then, it tests if the file exists [08:07] if the file exists and is empty, -> mount tmpfs [08:08] and generate + write into it [08:08] didrocks: tmpfs to where? [08:08] /etc/machine-id [08:08] (thanks to the flag writable = false; in the code) [08:09] didrocks: oh, heh; but that's still wrong for our purpose indeed, as we actually want a permanent ID [08:10] pitti: I guess it's working actually, because, it's also writing it as the dbus-id [08:10] and then, for generating, it's picking up the dbus-id if it's coherent :) [08:10] so your workaround kind of works [08:10] but kind of ugly as well :p [08:11] pitti: in case you didn't notice, but most of the machine-id that are generated I guess is thanks to systemd postinst script which is using the wrapper (calling exactly the same code) to stamp /etc/machine-id the first time [08:12] still, I don't see why I'm getting this errno at boot, while all seems fine on the FS… [08:12] didrocks: yes, I know that one; but that won't get re-applied after image build and cleaning /etc [08:12] right [08:13] didrocks: it's not apparmor that early during boot yet, right? (and we don't confine init anyway) [08:13] didrocks: so does the remount rw happen before? maybe that just takes a bit of time? [08:13] I didn't look at that, I didn't see anything in dmesg when getting the errorno [08:13] didrocks: did you try with putting an all-healing sleep(1) before the open()? [08:13] pitti: /proc/mounts should reflect the current state, right? [08:13] didrocks: yeah, it should [08:13] ah, I did use printf, not went to the sleep-hack yet [08:14] will try that [08:14] race condition debugger's best friend :) [08:14] heh ;) [08:14] * pitti bows to your new systemd hacker [08:14] ahah, the "printfer + sleeper" hacker :p [08:14] didrocks, pitti: yep we aren't applying apparmor policy that early or to init [08:15] yeah, dmesg was too quiet to blame apparmor :) [08:15] thanks for confirming [08:15] yeah, that wouldn't make sense either [08:15] for now my best suspicion is some kind of race between making r/w and open, although I would have assumed the remount to be synchronous [08:15] pitti: http://paste.ubuntu.com/9076436/ [08:16] see the second part, "without machine-id" [08:16] (I stripped some parts) [08:17] [ 2.739054] systemd[1]: Started Remount Root and Kernel File Systems. [08:17] didrocks: curious that this comes after the machine-id bits [08:17] yeah, I wondered about that one [08:17] systemd[1]: rootfs / rootfs rw [08:17] but look at /proc/mounts reports before [08:17] yeah [08:17] but that comes before (but only 0.06 s) [08:17] this is just before the open() [08:17] didrocks: oh wait, check line 47 [08:18] argh [08:18] ok, making sense [08:18] clearly didn't spot that one yesterday night [08:18] (when I added this debug) [08:18] didrocks: keep that pastebin, makes an excellent artifact for reporting the bug [08:18] so, there is another mount on / on top of the current one, which is ro? [08:19] didrocks: maybe / is still the initramfs rootfs, I'm not sure [08:19] oh, that early… [08:19] although that would be weird, AFAIK it's the initramfs itself which does the pivot_root [08:19] right [08:19] systemd doesn't run in initramfs [08:19] and systemd isn't in the initramfs yet [08:19] (I heard there is some plan) [08:19] oh, that channel is busy this morning [08:20] bonjour seb128 [08:20] seb128: yeah, welcome to #ubuntu-init-system-hackers :) [08:20] pitti: ok, I think bug report time then? or should I try to look where this mounted / is coming from? [08:21] didrocks: so the remounting is systemd-remount-fs.service, and thus happens out of process [08:21] pitti, wie gehts? [08:21] ExecStart=/lib/systemd/systemd-remount-fs [08:21] didrocks: yeah, I guess so, then you can discuss how the code should get reordered to generate machine-id on a r/w root [08:21] ah, indeed [08:21] yeah [08:21] seb128: gut, danke! und Dir? [08:21] pitti: let me add a sleep, just to confirm it's racy [08:22] didrocks: according to our current theory, no amount of sleep should help [08:22] pitti: you don't think this target was started before? [08:22] didrocks: if it does help, we are back to "async remounting r/w" [08:22] pitti, auch gut, danke [08:23] didrocks: nah, it first initializes, then starts units [08:23] didrocks: but actually, the "Booting up is supported only when" indicates that it might not even be considered a bug [08:23] you're right, of course, [ 2.326811] systemd[1]: Found ordering cycle on local-fs.target/start [08:23] the target is afterwards [08:24] didrocks: i. e. you run into case 3), while our workaround is case 2) [08:24] pitti: yeah, I wonder however how this work with the empty /etc then [08:24] didrocks: can't hurt to ask on the ML [08:24] pitti, didrocks, do you guys need systemd tester yet? or is that enough on the "known issue list" that you don't need more feedback yet? [08:25] seb128: both actually [08:25] pitti: starting a second thread today you mean? :) [08:25] seb128: we have a good deal of stuff to work on, but that by far doesn't mean that we know about every boot race or breakage with customizations [08:25] pitti, let me rephrase [08:25] didrocks: the other was yesterday :) [08:26] yeah, I didn't phrase it correctly :) [08:26] ok, let's do it [08:26] if I opt in, am I going to run into 10 known gotcha and waste half a day dealing with known issues? [08:26] e.g that wouldn't be useful work neither for me nor for you [08:26] seb128: ah -- no, it really should boot and work normally [08:26] seb128: I guess you won't get blocked (hopefully) on anything [08:26] or is that supposed to be "everything works, if you have an issue it's interesting and we should debug it together" [08:27] seb128: unless you use things like root on NFS or server-ish packages which only have an upstart job [08:27] seb128: you can even try one boot without turning the switch on definitively [08:27] seb128: if your desktop doesn't boot, or hardware doesn't work etc, please do tell me [08:27] k [08:27] seb128: as that's absolutely expected to work [08:27] do I need to install special packages or just systemd? [08:27] seb128: don't install anything; for the first try, just boot with init=/bin/systemd [08:27] g'morn [08:28] hey willcooke [08:28] seb128: if you want to do that more permanently, you can install systemd-sysv, but it works just fine on the kernel cmd line [08:28] hey willcooke [08:28] hey wik [08:28] hey willcooke [08:29] morning willcooke [08:33] seb128: so go ahead, do it, and be amazed of the Lennart mugshot on the framebuffer during boot! [08:34] lol [08:34] didrocks: ah, he thinks I was joking! [08:34] cute [08:34] thanks, as much I would like to throw darts in my laptop's screen, I sort of need the screen :p [08:35] oh, and no pets within 50 m! [08:35] pitti: "he"? [08:35] happyaron, [08:35] Dépaquetage de libfcitx-qt5-0:i386 (0.1.2-2ubuntu2) ... [08:35] dpkg: erreur de traitement de l'archive /var/cache/apt/archives/libfcitx-qt5-0_0.1.2-2ubuntu2_i386.deb (--unpack) : [08:35] tentative de remplacement de « /usr/lib/i386-linux-gnu/libfcitx-qt5.so.0.1 », qui appartient aussi au paquet fcitx-libs-qt5:i386 0.1.2-2build1 [08:35] happyaron, known issue? [08:36] didrocks: read the two lines above [08:36] ahah :) [08:36] nope, I "though" it was handled, but could be wrong [08:37] happyaron, just happened with today's vivid updates [08:37] brb, restarting [08:40] * didrocks grrrr at thunderbird freezing again [08:41] haven't received that update (by not using main archive server [08:45] pitti: sent, let's see how it goes, I hope my explanations are clear enough, feel free to annotate [08:45] didrocks: ack, thanks [08:45] yw [08:46] didrocks: sounds clear enough to me, thanks! [08:46] great ;) === alf is now known as alf_ [08:55] didrocks, pitti, if I've a process "/bin/systemd splash" running it's that systemd is in use? [08:55] well in use as pid1 [08:55] seb128: yes; you can call "systemctl" to see if it works, too [08:56] pitti, initctl works, but I guess that does magic to redirect to systemd jobs? [08:56] seb128: then, can you try systemctl status as well and report the "State:" line? [08:56] State: degraded [08:56] seb128: no, it doesn't; if "sudo initctl" works, then you have upstart running [08:56] seb128: but I believe you are looking at the session upstart [08:56] pitti, indeed [08:57] k, systemd is in use then [08:57] seb128: systemctl --failed ? [08:57] ok, systemctl should have red (failed) lines, what are they? [08:57] why is "Degraded"§? [08:57] seb128: (without the ?) [08:57] ● ifup@eth1.service loaded failed failed ifup for eth1 [08:57] ● ifup@virbr0.service loaded failed failed ifup for virbr0 [08:57] ● ifup@wlan0.service loaded failed failed ifup for wlan0 [08:57] ahah :) [08:57] pitti: see, I'm not alone! :) [08:57] sudo systemctl show ifup@eth1.service [08:57] is that fixed with the update from this morning? [08:57] didrocks: oh, you figure it's that same bug? [08:58] I'm pretty sure [08:58] seb128: no /run/network directory? [08:58] seb128: most likely, yes [08:58] pitti, http://paste.ubuntu.com/9091049/ [08:58] seb128: do you have ls -l /etc/rc*.d/*networking* [08:58] didrocks, no such dir [08:58] seb128: sorry, my bad -- I mean sudo systemctl status ifup@eth1.service [08:58] seb128: ok, same bug then -- I figure you don't have any sysvinit links to networking [08:59] pitti, http://paste.ubuntu.com/9091050/ [08:59] thus /etc/init.d/networking doesn't get started [08:59] lrwxrwxrwx 1 root root 20 août 11 10:19 /etc/rc0.d/K07networking -> ../init.d/networking [08:59] right, as I expected [08:59] seb128: [ Didier Roche ] [08:59] * debian/ifup@.service: add a ConditionPath on /run/network, to avoid [08:59] failing the unit if /etc/init.d/networking is disabled. (Closes: #769528) [08:59] seb128: so, that failure is fixed with -6ubuntu1 [08:59] yw ;) [08:59] didrocks, is that fixed with the update from this morning? [08:59] seb128: the more interesting question is: what disabled networking in the first place? [08:59] yeah, I read the changelog on -changes [08:59] so yeah ;) [08:59] pitti, not me [09:00] seb128: you don't care much, as you use NM [09:00] but on a server that would be wrong [09:00] right [09:00] pitti: yeah, same for me, I didn't remember to have done that [09:00] I suspect that this is some kind of upgrade failure [09:00] yeah [09:00] maybe ifupdown or initscripts do something funky [09:00] or it's a deskop-team cabale :) [09:00] lrwxrwxrwx 1 root root 20 Okt 5 17:59 /etc/rc0.d/K06networking -> ../init.d/networking [09:00] lrwxrwxrwx 1 root root 20 Okt 5 17:59 /etc/rc6.d/K06networking -> ../init.d/networking [09:00] lrwxrwxrwx 1 root root 20 Nov 17 11:00 /etc/rcS.d/S09networking -> ../init.d/networking [09:00] I have this [09:01] i. e. rcS.d. [09:01] I don't have the rcS.d one [09:01] seb128: ok, so that's interesting, thanks for pointing out; I was writing off didrocks' as local breakage [09:01] seb128: anyway, it's entirely harmless for now [09:01] k [09:01] seb128: (the "degraded") [09:01] well, I'm going to wait for the update to be available and reboot [09:02] seb128: desktop, networking, bluetooth etc. all as it should be? [09:02] pitti, yeah, I didn't notice a visible difference so far [09:02] oh, no [09:02] no bluetooth! [09:03] no indicator and the settings list no adapter found [09:03] wth? [09:03] seb128: you probably disabled it in sysetm-settings [09:03] disabled what? the indicator? [09:03] seb128: upstart scripts have a bug to never actually remember the status across reboots, although they try [09:03] seb128: bluetooth entirely [09:03] seb128: yeah, indicator bug -- it doesn't appear if you have BT soft-killed [09:03] no, unity-control-center sees no hardware [09:03] if you enable it in g-c-c, you should get the indicator [09:03] how [09:03] well I didn't kill it [09:03] oh* [09:03] right [09:04] g-c-c thinks there is no hardware [09:04] seb128: output of "rfkill list"? [09:04] 1: dell-bluetooth: Bluetooth [09:04] Soft blocked: yes [09:04] Hard blocked: no [09:04] there you go [09:04] that's the thing that the inidicator should look at [09:04] shrug [09:04] seb128: you can enable it in g-c-c [09:04] how did I do that? [09:04] no I can't [09:04] hm, I can [09:04] g-c-c says I've no hardware [09:05] hum, it's acting weird [09:05] seb128: "rfkill unblock 1" [09:05] if I toggle the switch it just goes back to disabled [09:05] that worked [09:05] seb128: does it display the indicator then? [09:05] seb128: there is a bug I noticed between the indicator and g-c-c [09:05] didrocks, no, but I disabled that, I don't need the indicator [09:06] seb128: right, that's what g-c-c is supposed to do; it does that here, even if it goes back to "disabled" [09:06] if you turn it on in g-c-c, it goes back like 80% to disable [09:06] good day systemd fans [09:06] if you use the indicator, it updates its status [09:06] hey Laney! [09:06] yeah, the indicator grays out if you disabled BT in the current session, but it doesn't appear at all if you boot with BT hard or soft killed [09:06] didrocks, pitti, is any of you looking to the "go back to disable"? [09:06] seb128: I don't think it's systemd, I got it with upstart [09:06] is that bug filed? [09:06] didrocks, I didn't say it was, but you seem to know about it [09:06] I was thinking it was blueman + indicator at the time [09:06] didrocks, so I was curious if it was filed/being debugged [09:06] no, the difference with systemd is merely that saving state across reboot actually works [09:07] pitti, ^ [09:07] and I just killed blueman to see if that was the cause, it wasn't [09:07] whereas with upstart it doesn't -- it always comes back enabled (at least here) [09:07] no everything has to do with systemd :p [09:07] we should still fix bugs [09:07] seb128: yes, absolutely [09:07] seb128: I filed a related bug many months ago, hang on [09:07] thanks [09:07] seb128: I can open the bug, I have just vague reproducer steps though and didn't look at it in detail, just mentionned it in the bluez session [09:08] didrocks, seems like pitti already did ;-) [09:08] ah, bug 1126108 [09:08] seb128: I bet it's an indicator + g-c-c kind of thing [09:08] bug 1126108 in indicator-bluetooth (Ubuntu) "Indicator disappears entirely when disabling / turning off bluetooth" [Medium,Fix released] https://launchpad.net/bugs/1126108 [09:08] hum [09:09] didrocks, oh, I remember that one, it's a different issue [09:09] still the very same description/symptom, apparently this either regressed, or wasn't fixed completly [09:09] the one I was speaking about is that the g-c-c control goes back to disabled when you try to enabled bt [09:09] seb128: you disabled the indicator and you still have it? [09:09] pitti, the issue was discussed a bit on robert_ancell's mp iirc [09:10] (it being the bug) [09:10] seb128: ah; no, I didn't file one for that [09:10] seb128: it feels like it doesn't wait long enough for BT to appear or something such [09:10] didrocks, no, I disabled showing the icon, the indicator is still running [09:10] seb128: I just chmod -x the indicator and killed it [09:10] the bug is still there [09:10] k [09:11] didrocks, you know you can "stop indicator-bluetooth"? ;-) [09:11] upstart management ftw [09:11] seb128: didrocks likes the blunt and heavy tools :) [09:11] seb128: I'm using old school trick learnt from an ex-french guy :) [09:11] lol [09:11] so anyway, it's not the indicator which is resetting the status under g-c-c feets [09:12] indeed not [09:12] seems it's shooting itself in its feet [09:12] speaking of ex-french -- seb128, hoe goed spreek je Nederlands door nu? [09:12] :) [09:12] pitti, ik ben Sebastien [09:13] danku [09:13] Ik hou van je! [09:13] that's about what I learnt :p [09:13] lol [09:13] * didrocks smells some google translate under this :) [09:14] * pitti looks innocent -- what is this Gogle trans-late? [09:15] hah [09:15] it's an early thingy [09:15] hoe gaat het [09:16] Laney: goed, dank je! [09:18] seb128: seems to be quite an old bug: bug #1056947 [09:18] bug 1056947 in gnome-control-center (Ubuntu) "Cannot turn on bluetooth after turning it off from the indicator" [Low,Confirmed] https://launchpad.net/bugs/1056947 [09:18] * mlankhorst tries reading afrikaans and giggles [09:19] didrocks, indeed, thanks for checking [09:19] https://bugs.launchpad.net/ubuntu/+source/gnome-control-center/+bug/1056947/comments/7 [09:19] yw [09:23] * Laney reboots with systemd too [09:23] * Laney doesn't want to be left out [09:24] Laney: tell us about your system state, if you are in degraded mode or not [09:24] and actually I use /e/n/i on my desktop :-) [09:24] old school guy! [09:25] seb128: what do you think about desktoppers not using n-m? :) [09:25] needed a bridge interface for lxc [09:25] LXC!!!!! [09:25] :) [09:25] lol [09:26] didrocks, be nice to old people [09:26] Laney: the default one doesn't do? I use LXC all the time [09:26] ;-) [09:26] heh [09:26] pitti: I wanted it to get real IPs on my LAN (+ avahi, etc), does it do that now? [09:26] Laney: ah no, the automatic virbr0 is just a 10.0.3.0/24 [09:27] nod [09:27] Laney: anyway, it should work just fine, did it? [09:27] dist-upgrading first [09:29] Laney: err, I mean lxcbr0, not virbr [09:29] it doesn't even add a global IPv6 address, sheesh! [09:30] oh noes, Errors were encountered while processing: [09:30] * Laney looks at this first [09:30] gnome-flashback [09:30] mitya57!!!!! [09:55] Hi pitti! === ara_ is now known as ara [10:19] ah, good to see Lennart's face [10:19] and now we're booted [10:19] didrocks: so what is degraded mode? [10:19] Laney: systemctl status | grep State [10:20] running [10:20] sweet, no failure then :) [10:20] it took a while, might have been fscking or something [10:20] no plymouth to tell me [10:21] yeah, this is known [10:21] otherwise, just try basic system like bluetooth and so on, and ensure it's working :) [10:21] but not having any units failing is already something [10:22] don't have any fancy hardware there [10:22] except the webcam? [10:22] holy shit, cheese is segfaulting [10:23] * Laney runs [10:23] it doesn't here, let me upgrade gstreamer :) [10:23] seems gtk-3.14-ish [10:23] I get a ton of criticals [10:24] seb128: confirm/deny? [10:24] ok, I'm on 3.12 [10:25] Laney, xfm [10:25] wfm even [10:25] no critical [10:27] with 3.14? [10:28] yes [10:29] what [10:29] WHAT [10:29] you capture my thoughts precisely [10:30] http://paste.ubuntu.com/9092839/ that's G_DEBUG=fatal-criticals [10:36] uh oh icon view [10:38] you have clue? [10:39] no, but icon view is notoriously bad to find issus in [10:40] this only started happening with systemd? [10:40] haha [10:40] I only tried it just now [10:40] could it really be that? [10:40] unlikely [10:41] no [10:41] just asking after glancing at the scrollback [10:41] I'm using systemd as well :p [10:41] we're all so futuristic [10:41] Laney, does it happen if you unset gtk_modules? [10:41] do you have o-s enabled? [10:41] criticals yes, segfault no [10:42]