[05:55] Good morning [06:51] morning everyone [06:52] larsu: since you said i should ping you if i don't get a response within a few days, here it comes! PING :D https://bugzilla.gnome.org/show_bug.cgi?id=744601 [06:52] Gnome bug 744601 in general "Titlebar is hidden when window is maximized" [Normal,Unconfirmed] [06:57] ochosi, won't that conflict with unity where they do hide titlebars for maximised apps? [06:58] darkxst: i'm not really an expert on unity over here, but since larsu approved it, i presume it's ok for them [06:59] seb128, does unity rely on set_hide_titlebar_when_maximized, or do the global menu bar hacks override that? [07:00] hum, to hide what? [07:00] titlebars are handled by the wm and always removed afaik [07:01] if it was relying on a flag it would work on every application [07:01] yeah, i think this line in evince is just a relic [07:02] bonjour seb128 [07:02] pitti, salut, ça va ? [07:03] seb128: ça va bien, mieux que systemd :) [07:03] seb128, Laney: I filed bug 1423811 [07:03] bug 1423811 in systemd (Ubuntu) "219-1ubuntu1 regression: boot hangs, logind fails" [Critical,In progress] https://launchpad.net/bugs/1423811 [07:03] pitti, are Laney and I the only ones to see the issue? [07:03] I now get this as well [07:03] oh [07:03] great! [07:03] some dist-upgrade or change or whatever triggered it [07:03] so I'm debugging it now [07:04] going to be easier for you to debug [07:04] let me know how you debug [07:04] I added the minimal set of packages to downgrade there [07:04] can be useful for next time :-) [07:04] seb128, Laney: would be nice if you could confirm that you have exactly the same, i. e. downgrading these four packages and doing no other modifications makes things work again? (with and without splash) [07:04] can do that in a bit [07:05] but I need to do some catchup on email and stuff first [07:05] seb128: no worries [07:05] seb128: I'm fairly sure it's the same issue, it woudl be too much of a coincidence [07:05] Laney, seb128: or, eaiser: just wait until I have a fix, and if things still go haywire for you, you file another bug (but chances are low) [07:06] right, I was going to say [07:06] still please keep us updated on what you do and what you found [07:08] seb128: I got a debug journal and did the usual package bisecting first, so pretty much what you did yesterday [07:08] seb128: I now build a package with a sysv-generator fix, to ensure that it's not that (or maybe it actually is) [07:08] seb128: and then I'll start bisecting individual /lib/systemd/systemd-* binaries [07:08] and also have a good deep look at the debug log [07:09] (wrt. "what do I do now"); but I keep track of stuff in the bug report too [07:09] k [07:10] seb128, well the hints come from gtk, but we have mostly been on CSD's for a while now, so.... [07:22] good morning [07:22] seb128: pitti: I got the boot issue once this morning [07:23] lut didrocks [07:23] spent 30 minutes rebooting/trying multiple things… [07:23] didrocks, pitti gets it as well [07:23] didn't retrigger it [07:23] he's debugging [07:23] ah [07:23] you guys and you "no, works fine here" :-) [07:23] seb128: well, I did boot twice with the new systemd without any issue [07:23] didrocks, happy friday btw! [07:23] and this morning, got it once on 10 boots! [07:24] so clearly annoying issue :/ [07:24] happy friday seb128 :) [07:24] bonjour didrocks [07:25] yeah, until yesterday 219-1ubuntu1 was humming away fine for me, too.. [07:25] pitti: I don't want to say, but seb128 saw it first [07:25] didrocks: hereux vendredi ! [07:25] I will definitively put the blame on him :) [07:25] and happily :p [07:25] * didrocks hugs seb128 [07:25] pitti: joyeux vendredi aussi! [07:25] * seb128 hugs didrocks back [07:25] didrocks: keeping track in bug 1423811 [07:25] bug 1423811 in systemd (Ubuntu) "219-1ubuntu1 regression: boot hangs, logind fails" [Critical,In progress] https://launchpad.net/bugs/1423811 [07:25] * didrocks opens [07:26] oh, you were able to use list-jobs [07:26] sudo works, yes [07:27] my vt1 login was totally screwed for me [07:27] without sudo it wants polkit which doesn't work [07:27] no sudo, no dbus… [07:27] the machine was really in a weird state [07:27] all commands hanging basically [07:28] I get something funny in qemu as well: I see the plymouth dots on top of my X session [07:29] so "sudo plymouth quit" didn't work [07:29] (and we ignore the return value for this job) [07:32] that's unexpectedc -- my locally built .debs with merging a few trivial fixes from debian work [07:32] * pitti rebuilds our exact vivid package locally to verify [07:33] but that may explain why it worked for me until yesterday, when I got my self-built debs replaced with the archive's [07:33] pitti: well, I always used the archive version, and so, on 12 boots since I upgraded, I only got the issue once [07:34] didrocks: obviously a race; it also works fine with teh archive debs in a VM [07:35] * pitti tries some other things, bbl [07:35] good luck! [07:42] WTF, I managed to kill the dpkg database [07:42] /var/lib/dpkg/status is mostly empty [07:46] seb128, didrocks, Laney: it works again here without persistant journal [07:47] but I rm'ed mine; could you perhaps mv /var/log/journal{,.old} to keep it for reproducing, and check if that helps? [07:47] pitti: I disabled persistent journal last week [07:47] pitti: and so, got the issue this morning [07:48] ack, noting that down [07:48] $ ls /var/log/journal [07:48] ls: impossible d'accéder à /var/log/journal: Aucun fichier ou dossier de ce type [07:48] so now I hope I can get the issue again somehow [07:48] yeah, definitively can't here :/ [08:19] didrocks: *phew*, I got it in a VM! https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1423811/comments/3 [08:19] Launchpad bug 1423811 in systemd (Ubuntu) "219-1ubuntu1 regression: boot hangs, logind fails" [Critical,In progress] [08:20] that makes things a ton easier [08:21] pitti: interesting, not sure how this is related as I didn't touch the journal for quite a while, but if this can trigger the race reliably, that's nice! [08:21] didrocks: well, reliable enough at least, for me [08:23] in qemu it only seems to hang once after that, but good enough I guess [08:25] interesting… [08:25] * pitti runs off to get a haircut, bbl [08:25] see you pitti :) [08:25] pitti, have fun! [08:46] morning [08:46] * willcooke <-- no here [08:47] *not [08:47] hey willcooke-london :) [08:48] So much for my day off [08:48] hmmm [08:48] guys i have no indicators at atll running [08:49] is it something i did? [08:49] or did some update land yesterday? [08:49] hey willcooke [08:49] tsdgeos: do you have unity-panel-service running? [08:49] tsdgeos, desktop, phone, Ubuntu version, desktop env? [08:49] didrocks: yes [08:49] seb128: desktop [08:50] vivid? [08:50] otherwise i'd be in #ubuntu-touch :D [08:50] seb128: yes [08:50] https://launchpad.net/ubuntu/+source/unity/7.3.1+15.04.20150219.2-0ubuntu1 landed [08:50] do you have anything in your .cache/upstart/unity7.log [08:51] or unity-panel-service.log [08:51] untiy7 is not interesting [08:52] this is unity panel service http://paste.ubuntu.com/10321094/ [08:52] not sure if interesting or not [08:54] Trevinho, ^ can you help there? [08:57] actually it's weird [08:57] since indicator time is running for example [08:57] but i can't see it [08:57] do you have any indicator showing or menus? [08:58] the title of the app [08:58] and the [x] to close [08:58] no idea about appmenu [08:58] since it's kind of broken in qt5 apps [08:59] but probably not since firefox does not show me the menus either [08:59] let me reboot [08:59] and see if it fixes anything [09:00] ochosi, darkxst: what seb128 says. Unity always removes title bars for maximized windows, whereas gnome shell only does it when the application requests it (basically if it has window controls inside of its main toolbar) [09:00] but this has gone out of style anyway for csd [09:03] ochosi: pinging upstream about it [09:04] morning! [09:04] morning Laney! [09:05] ah pitti, you managed to get the bug? [09:06] Laney: yeah, I keep notes in the LP bug report [09:06] do you remember what I said yesterday too? [09:06] hey Laney [09:07] 19/02 18:15:00 I removed BusName from polkitd.service and SystemdService from /u/s/dbus-1/system-services/org.freedesktop.PolicyKit1.service and now I have lightdm again [09:07] Laney: it's a bit finicky to reproduce, but I now found a recipe that at least works on my laptop and in qemu [09:07] Laney: not everything, sorry; I got sidetracked by some other pings [09:07] hopefully that wasn't a coincidence [09:08] oh, maybe it was, now I have low graphics ^o) [09:08] Laney: do you use persistant journal? [09:08] yep [09:09] wait [09:09] I have a hunch that this is related to the new ACL tmpfiles.d feature [09:09] I'm getting confused between systems [09:09] but I'm currently bisecting the changes [09:09] I need to check [09:09] I have four debs to test, then a couple of binaries, etc. [09:09] I enabled it on this laptop, but that system → I just installed systemd-sysv and rebooted [09:09] (the one with the bug) [09:10] no, no persistent journal there [09:10] didrocks got it without persistant journal once, too, but it also applies ACLs to /run/systemd/journal/ [09:10] so having a persistant journal is not "the" trigger, but it helps on my systems at least [09:10] yeah, the ACL one is a good lead I guess [09:10] pitti: tell us if you need help, I'm almost done with my backlog [09:12] seb128: a reboot "fixed" it btw [09:13] i'll see if it happens again [09:13] tsdgeos, weird :-/ [09:13] tsdgeos, k, let we know [09:13] didrocks: can you reproduce it in a VM? [09:13] * pitti getting diverted by another high urgency issue again, argh [09:15] pitti: I can give it a try, let me upgrade the VM to a more recent version [09:18] didrocks: btw, I edited /etc/default/grub to drop splash quiet and added systemd.debug-shell [09:18] then sudo update-grub [09:19] pitti: ok, doing with that + persistent journal (but downloading a new iso for now) [09:20] meh, and now it stopped reproducing in vm [09:23] hah, and it starts again when I do an sbuild in the background on my host, to slow it down again [09:25] ah no, boot is just slower [09:25] bummer [09:25] argh, using /bin/bash ../configure gives different results than ../configure [09:26] giving the useless ../libtool: preserve_args+= --tag CC: not found [09:31] ochosi: pushed to master. Thanks! === hikiko-lpt is now known as hikiko [09:48] pitti: tried multiple reboots without quiet splash, creating persistent logs and nothing… [09:49] let me know if you guys need help debugging [09:49] pitti: oh, however, my keyboard is now in qwerty, despite the indicator [09:49] so, something fishy… [09:50] LANG is right… [09:50] and no LC_* [09:52] loadkeys didn't work, and now, reboot and azerty… [09:55] ah, got it this time [10:03] didrocks: did you get the bug? [10:04] Hey guys on vivid desktop I can't update via software updater because of the extras repo is there any chance we can get them disabled if they are broken? [10:04] I turned of systemd dbus activation for logind too and it's booted a few times okay now ... [10:05] hi davmor2, do you mean upgrading from utopic? [10:05] Laney: on qemu, once, on my system one as well (as said this morning ;)) [10:05] Laney: no fresh install [10:05] when did you install? [10:06] change of plans, I will be here today [10:06] Laney weekend [10:06] didrocks: lot of backlog, didn't read it all ........ [10:06] willcooke: sorry man :/ [10:06] Laney: last Saturday infact [10:07] it's supposed to not be enabled any more [10:07] https://launchpad.net/ubuntu/+source/apt-setup/1:0.80ubuntu7 [10:07] try a new daily to double check? [10:08] didrocks, thanks :) It's not so bad. I figure I wont need to work over the weekend, which is probably the better idea [10:08] Laney: hmmmm should that not of got installed via update too though or will it not over write the existing list? [10:09] willcooke: heh, not untrue [10:09] grrrr, can't stop on shell-debug :/ [10:09] davmor2: what existing list? [10:09] that's why I asked you if it was an upgrade [10:09] Laney: only update from the fresh install on saturday [10:10] I don't think anything will fix it within vivid [10:10] only for new installs and upgrades after the fixes went in [10:10] pitti: wait! I tried multiple reboots with upstart, and got one hang [10:10] seb128: Laney ^ [10:11] didrocks: with upstart? [10:11] with upstart? [10:11] yeah [10:11] umm [10:11] Laney: right. I'll do a fresh install later then and see if it is fixed in that install [10:11] init=/sbin/upstart [10:11] what are you suspecting? [10:11] different pb? [10:11] don't know, but we got also newer kernels and such [10:11] can maybe be unrelated to systemd [10:11] didrocks: where does it hang? I occasionally get graphics lockups when I boot without "quiet splash", but that's something else [10:11] it worked when I downgraded systemd though [10:11] it works with upstart for me [10:12] hangs every time with systemd [10:12] seb128: Laney: well, it works 12 times in a raw here [10:12] with latest systemd [10:12] so seems $random, maybe systemd trigger it more… [10:12] let me retry other reboots with upstart [10:12] well [10:13] but I got the exact same behavior [10:13] we need to get at the bottom of it on a system having the issye [10:13] it's related to dbus service activation [10:13] hanging, then xfailsafe… [10:13] for me [10:13] trying to do random config boots and draw conclusions is not leading anywhere [10:13] it's not consistant enough to do that [10:14] confirming, hanging again, with upstart [10:14] [ OK ] on Started D-bus… [10:14] go to a vt [10:14] see if logind is running [10:14] Laney: well, I'm unsure why my qemu doesn't enable me doing that [10:14] I'm going to the qemu console [10:14] use sendkey… [10:15] I haven't yet managed to get a reliable reproducer either; when I want to test with a different deb, I never get it any more, and then I also don't get it any more with the vivid deb [10:15] didrocks: alt+f9 works fine here in qemu (you need to grab the keyboard first) [10:15] argh, it stopped being stalled after 60s and booted this time [10:17] pitti: alt+f9 doesn't work here (after grabbing the keyboard) [10:17] ok, hanging again [10:17] do it in virt-manager, then you get a nice menu to send those keys [10:18] Started D-Bus system message bus [10:18] and now unstall (I wasn't doing anything on my system) [10:18] so maybe not the same thing, just a way slower boot [10:19] could be [10:19] if that was impacting upstart I'm surprised nobody else reported it [10:19] it's like 3 systemd users reported it [10:19] and 0 upstart users [10:19] seb128: well, it boots in the end [10:19] and we probably have a lot more people on upstart than systemd [10:19] just hanging for a while [10:19] systemd doesn't boot at the end [10:19] plytmouth go away that's all [10:19] then you can get to a vt [10:19] yeah, but the root cause might be the same? [10:19] but you never get lightdm [10:20] could be I guess [10:20] systemd just not recovering from it? [10:20] well yes [10:20] because on systemd it breaks bus activation via systemd [10:20] on upstart that probably doesn't happen [10:20] yeah, and if downgrading to 218-10ubuntu2 also helps (just the 4 systemd binaries), that's another strong indication that it's systemd specific [10:20] ah bah, I think I've heisenbugged it [10:20] in the debug log I see lots of "[system] Failed to activate service 'org.freedesktop.PolicyKit1': timed out" [10:20] same here [10:21] yeah, same here (heisenbug) [10:21] I just reverted the disable-systemd-bus-activation changes I did [10:21] and it still boots [10:21] hm, not the polkit one actually, lemme try that [10:23] kernel takes ages to init one of the drives :( [10:23] nah, it works now [10:24] after I've reverted all of the changes I made [10:24] :( [10:24] Laney: told you, it's not 100%, maybe try multiple reboots as well? [10:24] it was 100% before [10:24] then it worked once and since then seems consistent [10:25] oh hang on, I forgot one more service ;-) [10:27] Laney: I had a certain success with https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1423811/comments/3 [10:27] Launchpad bug 1423811 in systemd (Ubuntu) "219-1ubuntu1 regression: boot hangs, logind fails" [Critical,In progress] [10:27] s/certain/some/ I mean [10:27] pitti: I never had persistent journal, but can try enabling that [10:27] just to slow things down a bit; plus booting without quiet/splash and with debug [10:28] rebooting in loop, nothing :/ [10:30] ok, I can get sometimes this ~30s hang with systemd as well on the system dbus thingy [10:30] but then, get to lightdm [10:30] and systemctl status shows all services succeeding… [10:31] maybe time for bootchart to see where it's waiting? [10:35] trying with debug/no quiet/no splash [10:36] no good [10:36] (in a world where 'good' is no good :P) [10:37] hum, init=/lib/systemd/systemd-bootchart triggers a kernel panic here [10:37] checked twice, no typo… [10:38] attempted to kill init? [10:40] yeah [10:42] * pitti wonders how unhealthy http://paste.ubuntu.com/10322345/ is (polkit errors) [10:43] didrocks: do you have busybox-static installed? if not, install it (could be bug 1421117) [10:43] bug 1421117 in initramfs-tools (Ubuntu) "fails to boot with "Attempted to kill init" in VMWare, absolute /sbin/init symlink does not work" [Medium,In progress] https://launchpad.net/bugs/1421117 [10:43] pitti: ah thanks for the reference, doing [10:45] pitti: was already installed [10:45] ok, need the error message then [10:48] pitti: the issue is that it can't open /run/log for writing [10:48] opening output file "/run/log/bootchart-" : No such file or directory [10:49] like https://bbs.archlinux.org/viewtopic.php?id=166013 [10:50] I doubt -i /lib/systemd/systemd would work, but let's see… [10:51] waow it does [10:51] didrocks: systemd-bootchart does have that option, you might need some quoting, i. e. init="... -i ..." ? [10:51] pitti: yeah, this works [10:51] didrocks: ok, bug report SVP; I'll write an autopkgtest for bootchart [10:51] seb128, do you know: I want to have a new category in the u8 dash, where you can drop down for "All, Communicaton, Games" etc [10:51] ok, something else to consider, not sure why the default isn't taken [10:51] pitti: doing [10:51] seb128, I want to create one called Productivity - is it easy? [10:52] Laney: didn't you try to get a bootchart yesterday? [10:53] didrocks: how can I disable failsafe-x? it messes up qemu [10:53] I added an ExecStartPre=/bin/sleep 1 to /lib/systemd/system/polkitd.service which seems to help [10:55] didrocks: masking failsafe-x.service, I figure? [10:55] pitti: that or disabling the fallback, one sec [10:56] pitti: disable the target rather [10:56] failsafe-graphical.target [10:57] (the target is what is isolated, so remove other services) [10:58] pitti: if that's not enough, maybe overriding graphical.target: /lib/systemd/system/graphical.target.d/xdiagnose.conf (this is where there is the OnFailure=failsafe-graphical.target) [10:59] pitti: btw: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1423867 [10:59] Launchpad bug 1423867 in systemd (Ubuntu) "init=/lib/systemd/systemd-bootchart on 219 triggers a kernel panic on kvm" [Undecided,New] [11:01] didrocks: confirmed, will look after that boot failure [11:01] pitti: grrrr, I can't get to any vt, I'm having the issue now :/ [11:01] debug-shell [11:01] ah, send-key works now [11:02] ok, need to disable failsafe as well [11:11] didrocks: masking failsafe-x WFM [11:11] larsu: thanks a lot! [11:12] larsu: btw, andrzejr (developer of xfce4-indicator-plugin) had some questions wrt libido and libindicator releases, because there haven't been any for a while [11:15] pitti: I guess the target is failing and so isolation on the target isn't called [11:19] ochosi: we don't do traditional releases for those anymore, but import them to ubuntu directly [11:20] larsu: yeah, i guess he noticed, the question was, are they intended for use within ubuntu only? [11:21] (personally i get his request, because i think the indicators are great and shouldn't be limited to ubuntu. they work really well with other DEs like xfce too) [11:24] ochosi: I consider them to be ubuntu-only, because I don't like giving any stability guarantees for them. Other people think otherwise though :) [11:25] ochosi: in any case, tedg is the right guy to ask for releases [11:25] ah ok :) [11:25] pitti: ok, so during the hangs (that sometimes prevents to boot to graphical, sometimes don't), I can see: "systemd[1]: Failed to get initial list of names: Connection timed out" [11:25] * ochosi will annoy tedg in that case [11:26] :) [11:26] didrocks: hm, I don't see that in two failed cases [11:26] ochosi, nothing is ubuntu only, but the flavours have to put in some leg work in that case [11:26] larsu: are there concrete plans for unity8 yet – i mean: will indicators break for other DEs with unity8? [11:26] didrocks: just always the timeouts with activating the dbus services [11:27] pitti: I have 16 jobs starting… and stuck at that number (which I guess is due to the dbus activation) [11:27] pitti, systemd exploded now? [11:27] didrocks: I dropped the --no-debug from polkitd.service [11:27] darkxst: yeah, it's a helluva bug :/ [11:27] didrocks: in succeeding cases polkitd starts up and mumbles some stuff, but in the timeout cases it appears to not even start [11:28] didrocks: maybe the added sleep 1 helped a bit, but I get it more regularly now in VM [11:28] urgh :/ [11:29] pitti, figured as much from the back log [11:29] pitti: journactl "systemd[1]: Looping too fast. Throttling execution a little/" [11:29] ochosi: they work as they are with unity8 and I'm not aware of any concrete plans [11:30] ochosi: I wouldn't bet on the design staying the same though [11:30] ochosi: and I do have some non-concrete plans myself (which would totally change them), but I don't think I'll work on that [11:31] larsu, but teams can share, I think Ubuntu GNOME has proven that ;) [11:31] darkxst: ah, I thought he meant non-ubuntu-flavors as well [11:31] darkxst: share what? [11:31] larsu, ochosi is xubuntu? [11:31] yup [11:31] tsdgeos: have you foxed your issues? Changes happened yesterday to indicators, so you need to make sure you reloaded everything properly [11:31] larsu, packages, [11:31] Trevinho: i rebotted and it worked [11:32] darkxst: only the project lead, not xubuntu as a whole though ;) [11:32] larsu, we have a huge overlap with -desktop [11:32] darkxst: I know, I thought this was about upstream, because why else would he care about releases? [11:32] Trevinho: but when i found the issue i just had rebooted too [11:32] Trevinho: so it's stading at 50% success/error at the moment since yesterday :D [11:32] tsdgeos: I see [11:32] darkxst: I know and I didn't say that was a bad thing!? [11:32] larsu, maybe I came into the convo late! [11:32] darkxst: haha okay :) [11:33] larsu: i think the idea was to get indicators playing nicely on other distros and getting them packaged [11:33] larsu: without releases that's rather hard [11:33] ochosi: right. Talk to tedg about that. I would not recommmend it, but that's only me [11:34] ok, either way, good to know [11:34] thanks for the heads up [11:34] probably it will be fine [11:46] larsu: one more thing, will the evince patch find its way to ubuntu naturally or should/can i do anything about this? [11:49] ochosi: will find its way naturally if you're fine with waiting until we get 3.16. Otherwise let's backport it [11:51] right, would be good to have it in 15.04 for us [11:51] it's a rather annoying bug in xubuntu, losing your window borders on a maximized window isn't very intuitive :) [11:51] can i do anything to help backporting? [11:52] right, if you want to help, make an MR with the patch for our evince package [11:52] and assign me or Laney as reviewer [11:52] ok, i'll try that [11:52] pitti: bah, going for a run, can't get any useful logs and as it's not reliably reproduceable… [11:52] pitti: I'm in a broken state in qemu right now, kept it like that [11:52]