[01:26] PR snapd#4079 opened: daemon: Allow Polkit authorization to cancel changes [02:04] fyi, starting RT#106813: Production SSO rollout (r1583) shortly [02:04] er, sorry, wrong channel [02:27] PR snapcraft#1640 opened: cli: add what, why, and how to fix to the common errors === kira is now known as Guest54752 [06:08] I have to get started with development of qt Application on ubuntu core how to proceed with snap please help [07:02] mvo: good morning [07:02] hey zyga-solus [07:02] mvo: I'm ready to help with 14.04 [07:02] mvo: I'll start by reproducing the problem unless you have some other suggestions [07:02] zyga-solus: cool, I tried to reproduce the error in https://bugs.launchpad.net/snapd/+bug/1721518 but failed so far [07:02] Bug #1721518: Latest snapd in Trusty is broken after reboot because of systemd units start ordering [07:03] zyga-solus: so if you can reproduce that would be great [07:03] ok, let's see what I can do [07:03] zyga-solus: I used the autopkgtest generated image, maybe that is the problem. its a very minimal image. I suspect some boot race but its all very foggy [07:03] mvo: I'll start with a desktop image as I have that around [07:04] zyga-solus: aha, cool [07:40] moin moin [07:43] mvo: how did you get 2.27.6? from launchpad somewhere or from the core snap revision? [07:56] mvo: hmm, I cannot install 2.27.6 and 2.28.5 is OK [08:01] zyga-solus: I used 2.27.5 and then installed core [08:01] zyga-solus: so maybe/probably I did not test .6 [08:01] zyga-solus: did you manage to reproduce anything? [08:02] mvo: no [08:02] zyga-solus: :( [08:02] mvo: I'll disable reexec and see [08:02] feels like a wild goose chase [08:02] mvo: though that's hardly what the OP did [08:03] zyga-solus: yeah, exactly. once you are happy (or unhappy) please followup in bug [08:04] zyga-solus: I give this one more try in a less minimal image and then give up [08:04] zyga-solus: interessting that cachio reproduced it [08:05] mvo: with 2.27.5 I get the same behavior [08:05] mvo: I bet there is something we're missing that is making this relevant [08:06] zyga-solus: yeah, hopefully that is also the glue what is wrong :) [08:06] I'll reboot a few times [08:06] maybe it really is timing [08:07] zyga-solus: no luck with a full trusty image either, disappointing [08:20] mvo: woaaah [08:20] mvo: it happened [08:21] mvo: not exactly like in the report but out of 3 installed snaps I lost some interfaces! [08:21] mvo: I just set up auto logging [08:21] mvo: and I was rebooting and running "snap interfaces" [08:21] mvo: I disabled reexec so this is stock 2.27.5 [08:21] mvo: I'm investigating the state now [08:21] woot [08:21] zyga-solus, ! [08:23] so the sad stuff is that I don't see any logs from snapd [08:23] zilch [08:23] not a single line [08:23] oh, wait I actually have something now [08:24] http://pastebin.ubuntu.com/25822002/ [08:25] theyory [08:25] *theory [08:25] very weak at the moment [08:25] there's a real bug where on certain systems YAML parsing is exploding in unexpected ways [08:25] what if it's just a race that for kernel config + vm reasons are making it more or less likely [08:25] and at random we cannot read yaml's [08:25] another theory: those are not mounted soon enough [08:26] and end up "broken" when the interface repository is setup [08:26] I'll look at the code to see what happens there [08:26] I think we are just saying "ooops, problem but not going to die on this" [08:27] I'll paste my logs in case someone can fish out facts out of timing [08:27] one sec [08:28] syslog: http://paste.ubuntu.com/25822015/ [08:29] journalctl http://paste.ubuntu.com/25822015/ [08:29] journalctl status snapd.service http://pastebin.ubuntu.com/25822002/ [08:29] oh, hold on, one of those is wrong [08:29] http://paste.ubuntu.com/25822017/ [08:29] that's journalctl (2nd paste) [08:30] snap list: http://paste.ubuntu.com/25822020/ [08:30] zyga-ubuntu: nice! yeah, the things-are-not-mounted-in-time is what the reporter also suggested [08:30] mvo: so my snapd process is in this state now [08:30] zyga-ubuntu: maybe systemd orders things differently in trusty [08:30] I'll keep it as such for now [08:31] I'll inspect units [08:31] mvo: if you have ideas for experiments to make do say, I'll dig around [08:31] zyga@ubuntu-trusty:~$ systemctl [08:31] Failed to issue method call: No such method 'ListUnits' [08:31] hmm, unexpected [08:31] is our systemd compiled correctly? [08:31] sudo ? [08:32] or maybe systemd is in a funny state [08:32] and it just happens to trigger this issue in snapd as a side-effect [08:32] aha, interesting [08:32] (just a very weak theory) [08:34] * zyga-ubuntu would love for snapd to log everything to a file manually [08:37] I've tweaked journal settings and rebooted [08:37] I'll try to reproduce the bug again [08:37] but hopefully with full output from snapd [08:37] mvo: is there a thread for this? I'd like to respond === JoshStrobl is now known as JoshStrobl|zzz [08:38] zyga-ubuntu: https://bugs.launchpad.net/snapd/+bug/1721518 [08:38] Bug #1721518: Latest snapd in Trusty is broken after reboot because of systemd units start ordering [08:43] Odd_Bloke the `.` is used to namespace the app name with the snap name [08:44] thanks! [08:46] mvo: hmm, why are there /etc/systemd/upstart/snapd.service and /etc/systemd/system/snapd.service? [08:46] popey your issue with lxd should be looked at by kalikiana, I feel that the general user experience on lxd has decayed a bit in the past months :-/ [08:47] thank you [08:47] (sorry, those are /lib, not /etc0 [08:47] zyga-ubuntu: I have no idea, I did not work on the trusty part of snapd :/ [08:47] zyga-ubuntu: sudo systemctl gives you the same error? [08:47] zyga-ubuntu: i.e. that systemctl is busted? anything in the logs that might indicate whats wrong? is systemctl status snapd working? [08:48] yes [08:49] one sec [08:49] ah, with sudo they work OK [08:49] ok [08:49] so not systemd [08:50] zyga-ubuntu: let me try something [08:50] I've tweaked the snapd unit [08:50] I now have logs each time we start [08:50] I'm in a reboot loop [08:50] let's see what we get [08:51] mvo: and it just happened again :) [08:52] http://pastebin.ubuntu.com/25822085/ [08:52] I'll enable debug [08:52] zyga-ubuntu: I wonder if something trivial like http://paste.ubuntu.com/25822089/ would be enough? [08:53] heh, maybe :-) [08:53] don't we do that? [08:53] zyga-ubuntu: maybe you can add it manually to the mount units that exist [08:53] how do we guarantee that those mount before snapd? [08:53] yep [08:53] zyga-ubuntu: I don't think so :( [08:53] let me enable debug and reproduce again before doing htis [08:54] zyga-ubuntu: I think in our systemd the mount stuff happens before multi-user target (where snapd is started). maybe (maybe!) this is different in old systemd? [08:54] zyga-ubuntu: thanks a lot [08:54] mvo: I was assuming that when there's a mount unit [08:54] systemd sets up automount thing [08:55] so that there cannot be a race if you simply attempt to access files s there [08:55] but maybe I'm wrong and that's not default at all [08:56] mvo: honestly another reason I'd cache the snap.yaml files [08:57] mvo: we could operate without mounted units [08:57] mvo: we could even mount them on demand [09:02] zyga-ubuntu: right [09:02] zyga-ubuntu: please keep me updated about if the before=snapd.service helps, that might be an easy fix [09:03] ok [09:03] yes, definitely a 2.29.x material if that helps === chihchun_afk is now known as chihchun [09:11] mvo: zyga-ubuntu: well on 14.04 systemd is not really pid 1, so not even sure when itself is started and what multi-user.target means there [09:13] PR snapcraft#1616 closed: store: guide to account creation upon login [09:14] mvo: don't at least the proxy.* config make sense on classic too? or is the code to set them too dangerous there? [09:14] kyrofa for when you come in snapcraft#1607 should be your priority for the week [09:14] PR snapcraft#1607: python plugin: use extracted pip [09:15] mvo: I added Before=snapd.service now, trying to reproduce [09:16] oh! [09:17] and on the 3rd boot it failed again! [09:17] in super weird way where core didn't load but others did [09:17] kyrofa kalikiana elopio every PR has a milestone now, DO NOT merge those marked 2.36 or we will never cut a release; focus on getting the 2.35 stuff finalized please [09:18] zyga-ubuntu: uhh, so strange [09:21] PR snapd#4080 opened: snapctl: added long help to stop/start/restart command [09:21] sergiusens: Alright. 5 PRs to get in by the looks of it. Let's push those hard! [09:24] zyga-ubuntu: does systemd own logs tells something about the order it did things? [09:25] pedronis: no, things are very crippled [09:25] pedronis: but let me look at one thing, it may indicate time [09:25] so... [09:25] core mounted on [09:25] mvo: hey, when you have a bit of time, can you point me at how you took care of the "autopkgtests" results in snapd's PRs? [09:25] Active: active (mounted) since czw 2017-10-26 11:16:39 CEST; 1min 51s ago [09:27] snapd started on [09:27] Active: active (running) since czw 2017-10-26 11:16:41 CEST; 10min ago [09:27] so core *was* mounted at the time snapd was started [09:28] I added this to the bug report [09:29] Saviq: sure, what do you mean with "took care of" ? how we added autpkgtests to the PRs? [09:29] zyga-ubuntu: sounds like we need more debug in why it fails to find core then [09:29] mvo: yes, I'm building snapd from 2.27.5 now === chihchun is now known as chihchun_afk [09:29] mvo: yes [09:29] I'll add verbose logging to that part [09:29] zyga-ubuntu: \o/ [09:29] where we add interfaces from snaps [09:30] Saviq: I talked to martin pitt, however nowdays you will need to talk to laney or sil2100 afaik, then your projects gets whitelisted and you add a webhook [09:30] mvo: aha so it's a feature of autopkgtests, nice :0 [09:31] .u.c, that is [09:31] Saviq: yeah [09:31] mvo: thanks! [09:31] yw === chihchun_afk is now known as chihchun [09:39] mcphail, i dont think the sheevaplug is ARMv7 ... (so same thing as RPi-1 ... Ubuntu wont run on it) [09:40] Thanks ogra_ [09:42] arm9 woo [09:45] i guess with the upcoming base snaps we could eventually have an armbian-base snap though ... [09:45] that would be v6 [09:46] (though might not work if snapd in the core snap isnt also built for v6) [09:48] ogra_: thought this through with kyrofa last night and realised i was not on to a winner [09:55] ok, so I have a logging build of 2.27.5 [09:55] no change apart from printfs in a few spots [09:56] * zyga-ubuntu is in a reboot loop again [09:56] wooooah [09:56] mvo: I know what the bug is now! [09:56] wow, that's golden :D [09:57] mvo: you'll love this [09:57] http://paste.ubuntu.com/25822364/ [09:57] so [09:57] we load snaps _after_ loading connections *somehow* [09:57] ohhh [09:57] a race on our side, how strange [09:57] yes [09:58] but that's almost crazy [09:58] I don't recall any goroutines or tasks there [09:59] load snaps? [09:59] zyga-ubuntu: yeah [09:59] pedronis: I updated the bug again [10:00] * zyga-ubuntu looks at what could cause this [10:00] zyga-ubuntu: silly question, could it still be snap.yaml missing for some reason? i.e. still the mount ordering race? [10:00] no [10:01] I verified that we started after the mount was ready [10:01] I suspect that starting snapd itself can trigger this [10:01] after everything else is settled [10:01] I'm looking at the code to see what happens there, I'll verify this later [10:02] zyga-ubuntu: ok, re mount unit - it might be the thing that system reports that it started the mount unit but the mount is not fully done when systemd returns and starts snapd or is that not a possibility? [10:04] mvo: look at my last pastebin [10:04] mvo: it's irrelevant because we do stuff on interfaces before we even loaded them [10:04] mvo: and we load all of them correctly later [10:04] zyga-ubuntu: ok [10:05] zyga-ubuntu: could you please pastebin your diff to produce this debug output? [10:05] no diff but I can pastebin the one file [10:06] http://paste.ubuntu.com/25822405/ [10:06] that's all the modifications I did on top of the 14.04 package [10:06] (maybe I could ask dpkg to get a diff somehow) [10:08] ta [10:08] sorry, IRL interrupts [10:08] back now [10:08] no worries [10:09] so no goroutines in how overlord starts [10:09] zyga-ubuntu, some quick feedback on your content interface pr [10:09] I'll look at what can cause the first output [10:09] pstolowski: thank you! [10:10] (the one where we are missing the snaps and go setup stuff) [10:10] mvo: hmmm [10:10] mvo: I think I made a mistake [10:10] mvo: there are two files, stdout and stderr [10:11] so odering is not what appears in the log file [10:11] I'll add more debug [10:11] mvo: it's clear we are loading the three snaps [10:11] and it's also clear that when reloadConnections is called we cannot find them [10:12] although [10:12] I just noticed this; [10:12] Plugs:map[string]*snap.PlugInfo(nil), [10:12] so they are indeed not present [10:12] how can we load a yaml [10:12] and not that !? [10:12] this version does not have that fancy validation that's using backchannels, right? [10:13] zyga-ubuntu: fwiw, I just unmount my /snap/*/* and added your debug info and ran snapd and I see similar behaviour, my debug info looks similar and I also get the "cannot connect plug..." messages [10:18] mvo: down to the list of snaps but no plug/slot? [10:18] mvo: something else is not showing an error message then [10:18] mvo: maybe another case of logger.Noticef not working [10:18] mvo: I know for a fact that it doesn't log in some cases (not sure what triggers this) [10:18] mvo: maybe we fail earlier but don't show it [10:19] zyga-ubuntu: I think what I see is similar, not sure if equal because I still haven't reproduce it on my box a single time :/ [10:21] logger.Noticef("cannot retrieve info for snap %q: %s", snapName, err) [10:21] this is how we log that failure [10:21] I'll patch logger to fmt.Printf just without any other fanciness [10:21] zyga-ubuntu: +1 [10:22] zyga-ubuntu: so yeah, if I manually umount and restart snapd I think I see exactly what is happening here. but but I'm super puzzled that before=snapd.service does not work in the mount units [10:25] mvo: let me paste my unit [10:25] maybe I did it incorrectly [10:25] (rebooting now) [10:25] with new logger [10:25] mvo: and BTW< the null loger is just MEH [10:25] why did we even have that? [10:25] early logger should be buffered [10:25] and replayed after configuration [10:26] ha [10:26] I'm lucky [10:26] boot and fail [10:27] zyga-ubuntu: nice! looking forward for the pastebin [10:28] PR snapd#4075 closed: many: reorg things in preparation to make handling of the base url in store dynamic [10:28] actually, nothing logged :/ [10:29] nothing more that is [10:29] I see my logging patch being there bug nothing, no errors [10:29] I wonder if this is a problem: [10:29] ...failed snap epoch cannot be empty [10:32] mvo: how are you running 14.04? [10:32] zyga-ubuntu: how does your boot unit look like? where you added the "before" thing? could you please pastebin one [10:32] zyga-ubuntu: in a qemu VM === chihchun is now known as chihchun_afk [10:34] mvo: SMP or UP? [10:34] mvo: I'll pastebin in a sec [10:35] log from failure: http://paste.ubuntu.com/25822552/ [10:35] http://paste.ubuntu.com/25822556/ [10:36] j [10:36] zyga-solus: defaults, so probably UP [10:36] zyga-ubuntu: before= needs to go into the [unit] [10:37] mvo: aha [10:37] mvo: so... [10:37] mvo: how can we load a yaml [10:37] see the apps [10:37] and not the plugs!? [10:37] look at line 6 [10:37] zyga-ubuntu: I suspect it did not load the yaml, but maybe I am wrong [10:37] zyga-ubuntu: try manually umount core [10:37] zyga-ubuntu: and restart snapd [10:38] apps are not in the snap state, are they? [10:38] zyga-ubuntu: I need to check, I don't know off-hand [10:40] mvo: my qemu is SMP [10:40] mvo: maybe that's relevant [10:40] I added logging to snap yaml loading [10:40] zyga-solus: I can try that. I also wonder if moving the before= helps [10:43] http://paste.ubuntu.com/25822591/ [10:43] shttp://paste.ubuntu.com/25822590/ [10:44] http://paste.ubuntu.com/25822590/ [10:45] mvo: look at this please [10:45] mvo: here canonical-livepatch did not load [10:45] zyga-solus: in which one of the three? [10:46] zyga-solus: aha, in the middle one, right? [10:46] mvo: just two [10:46] we didn't load canonical-livepatch [10:46] the other log is the same as before but with very verbose output when we load a yaml [10:46] snapstate.ActiveInfos() returned: []*snap.Info{(*snap.Info)(0xc82027c000), (*snap.Info)(0xc82027c780), (*snap.Info)(0xc82027cf00)} [10:46] this returns three infos [10:46] but only two yamls were loaded [10:46] a moment later we do something else that loads yamls [10:47] and then we load all three correctly [10:47] something is still not failing when meta/snap.yaml is absent [10:47] I'll look at snap manager now [10:50] Broken:"cannot read snap \"canonical-livepatch\": cannot find installed snap \"canonical-livepatch\" at revision 26 [10:50] mvo: I think you were right [10:51] I'll move the Before thing now [10:51] zyga-solus: yay, thank you, its a wild chase [10:51] mvo: we don't log that [10:51] mvo: because it goes into Broken: reason [10:51] I think we should be super vocal about broken snaps at startup [10:52] yes [10:52] *YES* :) [10:52] * mvo tries the SMP thing [10:52] mvo: btw, this also means that sometimes machies may not reexec [10:52] mvo: which is crazy bad if you think about it [10:54] zyga-solus: :( that is something we need to inspect too [10:54] zyga-solus: I can reproduce it now I think with -smp 4 in my qemu :-D [10:54] woot [10:54] great [10:56] so far so good, no failures [10:57] I appear to have a corrupt core snap on my system. https://forum.snapcraft.io/t/snapcraft-unable-to-install-core-snap/2599/27 [10:58] pedronis helped debug this, but I wonder how I got in this state and how one gets out of it [10:58] popey: what is the sha of the snap? [10:59] mvo: no failures [10:59] mvo: so good news and bad news [10:59] zyga-solus: sweet [10:59] mvo: good news is this is a simple fix [10:59] zyga-solus: what is the bad news here :) ? [10:59] mvo: bad news is that we never rewrite mount units [10:59] mvo: we need to switch to EnsureFiles.. for that [11:00] zyga-solus: meh, ok [11:00] and perhaps even restart snapd if we bot and found that we modified them (the function will tell us) [11:01] mvo: ok, I have no failures and I bet this is this now [11:02] zyga-solus: thank you so much for trakcing this down. I pushed a trivial PR to fix it for new units, I get lunch now, lets talk at the standup about the rewriting [11:02] oh [11:03] PR snapd#4081 opened: systemd: run all mount units before snapd.service to avoid race [11:03] I was just pushing :) [11:03] haha [11:03] ok [11:03] thank you [11:03] zyga-solus: autsch, sorry [11:03] no worries, can you please edit the commit message to explain what we found\ [11:03] I'll break too, I need to go to school (1st time, more later) [11:03] and pay for food there [11:04] zyga-solus: sure, I update the commit message and force push [11:04] mvo: after lunch I can explore updating mount units in the field [11:04] mvo: thank you! [11:04] mvo: or get back to content [11:04] mvo: I wonder what to do with 4008 [11:04] land it or wait until gustavo returns :/ [11:04] mvo: please think about it, it blocks everything I do for layouts/content [11:05] zyga-solus: yeah, lets talk priorities during the standup [11:05] zyga-solus: thanks, I will [11:05] zyga-solus: how do i calculate the sha? (honestly) [11:05] (on 16.04) [11:07] I think it is tricky popey [11:07] naive command returns bad data [11:07] pedronis: ^ [11:07] * zyga-solus AFK [11:25] zyga-solus: that's what I asked him to do the sha is wrong [11:25] we already know that [11:25] zyga-solus: what I don't know if the file is shorter or some bytes are different [11:25] I asked about that [11:26] he managed to download the same file again and get the right stuff [11:27] popey: the fix is easyish, stop snapd and any snap with daemons, replace the file and reboot or start again... doesn't give us a clue how you got in that state though [11:27] worst scenarios would be some kernel bug that writes back to a read-only squashfs :/ we obvisouly don't do any kind of writing to .snap files after downloading them [11:31] popey: you could also try to see (before fixing) if you have other corrupt snaps [11:33] how would I test every snap? [11:34] popey: something like: ls /var/lib/snapd/snaps/*.snap|xargs -n 1 snap info --verbose|grep -E "path|sha3-384" [11:34] give you all the hashes [11:34] sha3-384 doesn't exist on 16.04 AFAICT [11:35] ? [11:35] that's why I'm using snap info --verbose [11:35] oh duh, sorry [11:37] ok, so that dumps me out a bunch of snaps and sha3-384s, how do I know which are good/bad? [11:43] popey: try to see which ones gives a match with snap known ---remote snap-revision snap-sha3-384=SHA3-384 [11:44] that'll be fun with 90+ snaps installed :) [11:44] Thanks. [11:44] I could just switch channels then delete the broken snap I guess, and switch channel again to trigger a re-download? [11:45] well if you switch channel with refresh it will download something new [11:45]