[01:26] <mup> PR snapd#4079 opened: daemon: Allow Polkit authorization to cancel changes <Created by robert-ancell> <https://github.com/snapcore/snapd/pull/4079>
[02:04] <pjdc> fyi, starting RT#106813: Production SSO rollout (r1583) shortly
[02:04] <pjdc> er, sorry, wrong channel
[02:27] <mup> PR snapcraft#1640 opened: cli: add what, why, and how to fix to the common errors <Created by elopio> <https://github.com/snapcore/snapcraft/pull/1640>
[06:08] <Guest54752> I have to get started with development of qt Application on ubuntu core how to proceed with snap please help
[07:02] <zyga-solus> mvo: good morning
[07:02] <mvo> hey zyga-solus
[07:02] <zyga-solus> mvo: I'm ready to help with 14.04
[07:02] <zyga-solus> mvo: I'll start by reproducing the problem unless you have some other suggestions
[07:02] <mvo> zyga-solus: cool, I tried to reproduce the error in https://bugs.launchpad.net/snapd/+bug/1721518 but failed so far
[07:02] <mup> Bug #1721518: Latest snapd in Trusty is broken after reboot because of systemd units start ordering <snapd:In Progress by inaddy> <https://launchpad.net/bugs/1721518>
[07:03] <mvo> zyga-solus: so if you can reproduce that would be great
[07:03] <zyga-solus> ok, let's see what I can do
[07:03] <mvo> zyga-solus: I used the autopkgtest generated image, maybe that is the problem. its a very minimal image. I suspect some boot race but its all very foggy
[07:03] <zyga-solus> mvo: I'll start with a desktop image as I have that around
[07:04] <mvo> zyga-solus: aha, cool
[07:40] <kalikiana> moin moin
[07:43] <zyga-solus> mvo: how did you get 2.27.6? from launchpad somewhere or from the core snap revision?
[07:56] <zyga-solus> mvo: hmm, I cannot install 2.27.6 and 2.28.5 is OK
[08:01] <mvo> zyga-solus: I used 2.27.5 and then installed core
[08:01] <mvo> zyga-solus: so maybe/probably I did not test .6
[08:01] <mvo> zyga-solus: did you manage to reproduce anything?
[08:02] <zyga-solus> mvo: no
[08:02] <mvo> zyga-solus: :(
[08:02] <zyga-solus> mvo: I'll disable reexec and see
[08:02] <mvo> feels like a wild goose chase
[08:02] <zyga-solus> mvo: though that's hardly what the OP did
[08:03] <mvo> zyga-solus: yeah, exactly. once you are happy (or unhappy) please followup in bug
[08:04] <mvo> zyga-solus: I give this one more try in a less minimal image and then give up
[08:04] <mvo> zyga-solus: interessting that cachio  reproduced it
[08:05] <zyga-solus> mvo: with 2.27.5 I get the same behavior
[08:05] <zyga-solus> mvo: I bet there is something we're missing that is making this relevant
[08:06] <mvo> zyga-solus: yeah, hopefully that is also the glue what is wrong :)
[08:06] <zyga-solus> I'll reboot a few times
[08:06] <zyga-solus> maybe it really is timing
[08:07] <mvo> zyga-solus: no luck with a full trusty image either, disappointing
[08:20] <zyga-solus> mvo: woaaah
[08:20] <zyga-solus> mvo: it happened
[08:21] <zyga-solus> mvo: not exactly like in the report but out of 3 installed snaps I lost some interfaces!
[08:21] <zyga-solus> mvo: I just set up auto logging
[08:21] <zyga-solus> mvo: and I was rebooting and running "snap interfaces"
[08:21] <zyga-solus> mvo: I disabled reexec so this is stock 2.27.5
[08:21] <zyga-solus> mvo: I'm investigating the state now
[08:21] <pstolowski> woot
[08:21] <pstolowski> zyga-solus, !
[08:23] <zyga-solus> so the sad stuff is that I don't see any logs from snapd
[08:23] <zyga-solus> zilch
[08:23] <zyga-solus> not a single line
[08:23] <zyga-solus> oh, wait I actually have something now
[08:24] <zyga-ubuntu> http://pastebin.ubuntu.com/25822002/
[08:25] <zyga-ubuntu> theyory
[08:25] <zyga-ubuntu> *theory
[08:25] <zyga-ubuntu> very weak at the moment
[08:25] <zyga-ubuntu> there's a real bug where on certain systems YAML parsing is exploding in unexpected ways
[08:25] <zyga-ubuntu> what if it's just a race that for kernel config + vm reasons are making it more or less likely
[08:25] <zyga-ubuntu> and at random we cannot read yaml's
[08:25] <zyga-ubuntu> another theory: those are not mounted soon enough
[08:26] <zyga-ubuntu> and end up "broken" when the interface repository is setup
[08:26] <zyga-ubuntu> I'll look at the code to see what happens there
[08:26] <zyga-ubuntu> I think we are just saying "ooops, problem but not going to die on this"
[08:27] <zyga-solus> I'll paste my logs in case someone can fish out facts out of timing
[08:27] <zyga-solus> one sec
[08:28] <zyga-ubuntu> syslog: http://paste.ubuntu.com/25822015/
[08:29] <zyga-ubuntu> journalctl http://paste.ubuntu.com/25822015/
[08:29] <zyga-ubuntu> journalctl status snapd.service http://pastebin.ubuntu.com/25822002/
[08:29] <zyga-ubuntu> oh, hold on, one of those is wrong
[08:29] <zyga-ubuntu> http://paste.ubuntu.com/25822017/
[08:29] <zyga-ubuntu> that's journalctl (2nd paste)
[08:30] <zyga-ubuntu> snap list: http://paste.ubuntu.com/25822020/
[08:30] <mvo> zyga-ubuntu: nice! yeah, the things-are-not-mounted-in-time is what the reporter also suggested
[08:30] <zyga-ubuntu> mvo: so my snapd process is in this state now
[08:30] <mvo> zyga-ubuntu: maybe systemd orders things differently in trusty
[08:30] <zyga-ubuntu> I'll keep it as such for now
[08:31] <zyga-ubuntu> I'll inspect units
[08:31] <zyga-ubuntu> mvo: if you have ideas for experiments to make do say, I'll dig around
[08:31] <zyga-ubuntu> zyga@ubuntu-trusty:~$ systemctl
[08:31] <zyga-ubuntu> Failed to issue method call: No such method 'ListUnits'
[08:31] <zyga-ubuntu> hmm, unexpected
[08:31] <zyga-ubuntu> is our systemd compiled correctly?
[08:31] <mvo> sudo ?
[08:32] <mvo> or maybe systemd is in a funny state
[08:32] <mvo> and it just happens to trigger this issue in snapd as a side-effect
[08:32] <zyga-ubuntu> aha, interesting
[08:32] <mvo> (just a very weak theory)
[08:34]  * zyga-ubuntu would love for snapd to log everything to a file manually
[08:37] <zyga-ubuntu> I've tweaked journal settings and rebooted
[08:37] <zyga-ubuntu> I'll try to reproduce the bug again
[08:37] <zyga-ubuntu> but hopefully with full output from snapd
[08:37] <zyga-ubuntu> mvo: is there a thread for this? I'd like to respond
[08:38] <mvo> zyga-ubuntu: https://bugs.launchpad.net/snapd/+bug/1721518
[08:38] <mup> Bug #1721518: Latest snapd in Trusty is broken after reboot because of systemd units start ordering <snapd:In Progress by inaddy> <https://launchpad.net/bugs/1721518>
[08:43] <sergiusens> Odd_Bloke the `.` is used to namespace the app name with the snap name
[08:44] <zyga-ubuntu> thanks!
[08:46] <zyga-ubuntu> mvo: hmm, why are there /etc/systemd/upstart/snapd.service and /etc/systemd/system/snapd.service?
[08:46] <sergiusens> popey your issue with lxd should be looked at by kalikiana, I feel that the general user experience on lxd has decayed a bit in the past months :-/
[08:47] <popey> thank you
[08:47] <zyga-ubuntu> (sorry, those are /lib, not /etc0
[08:47] <mvo> zyga-ubuntu: I have no idea, I did not work on the trusty part of snapd :/
[08:47] <mvo> zyga-ubuntu: sudo systemctl gives you the same error?
[08:47] <mvo> zyga-ubuntu: i.e. that systemctl is busted? anything in the logs that might indicate whats wrong? is systemctl status snapd working?
[08:48] <zyga-ubuntu> yes
[08:49] <zyga-ubuntu> one sec
[08:49] <zyga-ubuntu> ah, with sudo they work OK
[08:49] <mvo> ok
[08:49] <mvo> so not systemd
[08:50] <mvo> zyga-ubuntu: let me try something
[08:50] <zyga-ubuntu> I've tweaked the snapd unit
[08:50] <zyga-ubuntu> I now have logs each time we start
[08:50] <zyga-ubuntu> I'm in a reboot loop
[08:50] <zyga-ubuntu> let's see what we get
[08:51] <zyga-ubuntu> mvo: and it just happened again :)
[08:52] <zyga-ubuntu> http://pastebin.ubuntu.com/25822085/
[08:52] <zyga-ubuntu> I'll enable debug
[08:52] <mvo> zyga-ubuntu: I wonder if something trivial like http://paste.ubuntu.com/25822089/ would be enough?
[08:53] <zyga-ubuntu> heh, maybe :-)
[08:53] <zyga-ubuntu> don't we do that?
[08:53] <mvo> zyga-ubuntu: maybe you can add it manually to the mount units that exist
[08:53] <zyga-ubuntu> how do we guarantee that those mount before snapd?
[08:53] <zyga-ubuntu> yep
[08:53] <mvo> zyga-ubuntu: I don't think so :(
[08:53] <zyga-ubuntu> let me enable debug and reproduce again before doing htis
[08:54] <mvo> zyga-ubuntu: I think in our systemd the mount stuff happens before multi-user target (where snapd is started). maybe (maybe!) this is different in old systemd?
[08:54] <mvo> zyga-ubuntu: thanks a lot
[08:54] <zyga-ubuntu> mvo: I was assuming that when there's a mount unit
[08:54] <zyga-ubuntu> systemd sets up automount thing
[08:55] <zyga-ubuntu> so that there cannot be a race if you simply attempt to access files s there
[08:55] <zyga-ubuntu> but maybe I'm wrong and that's not default at all
[08:56] <zyga-ubuntu> mvo: honestly another reason I'd cache the snap.yaml files
[08:57] <zyga-ubuntu> mvo: we could operate without mounted units
[08:57] <zyga-ubuntu> mvo: we could even mount them on demand
[09:02] <mvo> zyga-ubuntu: right
[09:02] <mvo> zyga-ubuntu: please keep me updated about if the before=snapd.service helps, that might be an easy fix
[09:03] <zyga-ubuntu> ok
[09:03] <zyga-ubuntu> yes, definitely a 2.29.x material if that helps
[09:11] <pedronis> mvo: zyga-ubuntu:  well on 14.04 systemd is not really pid 1, so not even sure when itself is started and what multi-user.target means there
[09:13] <mup> PR snapcraft#1616 closed: store: guide to account creation upon login <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/1616>
[09:14] <pedronis> mvo: don't at least the proxy.* config make sense on classic too? or is the code to set them too dangerous there?
[09:14] <sergiusens> kyrofa for when you come in snapcraft#1607 should be your priority for the week
[09:14] <mup> PR snapcraft#1607: python plugin: use extracted pip <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/1607>
[09:15] <zyga-ubuntu> mvo: I added Before=snapd.service now, trying to reproduce
[09:16] <zyga-ubuntu> oh!
[09:17] <zyga-ubuntu> and on the 3rd boot it failed again!
[09:17] <zyga-ubuntu> in super weird way where core didn't load but others did
[09:17] <sergiusens> kyrofa kalikiana elopio every PR has a milestone now, DO NOT merge those marked 2.36 or we will never cut a release; focus on getting the 2.35 stuff finalized please
[09:18] <mvo> zyga-ubuntu: uhh, so strange
[09:21] <mup> PR snapd#4080 opened: snapctl: added long help to stop/start/restart command <Created by stolowski> <https://github.com/snapcore/snapd/pull/4080>
[09:21] <kalikiana> sergiusens: Alright. 5 PRs to get in by the looks of it. Let's push those hard!
[09:24] <pedronis> zyga-ubuntu: does systemd  own logs tells something about the order it did things?
[09:25] <zyga-ubuntu> pedronis: no, things are very crippled
[09:25] <zyga-ubuntu> pedronis: but let me look at one thing, it may indicate time
[09:25] <zyga-ubuntu> so...
[09:25] <zyga-ubuntu> core mounted on
[09:25] <Saviq> mvo: hey, when you have a bit of time, can you point me at how you took care of the "autopkgtests" results in snapd's PRs?
[09:25] <zyga-ubuntu>    Active: active (mounted) since czw 2017-10-26 11:16:39 CEST; 1min 51s ago
[09:27] <zyga-ubuntu> snapd started on
[09:27] <zyga-ubuntu>    Active: active (running) since czw 2017-10-26 11:16:41 CEST; 10min ago
[09:27] <zyga-ubuntu> so core *was* mounted at the time snapd was started
[09:28] <zyga-ubuntu> I added this to the bug report
[09:29] <mvo> Saviq: sure, what do you mean with "took care of" ? how we added autpkgtests to the PRs?
[09:29] <mvo> zyga-ubuntu: sounds like we need more debug in why it fails to find core then
[09:29] <zyga-ubuntu> mvo: yes, I'm building snapd from 2.27.5 now
[09:29] <Saviq> mvo: yes
[09:29] <zyga-ubuntu> I'll add verbose logging to that part
[09:29] <mvo> zyga-ubuntu: \o/
[09:29] <zyga-ubuntu> where we add interfaces from snaps
[09:30] <mvo> Saviq: I talked to martin pitt, however nowdays you will need to talk to laney or sil2100 afaik, then your projects gets whitelisted and you add a webhook
[09:30] <Saviq> mvo: aha so it's a feature of autopkgtests, nice :0
[09:31] <Saviq> .u.c, that is
[09:31] <mvo> Saviq: yeah
[09:31] <Saviq> mvo: thanks!
[09:31] <mvo> yw
[09:39] <ogra_> mcphail, i dont think the sheevaplug is ARMv7 ... (so same thing as RPi-1 ... Ubuntu wont run on it)
[09:40] <mcphail> Thanks ogra_
[09:42] <mwhudson> arm9 woo
[09:45] <ogra_> i guess with the upcoming base snaps we could eventually have an armbian-base snap though ...
[09:45] <ogra_> that would be v6
[09:46] <ogra_> (though might not work if snapd in the core snap isnt also built for v6)
[09:48] <mcphail> ogra_: thought this through with kyrofa last night and realised i was not on to a winner
[09:55] <zyga-ubuntu> ok, so I have a logging build of 2.27.5
[09:55] <zyga-ubuntu> no change apart from printfs in a few spots
[09:56]  * zyga-ubuntu is in a reboot loop again
[09:56] <zyga-ubuntu> wooooah
[09:56] <zyga-ubuntu> mvo: I know what the bug is now!
[09:56] <zyga-ubuntu> wow, that's golden :D
[09:57] <zyga-ubuntu> mvo: you'll love this
[09:57] <zyga-ubuntu> http://paste.ubuntu.com/25822364/
[09:57] <zyga-ubuntu> so
[09:57] <zyga-ubuntu> we load snaps _after_ loading connections *somehow*
[09:57] <mvo> ohhh
[09:57] <mvo> a race on our side, how strange
[09:57] <zyga-ubuntu> yes
[09:58] <zyga-ubuntu> but that's almost crazy
[09:58] <zyga-ubuntu> I don't recall any goroutines or tasks there
[09:59] <pedronis> load snaps?
[09:59] <mvo> zyga-ubuntu: yeah
[09:59] <zyga-ubuntu> pedronis: I updated the bug again
[10:00]  * zyga-ubuntu looks at what could cause this
[10:00] <mvo> zyga-ubuntu: silly question, could it still be snap.yaml missing for some reason? i.e. still the mount ordering race?
[10:00] <zyga-ubuntu> no
[10:01] <zyga-ubuntu> I verified that we started after the mount was ready
[10:01] <zyga-ubuntu> I suspect that starting snapd itself can trigger this
[10:01] <zyga-ubuntu> after everything else is settled
[10:01] <zyga-ubuntu> I'm looking at the code to see what happens there, I'll verify this later
[10:02] <mvo> zyga-ubuntu: ok, re mount unit - it might be the thing that system reports that it started the mount unit but the mount is not fully done when systemd returns and starts snapd or is that not a possibility?
[10:04] <zyga-ubuntu> mvo: look at my last pastebin
[10:04] <zyga-ubuntu> mvo: it's irrelevant because we do stuff on interfaces before we even loaded them
[10:04] <zyga-ubuntu> mvo: and we load all of them correctly later
[10:04] <mvo> zyga-ubuntu: ok
[10:05] <mvo> zyga-ubuntu: could you please pastebin your diff to produce this debug output?
[10:05] <zyga-ubuntu> no diff but I can pastebin the one file
[10:06] <zyga-ubuntu> http://paste.ubuntu.com/25822405/
[10:06] <zyga-ubuntu> that's all the modifications I did on top of the 14.04 package
[10:06] <zyga-ubuntu> (maybe I could ask dpkg to get a diff somehow)
[10:08] <mvo> ta
[10:08] <zyga-ubuntu> sorry, IRL interrupts
[10:08] <zyga-ubuntu> back now
[10:08] <mvo> no worries
[10:09] <zyga-ubuntu> so no goroutines in how overlord starts
[10:09] <pstolowski> zyga-ubuntu, some quick feedback on your content interface pr
[10:09] <zyga-ubuntu> I'll look at what can cause the first output
[10:09] <zyga-ubuntu> pstolowski: thank you!
[10:10] <zyga-ubuntu> (the one where we are missing the snaps and go setup stuff)
[10:10] <zyga-ubuntu> mvo: hmmm
[10:10] <zyga-ubuntu> mvo: I think I made a mistake
[10:10] <zyga-ubuntu> mvo: there are two files, stdout and stderr
[10:11] <zyga-ubuntu> so odering is not what appears in the log file
[10:11] <zyga-ubuntu> I'll add more debug
[10:11] <zyga-ubuntu> mvo: it's clear we are loading the three snaps
[10:11] <zyga-ubuntu> and it's also clear that when reloadConnections is called we cannot find them
[10:12] <zyga-ubuntu> although
[10:12] <zyga-ubuntu> I just noticed this;
[10:12] <zyga-ubuntu> Plugs:map[string]*snap.PlugInfo(nil),
[10:12] <zyga-ubuntu> so they are indeed not present
[10:12] <zyga-ubuntu> how can we load a yaml
[10:12] <zyga-ubuntu> and not that !?
[10:12] <zyga-ubuntu> this version does not have that fancy validation that's using backchannels, right?
[10:13] <mvo> zyga-ubuntu: fwiw, I just unmount my /snap/*/* and added your debug info and ran snapd and I see similar behaviour, my debug info looks similar and I also get the "cannot connect plug..." messages
[10:18] <zyga-ubuntu> mvo: down to the list of snaps but no plug/slot?
[10:18] <zyga-ubuntu> mvo: something else is not showing an error message then
[10:18] <zyga-ubuntu> mvo: maybe another case of logger.Noticef not working
[10:18] <zyga-ubuntu> mvo: I know for a fact that it doesn't log in some cases (not sure what triggers this)
[10:18] <zyga-ubuntu> mvo: maybe we fail earlier but don't show it
[10:19] <mvo> zyga-ubuntu: I think what I see is similar, not sure if equal because I still haven't reproduce it on my box a single time :/
[10:21] <zyga-ubuntu>                         logger.Noticef("cannot retrieve info for snap %q: %s", snapName, err)
[10:21] <zyga-ubuntu> this is how we log that failure
[10:21] <zyga-ubuntu> I'll patch logger to fmt.Printf just without any other fanciness
[10:21] <mvo> zyga-ubuntu: +1
[10:22] <mvo> zyga-ubuntu: so yeah, if I manually umount and restart snapd I think I see exactly what is happening here. but but I'm super puzzled that before=snapd.service does not work in the mount units
[10:25] <zyga-ubuntu> mvo: let me paste my unit
[10:25] <zyga-ubuntu> maybe I did it incorrectly
[10:25] <zyga-ubuntu> (rebooting now)
[10:25] <zyga-ubuntu> with new logger
[10:25] <zyga-ubuntu> mvo: and BTW< the null loger is just MEH
[10:25] <zyga-ubuntu> why did we even have that?
[10:25] <zyga-ubuntu> early logger should be buffered
[10:25] <zyga-ubuntu> and replayed after configuration
[10:26] <zyga-ubuntu> ha
[10:26] <zyga-ubuntu> I'm lucky
[10:26] <zyga-ubuntu> boot and fail
[10:27] <mvo> zyga-ubuntu: nice! looking forward for the pastebin
[10:28] <mup> PR snapd#4075 closed: many: reorg things in preparation to make handling of the base url in store dynamic  <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/4075>
[10:28] <zyga-ubuntu> actually, nothing logged :/
[10:29] <zyga-ubuntu> nothing more that is
[10:29] <zyga-ubuntu> I see my logging patch being there bug nothing, no errors
[10:29] <zyga-ubuntu> I wonder if this is a problem:
[10:29] <zyga-ubuntu> ...failed snap epoch cannot be empty
[10:32] <zyga-ubuntu> mvo: how are you running 14.04?
[10:32] <mvo> zyga-ubuntu: how does your boot unit look like? where you added the "before" thing? could you please pastebin one
[10:32] <mvo> zyga-ubuntu: in a qemu VM
[10:34] <zyga-solus> mvo: SMP or UP?
[10:34] <zyga-solus> mvo: I'll pastebin in a sec
[10:35] <zyga-ubuntu> log from failure: http://paste.ubuntu.com/25822552/
[10:35] <zyga-ubuntu> http://paste.ubuntu.com/25822556/
[10:36] <mvo> j
[10:36] <mvo> zyga-solus: defaults, so probably UP
[10:36] <mvo> zyga-ubuntu: before= needs to go into the [unit]
[10:37] <zyga-ubuntu> mvo: aha
[10:37] <zyga-ubuntu> mvo: so...
[10:37] <zyga-ubuntu> mvo: how can we load a yaml
[10:37] <zyga-ubuntu> see the apps
[10:37] <zyga-ubuntu> and not the plugs!?
[10:37] <zyga-ubuntu> look at line 6
[10:37] <mvo> zyga-ubuntu: I suspect it did not load the yaml, but maybe I am wrong
[10:37] <mvo> zyga-ubuntu: try manually umount core
[10:37] <mvo> zyga-ubuntu: and restart snapd
[10:38] <zyga-ubuntu> apps are not in the snap state, are they?
[10:38] <mvo> zyga-ubuntu: I need to check, I don't know off-hand
[10:40] <zyga-solus> mvo: my qemu is SMP
[10:40] <zyga-solus> mvo: maybe that's relevant
[10:40] <zyga-solus> I added logging to snap yaml loading
[10:40] <mvo> zyga-solus: I can try that. I also wonder if moving the before= helps
[10:43] <zyga-ubuntu> http://paste.ubuntu.com/25822591/
[10:43] <zyga-ubuntu> shttp://paste.ubuntu.com/25822590/
[10:44] <zyga-ubuntu> http://paste.ubuntu.com/25822590/
[10:45] <zyga-solus> mvo: look at this please
[10:45] <zyga-solus> mvo: here canonical-livepatch did not load
[10:45] <mvo> zyga-solus: in which one of the three?
[10:46] <mvo> zyga-solus: aha, in the middle one, right?
[10:46] <zyga-solus> mvo: just two
[10:46] <zyga-solus> we didn't load canonical-livepatch
[10:46] <zyga-solus> the other log is the same as before but with very verbose output when we load a yaml
[10:46] <zyga-solus> snapstate.ActiveInfos() returned: []*snap.Info{(*snap.Info)(0xc82027c000), (*snap.Info)(0xc82027c780), (*snap.Info)(0xc82027cf00)}
[10:46] <zyga-solus> this returns three infos
[10:46] <zyga-solus> but only two yamls were loaded
[10:46] <zyga-solus> a moment later we do something else that loads yamls
[10:47] <zyga-solus> and then we load all three correctly
[10:47] <zyga-solus> something is still not failing when meta/snap.yaml is absent
[10:47] <zyga-solus> I'll look at snap manager now
[10:50] <zyga-solus> Broken:"cannot read snap \"canonical-livepatch\": cannot find installed snap \"canonical-livepatch\" at revision 26
[10:50] <zyga-solus> mvo: I think you were right
[10:51] <zyga-solus> I'll move the Before thing now
[10:51] <mvo> zyga-solus: yay, thank you, its a wild chase
[10:51] <zyga-solus> mvo: we don't log that
[10:51] <zyga-solus> mvo: because it goes into Broken: reason
[10:51] <zyga-solus> I think we should be super vocal about broken snaps at startup
[10:52] <mvo> yes
[10:52] <mvo> *YES* :)
[10:52]  * mvo tries the SMP thing
[10:52] <zyga-solus> mvo: btw, this also means that sometimes machies may not reexec
[10:52] <zyga-solus> mvo: which is crazy bad if you think about it
[10:54] <mvo> zyga-solus: :( that is something we need to inspect too
[10:54] <mvo> zyga-solus: I can reproduce it now I think with -smp 4 in my qemu :-D
[10:54] <zyga-solus> woot
[10:54] <zyga-solus> great
[10:56] <zyga-solus> so far so good, no failures
[10:57] <popey> I appear to have a corrupt core snap on my system. https://forum.snapcraft.io/t/snapcraft-unable-to-install-core-snap/2599/27
[10:58] <popey>  pedronis helped debug this, but I wonder how I got in this state and how one gets out of it
[10:58] <zyga-solus> popey: what is the sha of the snap?
[10:59] <zyga-solus> mvo: no failures
[10:59] <zyga-solus> mvo: so good news and bad news
[10:59] <mvo> zyga-solus: sweet
[10:59] <zyga-solus> mvo: good news is this is a simple fix
[10:59] <mvo> zyga-solus: what is the bad news here :) ?
[10:59] <zyga-solus> mvo: bad news is that we never rewrite mount units
[10:59] <zyga-solus> mvo: we need to switch to EnsureFiles.. for that
[11:00] <mvo> zyga-solus: meh, ok
[11:00] <zyga-solus> and perhaps even restart snapd if we bot and found that we modified them (the function will tell us)
[11:01] <zyga-solus> mvo: ok, I have no failures and I bet this is this now
[11:02] <mvo> zyga-solus: thank you so much for trakcing this down. I pushed a trivial PR to fix it for new units, I get lunch now, lets talk at the standup about the rewriting
[11:02] <zyga-solus> oh
[11:03] <mup> PR snapd#4081 opened: systemd: run all mount units before snapd.service to avoid race <Created by mvo5> <https://github.com/snapcore/snapd/pull/4081>
[11:03] <zyga-solus> I was just pushing :)
[11:03] <zyga-solus> haha
[11:03] <zyga-solus> ok
[11:03] <zyga-solus> thank you
[11:03] <mvo> zyga-solus: autsch, sorry
[11:03] <zyga-solus> no worries, can you please edit the commit message to explain what we found\
[11:03] <zyga-solus> I'll break too, I need to go to school (1st time, more later)
[11:03] <zyga-solus> and pay for food there
[11:04] <mvo> zyga-solus: sure, I update the commit message and force push
[11:04] <zyga-solus> mvo: after lunch I can explore updating mount units in the field
[11:04] <zyga-solus> mvo: thank you!
[11:04] <zyga-solus> mvo: or get back to content
[11:04] <zyga-solus> mvo: I wonder what to do with 4008
[11:04] <zyga-solus> land it or wait until gustavo returns :/
[11:04] <zyga-solus> mvo: please think about it, it blocks everything I do for layouts/content
[11:05] <mvo> zyga-solus: yeah, lets talk priorities during the standup
[11:05] <mvo> zyga-solus: thanks, I will
[11:05] <popey> zyga-solus: how do i calculate the sha? (honestly)
[11:05] <popey> (on 16.04)
[11:07] <zyga-solus> I think it is tricky popey
[11:07] <zyga-solus> naive command returns bad data
[11:07] <zyga-solus> pedronis: ^
[11:07]  * zyga-solus AFK
[11:25] <pedronis> zyga-solus: that's what I asked him to do the sha is wrong
[11:25] <pedronis> we already know that
[11:25] <pedronis> zyga-solus: what I don't know if the file is shorter or some bytes are different
[11:25] <pedronis> I asked about that
[11:26] <pedronis> he managed to download the same file again and get the right stuff
[11:27] <pedronis> popey: the fix is easyish, stop snapd and any snap with daemons,  replace the file and reboot or start again... doesn't give us a clue how you got in that state though
[11:27] <pedronis> worst scenarios would be some kernel bug that writes back to a read-only squashfs :/    we obvisouly don't do any kind of writing to .snap files after downloading them
[11:31] <pedronis> popey: you could also try to see (before fixing) if you have other corrupt snaps
[11:33] <popey> how would I test every snap?
[11:34] <pedronis> popey: something like:  ls /var/lib/snapd/snaps/*.snap|xargs -n 1  snap info --verbose|grep -E "path|sha3-384"
[11:34] <pedronis> give you all the hashes
[11:34] <popey> sha3-384 doesn't exist on 16.04 AFAICT
[11:35] <pedronis> ?
[11:35] <pedronis> that's why I'm using snap info --verbose
[11:35] <popey> oh duh, sorry
[11:37] <popey> ok, so that dumps me out a bunch of snaps and sha3-384s, how do I know which are good/bad?
[11:43] <pedronis> popey: try to see which ones gives a match with snap known ---remote  snap-revision snap-sha3-384=SHA3-384
[11:44] <popey> that'll be fun with 90+ snaps installed :)
[11:44] <popey> Thanks.
[11:44] <popey> I could just switch channels then delete the broken snap I guess, and switch channel again to trigger a re-download?
[11:45] <pedronis> well if you switch channel with refresh it will download something new
[11:45] <pedronis> but it would be good to know if you have other broken snaps
[11:45] <pedronis> it's a bit scary this issue
[11:46] <popey> it didnt re-download
[11:47] <popey> i switched and switched back earlier, it used the local snap
[11:47] <kalikiana> sergiusens, kyrofa, elopio: FYI I'll be taking a late longer break in ~2h in place of my  lunch break, and working later tonight
[11:50] <pedronis> popey:  this one liner might work to do the check:  ls /var/lib/snapd/snaps/*.snap|xargs -n 1  snap info --verbose|grep "sha3-384:"|cut -c10-|grep -Eo "[^ ]*"|xargs -n1 -I {}  snap known --remote snap-revision snap-sha3-384={}
[11:52] <pedronis> popey: we do caching now, that might be related, mvo might know more
[12:04] <Odd_Bloke> sergiusens: Well, the first . is, sure.  Any later dots would surely be fine?
[12:18] <mvo> popey: what version of snapd are you running?
[12:21] <ogra_> mvo, hrm
[12:21] <ogra_> what happened to my PATH?
[12:22] <ogra_> ogra@styx:~$ echo $PATH
[12:22] <ogra_> /home/ogra/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/var/lib/snapd/snap/bin
[12:22] <mvo> ogra_: you mean thta there are two /snap/bin in there now?
[12:22] <ogra_> (where do the latter two entries come from all of a sudden ? this is xenial 2.28.5)
[12:23] <ogra_> /etc/profile.d/apps-bin-path.sh has not changed
[12:24] <mvo> ogra_: I would suspect https://github.com/snapcore/snapd/pull/3398/files#diff-ba91021c40c2949f09b89cbc853a17daR1 is the relevant PR, however why you have /var/lib/snapd/snap/bin is a bit strange
[12:24] <mup> PR #3398: env: set XDG_DATA_DIRS for wayland et.al <Created by sergiusens> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/3398>
[12:25] <ogra_> ogra@styx:~$ locate snapd.sh
[12:25] <ogra_> ogra@styx:~$
[12:26] <ogra_> hmm
[12:27] <ogra_> and definitely not in /etc/profile.d ...
[12:28] <ogra_> mvo, hmm, that last file in that PR looks suspiciously as if nothing gets installed to /etc/profile.d anymore (packaging/ubuntu-16.04/snapd.install )
[12:29] <ogra_> that smells wrong
[12:30] <mvo> ogra_: https://github.com/snapcore/snapd/pull/3398/files#diff-f86ccdfbe771a0164e1e0c116db70386R174 should do that now
[12:30] <mup> PR #3398: env: set XDG_DATA_DIRS for wayland et.al <Created by sergiusens> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/3398>
[12:30] <mvo> ogra_: (i.e. line 174)
[12:31] <ogra_> mvo, thats debian/rules ... but the .install file explicitly doesnt install the dir
[12:31] <ogra_> also thats a conffile ...
[12:32] <ogra_> that would need some debhelper  postinst love, wouldnt it (to update the file on my disk)
[12:32] <ogra_> ogra@styx:~$ cat /etc/profile.d/apps-bin-path.sh
[12:32] <ogra_> # Expand the $PATH to include /snap/bin which is what snappy applications
[12:32] <ogra_> # use
[12:32] <ogra_> PATH=$PATH:/snap/bin
[12:33] <ogra_> i surely dont have the new file ... so it is still a bit beyond me where that PATH comes from
[12:34] <mvo> ogra_: check "wget https://launchpad.net/ubuntu/+archive/primary/+files/snapd_2.28.5_amd64.deb && (dpkg -c snapd_2.28.5_amd64.deb|grep etc/)" to see that the new file is in the package. what version of snapd are you running currenlty (apt list snapd and snap version) please?
[12:35] <mvo> popey: if it is the download cache thing that broke stuff, could you please pastebin "sudo ls /var/lib/snapd/cache"
[12:35] <mvo> popey: I wonder if that gives a hint
[12:37] <mvo> popey: also, do you still have the snap that has the wrong hash? if so, could you please put it somewhere so that I can download it?
[12:37] <popey> i do, one moment
[12:37] <mvo> popey: thank you
[12:38] <mvo> popey: and this all happend with 2.29~rc1 ?
[12:38] <mvo> popey: or a different version?
[12:39] <popey> core 3291
[12:39] <popey> so yes, 16-2.29~rc1
[12:40] <mvo> ta
[12:41] <popey> mvo: http://people.canonical.com/~alan/core_3291.snap
[12:44] <mvo> popey: ta
[12:44]  * ogra_ shakes fist at his DSL
[12:46] <ogra_> mvo, sorry, i got disconnected ... i dont have the new stuff in my /etc/profile.d here so i'm not sure where the additional PATH entries could come from ...
[12:46] <ogra_> also, my deb is still at 2.27.5 ...
[12:50] <mvo> popey: http://paste.ubuntu.com/25823191/ is what I found - I hexdumped both the snap download core --beta and your snap. the difference are 4 bytes (32bit) so I suspect corruption when writing to disk
[12:51] <popey> wow
[12:51] <mvo> ogra_: any hints in sudo grep -r var\/lib\/snapd\/snap\/bin /etc ?
[12:52] <pedronis> mvo: remember that we check the hash before mounting, so this happened after the first use
[12:53] <pedronis> mvo: the file was fine, and then later 4 bytes changed :/
[12:54] <mvo> pedronis: hm, we check the hash during the download (the hash from the data streaming via the download). do we check the on-disk file as well again? I vaguely remember we discussed this
[12:54] <pedronis> we do
[12:54] <ogra_> GRMBL ... whats up with my DSL
[12:54] <ogra_> mvo, did you get my last lines ?
[12:54] <mvo> yeah, in this case its more alarming, the file was fine on disk at least once then
[12:54] <mvo> ogra_: not sure, lsat that I wrote was: <mvo> ogra_: any hints in sudo grep -r var\/lib\/snapd\/snap\/bin /etc ?
 mvo, sorry, i got disconnected ... i dont have the new stuff in my /etc/profile.d here so i'm not sure where the additional PATH entries could come from ...
 also, my deb is still at 2.27.5 ...
[12:55] <ogra_> thats the last i typed
[12:55] <mvo> ogra_: yeah 2.28.5 is still in proposed
[12:55] <ogra_> right
[12:55] <ogra_> so i cant have the new stuff yet
[12:55] <ogra_> though i do have these PATH entries ... let me try that grep
[12:56] <ogra_> ogra@styx:~$ sudo grep -r var\/lib\/snapd\/snap\/bin /etc
[12:56] <ogra_> ogra@styx:~$
[12:56] <ogra_> nope
[12:57] <ogra_> /etc/nevironment looks normal too ... nothing in /etc/bash* or so ...
[12:57] <mvo> pedronis: you are right (of course :) - we download and then seek to zero and take the hash. then we rename. so the on-disk data was valid. unless of course we never read back from disk but instead linux just gave us what was in the memory cache. i.e. the bits on disk were wrong but because the file was fully in the cache we never read it back
[12:57] <pedronis> mvo: we do in validate-snap
[12:58] <pedronis> mvo: it's not a fun bug, unless popey has more general filesystem/disk issues
[12:59] <mvo> pedronis: could it be the same problem though? I mean, could it be that because its all in the linux disk cache we did not detect that the on-disk bits are wrong?
[12:59] <mvo> pedronis: i.e. linux itself never hit the disk to read it back in (because it was in the cache already)?
[12:59] <pedronis> mvo: could be, but darkest fear would be some obscure bug when the kernel decide to write back to the squashfs for reason
[13:00] <mvo> pedronis: yeah, that would be *wuuuarh* :)
[13:02] <pstolowski> standup?
[13:10] <mup> PR snapcraft#1641 opened: lxd: catch CalledProcessError in _container_run <Created by kalikiana> <https://github.com/snapcore/snapcraft/pull/1641>
[13:23] <zyga-solus> mvo: your PR fails tests
[13:28] <zyga-solus> mvo: I think some tests failed (unit test)
[13:28] <zyga-solus> does go test in systemd/ works?
[13:28] <mvo> zyga-solus: checking
[13:29] <pedronis> mvo:  this is topic: https://forum.snapcraft.io/t/snapcraft-unable-to-install-core-snap/2599 btw
[13:34] <mvo> pedronis: thank you
[13:43] <ryebot> snapcraft.io down?
[13:45] <mvo> ryebot: looks like it :/
[13:45] <ryebot> kk thanks
[13:48] <mvo> ryebot: looks like more infra is affected :/
[13:48] <ryebot> yeah, just failed to hit jujucharms as well
[13:48] <ryebot> whee
[13:50] <zyga-solus> darn
[13:50] <zyga-solus> tests cannot run now
[13:50] <mvo> weeeh :(
[13:51] <zyga-solus> mvo: also does --resend cache core snap?
[13:51] <mvo> zyga-solus: uh, I don't know
[13:58] <ryebot> looks like things are back up
[14:05]  * kalikiana back later
[14:09] <mup> PR snapd#4077 closed: spdx: fix for WITH syntax, require a license name before the operator <Created by matiasb> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4077>
[14:12] <mup> PR snapd#4071 closed: release: merge 2.29~rc1 changelog <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4071>
[14:16] <mup> PR snapd#4061 closed: daemon, store: forward SSO invalid credentials errors as 401 Unauthorized responses <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4061>
[14:17] <mup> PR snapd#4055 closed: daemon: generate a forbidden response message if polkit dialog is dismissed <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4055>
[14:18] <mup> PR snapd#4050 closed: cmd/snap: tell translators about arg names and descs req's <Created by chipaca> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4050>
[14:18] <mup> PR snapd#4053 closed: tests,po: sync translations from LP and add regression test for LP:1723974 <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4053>
[14:18] <mvo> yay, back to 25 open PRs!
[14:22] <mvo> jdstrand: a second review for 4054 would be great. I applied your feedback
[14:24] <jdstrand> mvo: ok
[14:25] <jdstrand> mvo: is the xdg-settings one ready for me to review again?
[14:25] <jdstrand> mvo: also, I'm going to have another policy updates PR that I'd like to get into 2.29. if I submit to master and 2.29, will that be a problem?
[14:27] <mvo> jdstrand: the xdg-settings one is not yet ready, need to address the open point about info disclousure
[14:34] <jdstrand> mvo: 4054 approved. did you see my question about 2.29?
[14:34] <mvo> jdstrand: I have not seen your question
[14:34] <mvo> jdstrand: 4060 is also ready (review points addressed)
[14:35] <jdstrand> 09:25 < jdstrand> mvo: also, I'm going to have another policy updates PR that I'd like to get into 2.29. if I submit to master and 2.29, will that be a problem?
[14:36] <jdstrand> 4060 was already on my list for today. let me take a look now
[14:36] <pedronis> jdstrand: what impact on preexisting snaps will that policy change have?
[14:38] <jdstrand> pedronis: only more access
[14:39] <mvo> jdstrand: if you submit to master and tag 2.29, that should be fine
[14:39] <jdstrand> mvo: ok thanks. need to get out from under PR reviews and forum discussions, then will submit. striving for eod tomorrow
[14:40]  * jdstrand crosses fingers
[14:40] <mvo> ta
[14:40] <mvo> jdstrand: no worries we have still some critical 2.29 prs pending so no worries
[14:46] <jdstrand> mvo: 4060 done (note, see my comment on missing assert)
[14:50] <mvo> jdstrand: ta
[14:51]  * mvo takes a short break while waiting for tests
[15:40] <pstolowski> pedronis, mvo added help for --long options to 4080, can you take a quick look and let me know wdyt?
[15:40] <pedronis> in a bit
[16:02] <pedronis> pstolowski: commented
[16:02] <pedronis> I have no idea if it's ok to refer to systemd there or not
[16:03] <pstolowski> pedronis, thanks, I've just fixed caitalization.
[16:03] <pstolowski> pedronis, yeah, me neither, but it's difficult not to since we depend so much on i
[16:03] <pstolowski> *it
[16:08] <pstolowski> mvo, your thoughts on that? ^
[16:36] <zyga-solus> re
[16:36] <zyga-solus> I'm still AFK, working with kids
[16:36] <zyga-solus> jdstrand: hey, how's life?
[16:36] <zyga-solus> jdstrand: no updates from me yet, I think we will see a PR tomorrow though
[16:58] <mup> Bug #1727787 opened: segmentation fault <Snappy:New> <https://launchpad.net/bugs/1727787>
[17:06] <gsilvapt> elopio, you around=
[17:06] <gsilvapt> s/=/?
[17:09] <elopio> gsilvapt: yes. Hello
[17:09] <gsilvapt> Can I talk to you via pvt?
[17:10] <gsilvapt> elopio ^ (I can forgetting to reference people, sorry :P )
[17:13] <elopio> gsilvapt: sure.
[17:24] <jdstrand> zyga-solus: hey, things going fine here. just trying to get through the PR reviews and forum discussions. how about you?
[17:48] <zyga-solus> jdstrand: trying to catch up with my son's classes and all the things he needs to learn in this school; otherwise pondering about overlayfs, I may have made a mistake and not realized it
[17:49] <zyga-solus> jdstrand: but no conclusions yet, I just realized something but I didn't finish investigating
[19:50] <mup> Bug #1727787 changed: segmentation fault <Snapcraft:New> <https://launchpad.net/bugs/1727787>
[20:41] <kyrofa> snappy-m-o, autopkgtest 1583 xenial:amd64
[20:41] <snappy-m-o> kyrofa: I've just triggered your test.
[20:50] <kyrofa> elopio, I've seen this error several times now: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-snapcraft-daily/xenial/amd64/s/snapcraft/20171025_182047_d2762@/log.gz
[20:50] <kyrofa> plainbox test failing, unable to find pyramid
[20:51] <kyrofa> And ROS ones, looks like
[20:51] <elopio> Mmm, let me see...
[20:52] <kyrofa> elopio, wait, no-- plainbox has a valid-looking match error
[20:52] <kyrofa> Like something changed out from under us
[20:52] <kyrofa> 2013.com.canonical.plainbox::collect-manifest versus no 2013
[20:52] <kyrofa> But the ROS ones suddenly can't find pyramid
[20:57] <kalikiana> kyrofa: I realized I hadn't actually approved snapcraft#1637 but this is good to go
[20:57] <mup> PR snapcraft#1637: repo: add elementary to deb distros <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/1637>
[20:58] <jdstrand> stgraber: hey, I see this denial: apparmor="DENIED" operation="open" profile="snap.lxd.lxc" name="/var/lib/snapd/hostfs/usr/lib/os-release" pid=10047 comm="snap-exec" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[20:59] <jdstrand> stgraber: you can fix that by plugging 'system-observe' in the lxc command
[21:10] <nanodrone> is snappy installable on puppylinux?
[21:13] <stgraber> jdstrand: why would I need that though?
[21:13] <stgraber> jdstrand: snap.lxd.lxc runs through "aa-exec -p unconfined"
[21:13] <stgraber> jdstrand: the line above seems to indicate that it's "snap-exec" which is attempting to read that file, not our script, so it'd point to a profile issue for snap-exec rather than lxc
[21:32] <elopio> kyrofa: pyramid is not on debian/tests/control for snapstests.
[21:32] <elopio> What I don't know is what changed to require it now.
[21:32] <kyrofa> elopio, huh, no kidding
[21:45] <elopio> kyrofa: you are adding a dependency on the integration tests with that skip_unless_codename
[21:45] <kyrofa> elopio, ohhhhhhhh
[21:45] <kyrofa> Dangit!
[21:46] <elopio> to be honest, I dislike the skip annotations. But, with my refactor on the suites, you would be able to put it in snapcraft/tests/skips, and just import that.
[21:46] <elopio> actually, you could do that right now, no need to wait.
[21:46] <kyrofa> elopio, you mean you don't like the decorators?
[21:47] <elopio> for the plainbox thing, I'll report a bug for joc to see. It's late for him, and I'm not sure if we can just update the expected result.
[21:47] <elopio> kyrofa: yes, the decorators.
[21:47] <kyrofa> elopio, you'd rather be using self.skipTest or whatever in the body of the test?
[21:47] <kyrofa> I feel like that adds code to the test that has nothing to do with the test
[21:48] <elopio> that is more readable to me. I'm not strongly opposing to the decorators, I just don't like them.
[21:54] <kyrofa> elopio, so to be clear, you want me to create a new snapcraft/tests/skips.py and put the skip code in there, then import from there for both the integration_tests and the units?
[21:55] <elopio> kyrofa: yes, I think that's the future. Things common will go in there, things specific will go in snapcraft/tests/unit or snapcraft/tests/integration.
[21:56] <kyrofa> elopio, good deal
[21:56] <kyrofa> elopio, and yeah, that was my favorite part of your refactor
[21:57] <elopio> my favorite part is when we get rid of snaps_tests completely. I will use this chance to move the plainbox test.
[21:58] <elopio> kyrofa, can you land snapcraft#1630 ?
[21:58] <mup> PR snapcraft#1630: tests: allow to select a suite when running autopkgtests <Created by elopio> <https://github.com/snapcore/snapcraft/pull/1630>
[21:58] <elopio> I want to start experimenting with the adt-lxd.
[22:06] <kyrofa> elopio, sure thing-- done
[22:07] <elopio> :D
[22:09] <mup> PR snapcraft#1630 closed: tests: allow to select a suite when running autopkgtests <Created by elopio> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/1630>
[22:09] <mup> PR snapcraft#1637 closed: repo: add elementary to deb distros <Created by kyrofa> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/1637>
[22:18] <kalikiana> 3 more to go til we can release :-D https://github.com/snapcore/snapcraft/pulls?q=is%3Aopen+is%3Apr+milestone%3A2.35
[22:23] <kyrofa> kalikiana, still need autopkgtests to pass...
[22:24] <kyrofa> plainbox seems busted