/srv/irclogs.ubuntu.com/2017/10/26/#snappy.txt

mupPR snapd#4079 opened: daemon: Allow Polkit authorization to cancel changes <Created by robert-ancell> <https://github.com/snapcore/snapd/pull/4079>01:26
pjdcfyi, starting RT#106813: Production SSO rollout (r1583) shortly02:04
pjdcer, sorry, wrong channel02:04
mupPR snapcraft#1640 opened: cli: add what, why, and how to fix to the common errors <Created by elopio> <https://github.com/snapcore/snapcraft/pull/1640>02:27
=== kira is now known as Guest54752
Guest54752I have to get started with development of qt Application on ubuntu core how to proceed with snap please help06:08
zyga-solusmvo: good morning07:02
mvohey zyga-solus07:02
zyga-solusmvo: I'm ready to help with 14.0407:02
zyga-solusmvo: I'll start by reproducing the problem unless you have some other suggestions07:02
mvozyga-solus: cool, I tried to reproduce the error in https://bugs.launchpad.net/snapd/+bug/1721518 but failed so far07:02
mupBug #1721518: Latest snapd in Trusty is broken after reboot because of systemd units start ordering <snapd:In Progress by inaddy> <https://launchpad.net/bugs/1721518>07:02
mvozyga-solus: so if you can reproduce that would be great07:03
zyga-solusok, let's see what I can do07:03
mvozyga-solus: I used the autopkgtest generated image, maybe that is the problem. its a very minimal image. I suspect some boot race but its all very foggy07:03
zyga-solusmvo: I'll start with a desktop image as I have that around07:03
mvozyga-solus: aha, cool07:04
kalikianamoin moin07:40
zyga-solusmvo: how did you get 2.27.6? from launchpad somewhere or from the core snap revision?07:43
zyga-solusmvo: hmm, I cannot install 2.27.6 and 2.28.5 is OK07:56
mvozyga-solus: I used 2.27.5 and then installed core08:01
mvozyga-solus: so maybe/probably I did not test .608:01
mvozyga-solus: did you manage to reproduce anything?08:01
zyga-solusmvo: no08:02
mvozyga-solus: :(08:02
zyga-solusmvo: I'll disable reexec and see08:02
mvofeels like a wild goose chase08:02
zyga-solusmvo: though that's hardly what the OP did08:02
mvozyga-solus: yeah, exactly. once you are happy (or unhappy) please followup in bug08:03
mvozyga-solus: I give this one more try in a less minimal image and then give up08:04
mvozyga-solus: interessting that cachio  reproduced it08:04
zyga-solusmvo: with 2.27.5 I get the same behavior08:05
zyga-solusmvo: I bet there is something we're missing that is making this relevant08:05
mvozyga-solus: yeah, hopefully that is also the glue what is wrong :)08:06
zyga-solusI'll reboot a few times08:06
zyga-solusmaybe it really is timing08:06
mvozyga-solus: no luck with a full trusty image either, disappointing08:07
zyga-solusmvo: woaaah08:20
zyga-solusmvo: it happened08:20
zyga-solusmvo: not exactly like in the report but out of 3 installed snaps I lost some interfaces!08:21
zyga-solusmvo: I just set up auto logging08:21
zyga-solusmvo: and I was rebooting and running "snap interfaces"08:21
zyga-solusmvo: I disabled reexec so this is stock 2.27.508:21
zyga-solusmvo: I'm investigating the state now08:21
pstolowskiwoot08:21
pstolowskizyga-solus, !08:21
zyga-solusso the sad stuff is that I don't see any logs from snapd08:23
zyga-soluszilch08:23
zyga-solusnot a single line08:23
zyga-solusoh, wait I actually have something now08:23
zyga-ubuntuhttp://pastebin.ubuntu.com/25822002/08:24
zyga-ubuntutheyory08:25
zyga-ubuntu*theory08:25
zyga-ubuntuvery weak at the moment08:25
zyga-ubuntuthere's a real bug where on certain systems YAML parsing is exploding in unexpected ways08:25
zyga-ubuntuwhat if it's just a race that for kernel config + vm reasons are making it more or less likely08:25
zyga-ubuntuand at random we cannot read yaml's08:25
zyga-ubuntuanother theory: those are not mounted soon enough08:25
zyga-ubuntuand end up "broken" when the interface repository is setup08:26
zyga-ubuntuI'll look at the code to see what happens there08:26
zyga-ubuntuI think we are just saying "ooops, problem but not going to die on this"08:26
zyga-solusI'll paste my logs in case someone can fish out facts out of timing08:27
zyga-solusone sec08:27
zyga-ubuntusyslog: http://paste.ubuntu.com/25822015/08:28
zyga-ubuntujournalctl http://paste.ubuntu.com/25822015/08:29
zyga-ubuntujournalctl status snapd.service http://pastebin.ubuntu.com/25822002/08:29
zyga-ubuntuoh, hold on, one of those is wrong08:29
zyga-ubuntuhttp://paste.ubuntu.com/25822017/08:29
zyga-ubuntuthat's journalctl (2nd paste)08:29
zyga-ubuntusnap list: http://paste.ubuntu.com/25822020/08:30
mvozyga-ubuntu: nice! yeah, the things-are-not-mounted-in-time is what the reporter also suggested08:30
zyga-ubuntumvo: so my snapd process is in this state now08:30
mvozyga-ubuntu: maybe systemd orders things differently in trusty08:30
zyga-ubuntuI'll keep it as such for now08:30
zyga-ubuntuI'll inspect units08:31
zyga-ubuntumvo: if you have ideas for experiments to make do say, I'll dig around08:31
zyga-ubuntuzyga@ubuntu-trusty:~$ systemctl08:31
zyga-ubuntuFailed to issue method call: No such method 'ListUnits'08:31
zyga-ubuntuhmm, unexpected08:31
zyga-ubuntuis our systemd compiled correctly?08:31
mvosudo ?08:31
mvoor maybe systemd is in a funny state08:32
mvoand it just happens to trigger this issue in snapd as a side-effect08:32
zyga-ubuntuaha, interesting08:32
mvo(just a very weak theory)08:32
* zyga-ubuntu would love for snapd to log everything to a file manually08:34
zyga-ubuntuI've tweaked journal settings and rebooted08:37
zyga-ubuntuI'll try to reproduce the bug again08:37
zyga-ubuntubut hopefully with full output from snapd08:37
zyga-ubuntumvo: is there a thread for this? I'd like to respond08:37
=== JoshStrobl is now known as JoshStrobl|zzz
mvozyga-ubuntu: https://bugs.launchpad.net/snapd/+bug/172151808:38
mupBug #1721518: Latest snapd in Trusty is broken after reboot because of systemd units start ordering <snapd:In Progress by inaddy> <https://launchpad.net/bugs/1721518>08:38
sergiusensOdd_Bloke the `.` is used to namespace the app name with the snap name08:43
zyga-ubuntuthanks!08:44
zyga-ubuntumvo: hmm, why are there /etc/systemd/upstart/snapd.service and /etc/systemd/system/snapd.service?08:46
sergiusenspopey your issue with lxd should be looked at by kalikiana, I feel that the general user experience on lxd has decayed a bit in the past months :-/08:46
popeythank you08:47
zyga-ubuntu(sorry, those are /lib, not /etc008:47
mvozyga-ubuntu: I have no idea, I did not work on the trusty part of snapd :/08:47
mvozyga-ubuntu: sudo systemctl gives you the same error?08:47
mvozyga-ubuntu: i.e. that systemctl is busted? anything in the logs that might indicate whats wrong? is systemctl status snapd working?08:47
zyga-ubuntuyes08:48
zyga-ubuntuone sec08:49
zyga-ubuntuah, with sudo they work OK08:49
mvook08:49
mvoso not systemd08:49
mvozyga-ubuntu: let me try something08:50
zyga-ubuntuI've tweaked the snapd unit08:50
zyga-ubuntuI now have logs each time we start08:50
zyga-ubuntuI'm in a reboot loop08:50
zyga-ubuntulet's see what we get08:50
zyga-ubuntumvo: and it just happened again :)08:51
zyga-ubuntuhttp://pastebin.ubuntu.com/25822085/08:52
zyga-ubuntuI'll enable debug08:52
mvozyga-ubuntu: I wonder if something trivial like http://paste.ubuntu.com/25822089/ would be enough?08:52
zyga-ubuntuheh, maybe :-)08:53
zyga-ubuntudon't we do that?08:53
mvozyga-ubuntu: maybe you can add it manually to the mount units that exist08:53
zyga-ubuntuhow do we guarantee that those mount before snapd?08:53
zyga-ubuntuyep08:53
mvozyga-ubuntu: I don't think so :(08:53
zyga-ubuntulet me enable debug and reproduce again before doing htis08:53
mvozyga-ubuntu: I think in our systemd the mount stuff happens before multi-user target (where snapd is started). maybe (maybe!) this is different in old systemd?08:54
mvozyga-ubuntu: thanks a lot08:54
zyga-ubuntumvo: I was assuming that when there's a mount unit08:54
zyga-ubuntusystemd sets up automount thing08:54
zyga-ubuntuso that there cannot be a race if you simply attempt to access files s there08:55
zyga-ubuntubut maybe I'm wrong and that's not default at all08:55
zyga-ubuntumvo: honestly another reason I'd cache the snap.yaml files08:56
zyga-ubuntumvo: we could operate without mounted units08:57
zyga-ubuntumvo: we could even mount them on demand08:57
mvozyga-ubuntu: right09:02
mvozyga-ubuntu: please keep me updated about if the before=snapd.service helps, that might be an easy fix09:02
zyga-ubuntuok09:03
zyga-ubuntuyes, definitely a 2.29.x material if that helps09:03
=== chihchun_afk is now known as chihchun
pedronismvo: zyga-ubuntu:  well on 14.04 systemd is not really pid 1, so not even sure when itself is started and what multi-user.target means there09:11
mupPR snapcraft#1616 closed: store: guide to account creation upon login <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/1616>09:13
pedronismvo: don't at least the proxy.* config make sense on classic too? or is the code to set them too dangerous there?09:14
sergiusenskyrofa for when you come in snapcraft#1607 should be your priority for the week09:14
mupPR snapcraft#1607: python plugin: use extracted pip <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/1607>09:14
zyga-ubuntumvo: I added Before=snapd.service now, trying to reproduce09:15
zyga-ubuntuoh!09:16
zyga-ubuntuand on the 3rd boot it failed again!09:17
zyga-ubuntuin super weird way where core didn't load but others did09:17
sergiusenskyrofa kalikiana elopio every PR has a milestone now, DO NOT merge those marked 2.36 or we will never cut a release; focus on getting the 2.35 stuff finalized please09:17
mvozyga-ubuntu: uhh, so strange09:18
mupPR snapd#4080 opened: snapctl: added long help to stop/start/restart command <Created by stolowski> <https://github.com/snapcore/snapd/pull/4080>09:21
kalikianasergiusens: Alright. 5 PRs to get in by the looks of it. Let's push those hard!09:21
pedroniszyga-ubuntu: does systemd  own logs tells something about the order it did things?09:24
zyga-ubuntupedronis: no, things are very crippled09:25
zyga-ubuntupedronis: but let me look at one thing, it may indicate time09:25
zyga-ubuntuso...09:25
zyga-ubuntucore mounted on09:25
Saviqmvo: hey, when you have a bit of time, can you point me at how you took care of the "autopkgtests" results in snapd's PRs?09:25
zyga-ubuntu   Active: active (mounted) since czw 2017-10-26 11:16:39 CEST; 1min 51s ago09:25
zyga-ubuntusnapd started on09:27
zyga-ubuntu   Active: active (running) since czw 2017-10-26 11:16:41 CEST; 10min ago09:27
zyga-ubuntuso core *was* mounted at the time snapd was started09:27
zyga-ubuntuI added this to the bug report09:28
mvoSaviq: sure, what do you mean with "took care of" ? how we added autpkgtests to the PRs?09:29
mvozyga-ubuntu: sounds like we need more debug in why it fails to find core then09:29
zyga-ubuntumvo: yes, I'm building snapd from 2.27.5 now09:29
=== chihchun is now known as chihchun_afk
Saviqmvo: yes09:29
zyga-ubuntuI'll add verbose logging to that part09:29
mvozyga-ubuntu: \o/09:29
zyga-ubuntuwhere we add interfaces from snaps09:29
mvoSaviq: I talked to martin pitt, however nowdays you will need to talk to laney or sil2100 afaik, then your projects gets whitelisted and you add a webhook09:30
Saviqmvo: aha so it's a feature of autopkgtests, nice :009:30
Saviq.u.c, that is09:31
mvoSaviq: yeah09:31
Saviqmvo: thanks!09:31
mvoyw09:31
=== chihchun_afk is now known as chihchun
ogra_mcphail, i dont think the sheevaplug is ARMv7 ... (so same thing as RPi-1 ... Ubuntu wont run on it)09:39
mcphailThanks ogra_09:40
mwhudsonarm9 woo09:42
ogra_i guess with the upcoming base snaps we could eventually have an armbian-base snap though ...09:45
ogra_that would be v609:45
ogra_(though might not work if snapd in the core snap isnt also built for v6)09:46
mcphailogra_: thought this through with kyrofa last night and realised i was not on to a winner09:48
zyga-ubuntuok, so I have a logging build of 2.27.509:55
zyga-ubuntuno change apart from printfs in a few spots09:55
* zyga-ubuntu is in a reboot loop again09:56
zyga-ubuntuwooooah09:56
zyga-ubuntumvo: I know what the bug is now!09:56
zyga-ubuntuwow, that's golden :D09:56
zyga-ubuntumvo: you'll love this09:57
zyga-ubuntuhttp://paste.ubuntu.com/25822364/09:57
zyga-ubuntuso09:57
zyga-ubuntuwe load snaps _after_ loading connections *somehow*09:57
mvoohhh09:57
mvoa race on our side, how strange09:57
zyga-ubuntuyes09:57
zyga-ubuntubut that's almost crazy09:58
zyga-ubuntuI don't recall any goroutines or tasks there09:58
pedronisload snaps?09:59
mvozyga-ubuntu: yeah09:59
zyga-ubuntupedronis: I updated the bug again09:59
* zyga-ubuntu looks at what could cause this10:00
mvozyga-ubuntu: silly question, could it still be snap.yaml missing for some reason? i.e. still the mount ordering race?10:00
zyga-ubuntuno10:00
zyga-ubuntuI verified that we started after the mount was ready10:01
zyga-ubuntuI suspect that starting snapd itself can trigger this10:01
zyga-ubuntuafter everything else is settled10:01
zyga-ubuntuI'm looking at the code to see what happens there, I'll verify this later10:01
mvozyga-ubuntu: ok, re mount unit - it might be the thing that system reports that it started the mount unit but the mount is not fully done when systemd returns and starts snapd or is that not a possibility?10:02
zyga-ubuntumvo: look at my last pastebin10:04
zyga-ubuntumvo: it's irrelevant because we do stuff on interfaces before we even loaded them10:04
zyga-ubuntumvo: and we load all of them correctly later10:04
mvozyga-ubuntu: ok10:04
mvozyga-ubuntu: could you please pastebin your diff to produce this debug output?10:05
zyga-ubuntuno diff but I can pastebin the one file10:05
zyga-ubuntuhttp://paste.ubuntu.com/25822405/10:06
zyga-ubuntuthat's all the modifications I did on top of the 14.04 package10:06
zyga-ubuntu(maybe I could ask dpkg to get a diff somehow)10:06
mvota10:08
zyga-ubuntusorry, IRL interrupts10:08
zyga-ubuntuback now10:08
mvono worries10:08
zyga-ubuntuso no goroutines in how overlord starts10:09
pstolowskizyga-ubuntu, some quick feedback on your content interface pr10:09
zyga-ubuntuI'll look at what can cause the first output10:09
zyga-ubuntupstolowski: thank you!10:09
zyga-ubuntu(the one where we are missing the snaps and go setup stuff)10:10
zyga-ubuntumvo: hmmm10:10
zyga-ubuntumvo: I think I made a mistake10:10
zyga-ubuntumvo: there are two files, stdout and stderr10:10
zyga-ubuntuso odering is not what appears in the log file10:11
zyga-ubuntuI'll add more debug10:11
zyga-ubuntumvo: it's clear we are loading the three snaps10:11
zyga-ubuntuand it's also clear that when reloadConnections is called we cannot find them10:11
zyga-ubuntualthough10:12
zyga-ubuntuI just noticed this;10:12
zyga-ubuntuPlugs:map[string]*snap.PlugInfo(nil),10:12
zyga-ubuntuso they are indeed not present10:12
zyga-ubuntuhow can we load a yaml10:12
zyga-ubuntuand not that !?10:12
zyga-ubuntuthis version does not have that fancy validation that's using backchannels, right?10:12
mvozyga-ubuntu: fwiw, I just unmount my /snap/*/* and added your debug info and ran snapd and I see similar behaviour, my debug info looks similar and I also get the "cannot connect plug..." messages10:13
zyga-ubuntumvo: down to the list of snaps but no plug/slot?10:18
zyga-ubuntumvo: something else is not showing an error message then10:18
zyga-ubuntumvo: maybe another case of logger.Noticef not working10:18
zyga-ubuntumvo: I know for a fact that it doesn't log in some cases (not sure what triggers this)10:18
zyga-ubuntumvo: maybe we fail earlier but don't show it10:18
mvozyga-ubuntu: I think what I see is similar, not sure if equal because I still haven't reproduce it on my box a single time :/10:19
zyga-ubuntu                        logger.Noticef("cannot retrieve info for snap %q: %s", snapName, err)10:21
zyga-ubuntuthis is how we log that failure10:21
zyga-ubuntuI'll patch logger to fmt.Printf just without any other fanciness10:21
mvozyga-ubuntu: +110:21
mvozyga-ubuntu: so yeah, if I manually umount and restart snapd I think I see exactly what is happening here. but but I'm super puzzled that before=snapd.service does not work in the mount units10:22
zyga-ubuntumvo: let me paste my unit10:25
zyga-ubuntumaybe I did it incorrectly10:25
zyga-ubuntu(rebooting now)10:25
zyga-ubuntuwith new logger10:25
zyga-ubuntumvo: and BTW< the null loger is just MEH10:25
zyga-ubuntuwhy did we even have that?10:25
zyga-ubuntuearly logger should be buffered10:25
zyga-ubuntuand replayed after configuration10:25
zyga-ubuntuha10:26
zyga-ubuntuI'm lucky10:26
zyga-ubuntuboot and fail10:26
mvozyga-ubuntu: nice! looking forward for the pastebin10:27
mupPR snapd#4075 closed: many: reorg things in preparation to make handling of the base url in store dynamic  <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/4075>10:28
zyga-ubuntuactually, nothing logged :/10:28
zyga-ubuntunothing more that is10:29
zyga-ubuntuI see my logging patch being there bug nothing, no errors10:29
zyga-ubuntuI wonder if this is a problem:10:29
zyga-ubuntu...failed snap epoch cannot be empty10:29
zyga-ubuntumvo: how are you running 14.04?10:32
mvozyga-ubuntu: how does your boot unit look like? where you added the "before" thing? could you please pastebin one10:32
mvozyga-ubuntu: in a qemu VM10:32
=== chihchun is now known as chihchun_afk
zyga-solusmvo: SMP or UP?10:34
zyga-solusmvo: I'll pastebin in a sec10:34
zyga-ubuntulog from failure: http://paste.ubuntu.com/25822552/10:35
zyga-ubuntuhttp://paste.ubuntu.com/25822556/10:35
mvoj10:36
mvozyga-solus: defaults, so probably UP10:36
mvozyga-ubuntu: before= needs to go into the [unit]10:36
zyga-ubuntumvo: aha10:37
zyga-ubuntumvo: so...10:37
zyga-ubuntumvo: how can we load a yaml10:37
zyga-ubuntusee the apps10:37
zyga-ubuntuand not the plugs!?10:37
zyga-ubuntulook at line 610:37
mvozyga-ubuntu: I suspect it did not load the yaml, but maybe I am wrong10:37
mvozyga-ubuntu: try manually umount core10:37
mvozyga-ubuntu: and restart snapd10:37
zyga-ubuntuapps are not in the snap state, are they?10:38
mvozyga-ubuntu: I need to check, I don't know off-hand10:38
zyga-solusmvo: my qemu is SMP10:40
zyga-solusmvo: maybe that's relevant10:40
zyga-solusI added logging to snap yaml loading10:40
mvozyga-solus: I can try that. I also wonder if moving the before= helps10:40
zyga-ubuntuhttp://paste.ubuntu.com/25822591/10:43
zyga-ubuntushttp://paste.ubuntu.com/25822590/10:43
zyga-ubuntuhttp://paste.ubuntu.com/25822590/10:44
zyga-solusmvo: look at this please10:45
zyga-solusmvo: here canonical-livepatch did not load10:45
mvozyga-solus: in which one of the three?10:45
mvozyga-solus: aha, in the middle one, right?10:46
zyga-solusmvo: just two10:46
zyga-soluswe didn't load canonical-livepatch10:46
zyga-solusthe other log is the same as before but with very verbose output when we load a yaml10:46
zyga-solussnapstate.ActiveInfos() returned: []*snap.Info{(*snap.Info)(0xc82027c000), (*snap.Info)(0xc82027c780), (*snap.Info)(0xc82027cf00)}10:46
zyga-solusthis returns three infos10:46
zyga-solusbut only two yamls were loaded10:46
zyga-solusa moment later we do something else that loads yamls10:46
zyga-solusand then we load all three correctly10:47
zyga-solussomething is still not failing when meta/snap.yaml is absent10:47
zyga-solusI'll look at snap manager now10:47
zyga-solusBroken:"cannot read snap \"canonical-livepatch\": cannot find installed snap \"canonical-livepatch\" at revision 2610:50
zyga-solusmvo: I think you were right10:50
zyga-solusI'll move the Before thing now10:51
mvozyga-solus: yay, thank you, its a wild chase10:51
zyga-solusmvo: we don't log that10:51
zyga-solusmvo: because it goes into Broken: reason10:51
zyga-solusI think we should be super vocal about broken snaps at startup10:51
mvoyes10:52
mvo*YES* :)10:52
* mvo tries the SMP thing10:52
zyga-solusmvo: btw, this also means that sometimes machies may not reexec10:52
zyga-solusmvo: which is crazy bad if you think about it10:52
mvozyga-solus: :( that is something we need to inspect too10:54
mvozyga-solus: I can reproduce it now I think with -smp 4 in my qemu :-D10:54
zyga-soluswoot10:54
zyga-solusgreat10:54
zyga-solusso far so good, no failures10:56
popeyI appear to have a corrupt core snap on my system. https://forum.snapcraft.io/t/snapcraft-unable-to-install-core-snap/2599/2710:57
popey pedronis helped debug this, but I wonder how I got in this state and how one gets out of it10:58
zyga-soluspopey: what is the sha of the snap?10:58
zyga-solusmvo: no failures10:59
zyga-solusmvo: so good news and bad news10:59
mvozyga-solus: sweet10:59
zyga-solusmvo: good news is this is a simple fix10:59
mvozyga-solus: what is the bad news here :) ?10:59
zyga-solusmvo: bad news is that we never rewrite mount units10:59
zyga-solusmvo: we need to switch to EnsureFiles.. for that10:59
mvozyga-solus: meh, ok11:00
zyga-solusand perhaps even restart snapd if we bot and found that we modified them (the function will tell us)11:00
zyga-solusmvo: ok, I have no failures and I bet this is this now11:01
mvozyga-solus: thank you so much for trakcing this down. I pushed a trivial PR to fix it for new units, I get lunch now, lets talk at the standup about the rewriting11:02
zyga-solusoh11:02
mupPR snapd#4081 opened: systemd: run all mount units before snapd.service to avoid race <Created by mvo5> <https://github.com/snapcore/snapd/pull/4081>11:03
zyga-solusI was just pushing :)11:03
zyga-solushaha11:03
zyga-solusok11:03
zyga-solusthank you11:03
mvozyga-solus: autsch, sorry11:03
zyga-solusno worries, can you please edit the commit message to explain what we found\11:03
zyga-solusI'll break too, I need to go to school (1st time, more later)11:03
zyga-solusand pay for food there11:03
mvozyga-solus: sure, I update the commit message and force push11:04
zyga-solusmvo: after lunch I can explore updating mount units in the field11:04
zyga-solusmvo: thank you!11:04
zyga-solusmvo: or get back to content11:04
zyga-solusmvo: I wonder what to do with 400811:04
zyga-solusland it or wait until gustavo returns :/11:04
zyga-solusmvo: please think about it, it blocks everything I do for layouts/content11:04
mvozyga-solus: yeah, lets talk priorities during the standup11:05
mvozyga-solus: thanks, I will11:05
popeyzyga-solus: how do i calculate the sha? (honestly)11:05
popey(on 16.04)11:05
zyga-solusI think it is tricky popey11:07
zyga-solusnaive command returns bad data11:07
zyga-soluspedronis: ^11:07
* zyga-solus AFK11:07
pedroniszyga-solus: that's what I asked him to do the sha is wrong11:25
pedroniswe already know that11:25
pedroniszyga-solus: what I don't know if the file is shorter or some bytes are different11:25
pedronisI asked about that11:25
pedronishe managed to download the same file again and get the right stuff11:26
pedronispopey: the fix is easyish, stop snapd and any snap with daemons,  replace the file and reboot or start again... doesn't give us a clue how you got in that state though11:27
pedronisworst scenarios would be some kernel bug that writes back to a read-only squashfs :/    we obvisouly don't do any kind of writing to .snap files after downloading them11:27
pedronispopey: you could also try to see (before fixing) if you have other corrupt snaps11:31
popeyhow would I test every snap?11:33
pedronispopey: something like:  ls /var/lib/snapd/snaps/*.snap|xargs -n 1  snap info --verbose|grep -E "path|sha3-384"11:34
pedronisgive you all the hashes11:34
popeysha3-384 doesn't exist on 16.04 AFAICT11:34
pedronis?11:35
pedronisthat's why I'm using snap info --verbose11:35
popeyoh duh, sorry11:35
popeyok, so that dumps me out a bunch of snaps and sha3-384s, how do I know which are good/bad?11:37
pedronispopey: try to see which ones gives a match with snap known ---remote  snap-revision snap-sha3-384=SHA3-38411:43
popeythat'll be fun with 90+ snaps installed :)11:44
popeyThanks.11:44
popeyI could just switch channels then delete the broken snap I guess, and switch channel again to trigger a re-download?11:44
pedroniswell if you switch channel with refresh it will download something new11:45
pedronisbut it would be good to know if you have other broken snaps11:45
pedronisit's a bit scary this issue11:45
popeyit didnt re-download11:46
popeyi switched and switched back earlier, it used the local snap11:47
kalikianasergiusens, kyrofa, elopio: FYI I'll be taking a late longer break in ~2h in place of my  lunch break, and working later tonight11:47
pedronispopey:  this one liner might work to do the check:  ls /var/lib/snapd/snaps/*.snap|xargs -n 1  snap info --verbose|grep "sha3-384:"|cut -c10-|grep -Eo "[^ ]*"|xargs -n1 -I {}  snap known --remote snap-revision snap-sha3-384={}11:50
pedronispopey: we do caching now, that might be related, mvo might know more11:52
Odd_Blokesergiusens: Well, the first . is, sure.  Any later dots would surely be fine?12:04
mvopopey: what version of snapd are you running?12:18
ogra_mvo, hrm12:21
ogra_what happened to my PATH?12:21
ogra_ogra@styx:~$ echo $PATH12:22
ogra_/home/ogra/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/var/lib/snapd/snap/bin12:22
mvoogra_: you mean thta there are two /snap/bin in there now?12:22
ogra_(where do the latter two entries come from all of a sudden ? this is xenial 2.28.5)12:22
ogra_/etc/profile.d/apps-bin-path.sh has not changed12:23
mvoogra_: I would suspect https://github.com/snapcore/snapd/pull/3398/files#diff-ba91021c40c2949f09b89cbc853a17daR1 is the relevant PR, however why you have /var/lib/snapd/snap/bin is a bit strange12:24
mupPR #3398: env: set XDG_DATA_DIRS for wayland et.al <Created by sergiusens> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/3398>12:24
ogra_ogra@styx:~$ locate snapd.sh12:25
ogra_ogra@styx:~$12:25
ogra_hmm12:26
ogra_and definitely not in /etc/profile.d ...12:27
ogra_mvo, hmm, that last file in that PR looks suspiciously as if nothing gets installed to /etc/profile.d anymore (packaging/ubuntu-16.04/snapd.install )12:28
ogra_that smells wrong12:29
mvoogra_: https://github.com/snapcore/snapd/pull/3398/files#diff-f86ccdfbe771a0164e1e0c116db70386R174 should do that now12:30
mupPR #3398: env: set XDG_DATA_DIRS for wayland et.al <Created by sergiusens> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/3398>12:30
mvoogra_: (i.e. line 174)12:30
ogra_mvo, thats debian/rules ... but the .install file explicitly doesnt install the dir12:31
ogra_also thats a conffile ...12:31
ogra_that would need some debhelper  postinst love, wouldnt it (to update the file on my disk)12:32
ogra_ogra@styx:~$ cat /etc/profile.d/apps-bin-path.sh12:32
ogra_# Expand the $PATH to include /snap/bin which is what snappy applications12:32
ogra_# use12:32
ogra_PATH=$PATH:/snap/bin12:32
ogra_i surely dont have the new file ... so it is still a bit beyond me where that PATH comes from12:33
mvoogra_: check "wget https://launchpad.net/ubuntu/+archive/primary/+files/snapd_2.28.5_amd64.deb && (dpkg -c snapd_2.28.5_amd64.deb|grep etc/)" to see that the new file is in the package. what version of snapd are you running currenlty (apt list snapd and snap version) please?12:34
mvopopey: if it is the download cache thing that broke stuff, could you please pastebin "sudo ls /var/lib/snapd/cache"12:35
mvopopey: I wonder if that gives a hint12:35
mvopopey: also, do you still have the snap that has the wrong hash? if so, could you please put it somewhere so that I can download it?12:37
popeyi do, one moment12:37
mvopopey: thank you12:37
mvopopey: and this all happend with 2.29~rc1 ?12:38
mvopopey: or a different version?12:38
popeycore 329112:39
popeyso yes, 16-2.29~rc112:39
mvota12:40
popeymvo: http://people.canonical.com/~alan/core_3291.snap12:41
mvopopey: ta12:44
* ogra_ shakes fist at his DSL12:44
ogra_mvo, sorry, i got disconnected ... i dont have the new stuff in my /etc/profile.d here so i'm not sure where the additional PATH entries could come from ...12:46
ogra_also, my deb is still at 2.27.5 ...12:46
mvopopey: http://paste.ubuntu.com/25823191/ is what I found - I hexdumped both the snap download core --beta and your snap. the difference are 4 bytes (32bit) so I suspect corruption when writing to disk12:50
popeywow12:51
mvoogra_: any hints in sudo grep -r var\/lib\/snapd\/snap\/bin /etc ?12:51
pedronismvo: remember that we check the hash before mounting, so this happened after the first use12:52
pedronismvo: the file was fine, and then later 4 bytes changed :/12:53
mvopedronis: hm, we check the hash during the download (the hash from the data streaming via the download). do we check the on-disk file as well again? I vaguely remember we discussed this12:54
pedroniswe do12:54
ogra_GRMBL ... whats up with my DSL12:54
ogra_mvo, did you get my last lines ?12:54
mvoyeah, in this case its more alarming, the file was fine on disk at least once then12:54
mvoogra_: not sure, lsat that I wrote was: <mvo> ogra_: any hints in sudo grep -r var\/lib\/snapd\/snap\/bin /etc ?12:54
ogra_<ogra_> mvo, sorry, i got disconnected ... i dont have the new stuff in my /etc/profile.d here so i'm not sure where the additional PATH entries could come from ...12:55
ogra_<ogra_> also, my deb is still at 2.27.5 ...12:55
ogra_thats the last i typed12:55
mvoogra_: yeah 2.28.5 is still in proposed12:55
ogra_right12:55
ogra_so i cant have the new stuff yet12:55
ogra_though i do have these PATH entries ... let me try that grep12:55
ogra_ogra@styx:~$ sudo grep -r var\/lib\/snapd\/snap\/bin /etc12:56
ogra_ogra@styx:~$12:56
ogra_nope12:56
ogra_/etc/nevironment looks normal too ... nothing in /etc/bash* or so ...12:57
mvopedronis: you are right (of course :) - we download and then seek to zero and take the hash. then we rename. so the on-disk data was valid. unless of course we never read back from disk but instead linux just gave us what was in the memory cache. i.e. the bits on disk were wrong but because the file was fully in the cache we never read it back12:57
pedronismvo: we do in validate-snap12:57
pedronismvo: it's not a fun bug, unless popey has more general filesystem/disk issues12:58
mvopedronis: could it be the same problem though? I mean, could it be that because its all in the linux disk cache we did not detect that the on-disk bits are wrong?12:59
mvopedronis: i.e. linux itself never hit the disk to read it back in (because it was in the cache already)?12:59
pedronismvo: could be, but darkest fear would be some obscure bug when the kernel decide to write back to the squashfs for reason12:59
mvopedronis: yeah, that would be *wuuuarh* :)13:00
pstolowskistandup?13:02
mupPR snapcraft#1641 opened: lxd: catch CalledProcessError in _container_run <Created by kalikiana> <https://github.com/snapcore/snapcraft/pull/1641>13:10
zyga-solusmvo: your PR fails tests13:23
zyga-solusmvo: I think some tests failed (unit test)13:28
zyga-solusdoes go test in systemd/ works?13:28
mvozyga-solus: checking13:28
pedronismvo:  this is topic: https://forum.snapcraft.io/t/snapcraft-unable-to-install-core-snap/2599 btw13:29
mvopedronis: thank you13:34
ryebotsnapcraft.io down?13:43
mvoryebot: looks like it :/13:45
ryebotkk thanks13:45
mvoryebot: looks like more infra is affected :/13:48
ryebotyeah, just failed to hit jujucharms as well13:48
ryebotwhee13:48
zyga-solusdarn13:50
zyga-solustests cannot run now13:50
mvoweeeh :(13:50
zyga-solusmvo: also does --resend cache core snap?13:51
mvozyga-solus: uh, I don't know13:51
ryebotlooks like things are back up13:58
* kalikiana back later14:05
mupPR snapd#4077 closed: spdx: fix for WITH syntax, require a license name before the operator <Created by matiasb> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4077>14:09
mupPR snapd#4071 closed: release: merge 2.29~rc1 changelog <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4071>14:12
mupPR snapd#4061 closed: daemon, store: forward SSO invalid credentials errors as 401 Unauthorized responses <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4061>14:16
mupPR snapd#4055 closed: daemon: generate a forbidden response message if polkit dialog is dismissed <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4055>14:17
mupPR snapd#4050 closed: cmd/snap: tell translators about arg names and descs req's <Created by chipaca> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4050>14:18
mupPR snapd#4053 closed: tests,po: sync translations from LP and add regression test for LP:1723974 <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4053>14:18
mvoyay, back to 25 open PRs!14:18
mvojdstrand: a second review for 4054 would be great. I applied your feedback14:22
jdstrandmvo: ok14:24
jdstrandmvo: is the xdg-settings one ready for me to review again?14:25
jdstrandmvo: also, I'm going to have another policy updates PR that I'd like to get into 2.29. if I submit to master and 2.29, will that be a problem?14:25
=== chihchun_afk is now known as chihchun
mvojdstrand: the xdg-settings one is not yet ready, need to address the open point about info disclousure14:27
jdstrandmvo: 4054 approved. did you see my question about 2.29?14:34
mvojdstrand: I have not seen your question14:34
mvojdstrand: 4060 is also ready (review points addressed)14:34
jdstrand09:25 < jdstrand> mvo: also, I'm going to have another policy updates PR that I'd like to get into 2.29. if I submit to master and 2.29, will that be a problem?14:35
jdstrand4060 was already on my list for today. let me take a look now14:36
pedronisjdstrand: what impact on preexisting snaps will that policy change have?14:36
jdstrandpedronis: only more access14:38
mvojdstrand: if you submit to master and tag 2.29, that should be fine14:39
jdstrandmvo: ok thanks. need to get out from under PR reviews and forum discussions, then will submit. striving for eod tomorrow14:39
* jdstrand crosses fingers14:40
mvota14:40
mvojdstrand: no worries we have still some critical 2.29 prs pending so no worries14:40
jdstrandmvo: 4060 done (note, see my comment on missing assert)14:46
mvojdstrand: ta14:50
* mvo takes a short break while waiting for tests14:51
=== JoshStrobl|zzz is now known as JoshStrobl
pstolowskipedronis, mvo added help for --long options to 4080, can you take a quick look and let me know wdyt?15:40
pedronisin a bit15:40
=== chihchun is now known as chihchun_afk
pedronispstolowski: commented16:02
pedronisI have no idea if it's ok to refer to systemd there or not16:02
pstolowskipedronis, thanks, I've just fixed caitalization.16:03
pstolowskipedronis, yeah, me neither, but it's difficult not to since we depend so much on i16:03
pstolowski*it16:03
pstolowskimvo, your thoughts on that? ^16:08
zyga-solusre16:36
zyga-solusI'm still AFK, working with kids16:36
zyga-solusjdstrand: hey, how's life?16:36
zyga-solusjdstrand: no updates from me yet, I think we will see a PR tomorrow though16:36
mupBug #1727787 opened: segmentation fault <Snappy:New> <https://launchpad.net/bugs/1727787>16:58
gsilvaptelopio, you around=17:06
gsilvapts/=/?17:06
elopiogsilvapt: yes. Hello17:09
gsilvaptCan I talk to you via pvt?17:09
gsilvaptelopio ^ (I can forgetting to reference people, sorry :P )17:10
elopiogsilvapt: sure.17:13
=== chihchun_afk is now known as chihchun
jdstrandzyga-solus: hey, things going fine here. just trying to get through the PR reviews and forum discussions. how about you?17:24
zyga-solusjdstrand: trying to catch up with my son's classes and all the things he needs to learn in this school; otherwise pondering about overlayfs, I may have made a mistake and not realized it17:48
zyga-solusjdstrand: but no conclusions yet, I just realized something but I didn't finish investigating17:49
=== chihchun is now known as chihchun_afk
mupBug #1727787 changed: segmentation fault <Snapcraft:New> <https://launchpad.net/bugs/1727787>19:50
kyrofasnappy-m-o, autopkgtest 1583 xenial:amd6420:41
snappy-m-okyrofa: I've just triggered your test.20:41
kyrofaelopio, I've seen this error several times now: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-snapcraft-daily/xenial/amd64/s/snapcraft/20171025_182047_d2762@/log.gz20:50
kyrofaplainbox test failing, unable to find pyramid20:50
kyrofaAnd ROS ones, looks like20:51
elopioMmm, let me see...20:51
kyrofaelopio, wait, no-- plainbox has a valid-looking match error20:52
kyrofaLike something changed out from under us20:52
kyrofa2013.com.canonical.plainbox::collect-manifest versus no 201320:52
kyrofaBut the ROS ones suddenly can't find pyramid20:52
kalikianakyrofa: I realized I hadn't actually approved snapcraft#1637 but this is good to go20:57
mupPR snapcraft#1637: repo: add elementary to deb distros <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/1637>20:57
jdstrandstgraber: hey, I see this denial: apparmor="DENIED" operation="open" profile="snap.lxd.lxc" name="/var/lib/snapd/hostfs/usr/lib/os-release" pid=10047 comm="snap-exec" requested_mask="r" denied_mask="r" fsuid=1000 ouid=020:58
jdstrandstgraber: you can fix that by plugging 'system-observe' in the lxc command20:59
nanodroneis snappy installable on puppylinux?21:10
stgraberjdstrand: why would I need that though?21:13
stgraberjdstrand: snap.lxd.lxc runs through "aa-exec -p unconfined"21:13
stgraberjdstrand: the line above seems to indicate that it's "snap-exec" which is attempting to read that file, not our script, so it'd point to a profile issue for snap-exec rather than lxc21:13
elopiokyrofa: pyramid is not on debian/tests/control for snapstests.21:32
elopioWhat I don't know is what changed to require it now.21:32
kyrofaelopio, huh, no kidding21:32
elopiokyrofa: you are adding a dependency on the integration tests with that skip_unless_codename21:45
kyrofaelopio, ohhhhhhhh21:45
kyrofaDangit!21:45
elopioto be honest, I dislike the skip annotations. But, with my refactor on the suites, you would be able to put it in snapcraft/tests/skips, and just import that.21:46
elopioactually, you could do that right now, no need to wait.21:46
kyrofaelopio, you mean you don't like the decorators?21:46
elopiofor the plainbox thing, I'll report a bug for joc to see. It's late for him, and I'm not sure if we can just update the expected result.21:47
elopiokyrofa: yes, the decorators.21:47
kyrofaelopio, you'd rather be using self.skipTest or whatever in the body of the test?21:47
kyrofaI feel like that adds code to the test that has nothing to do with the test21:47
elopiothat is more readable to me. I'm not strongly opposing to the decorators, I just don't like them.21:48
kyrofaelopio, so to be clear, you want me to create a new snapcraft/tests/skips.py and put the skip code in there, then import from there for both the integration_tests and the units?21:54
elopiokyrofa: yes, I think that's the future. Things common will go in there, things specific will go in snapcraft/tests/unit or snapcraft/tests/integration.21:55
kyrofaelopio, good deal21:56
kyrofaelopio, and yeah, that was my favorite part of your refactor21:56
elopiomy favorite part is when we get rid of snaps_tests completely. I will use this chance to move the plainbox test.21:57
elopiokyrofa, can you land snapcraft#1630 ?21:58
mupPR snapcraft#1630: tests: allow to select a suite when running autopkgtests <Created by elopio> <https://github.com/snapcore/snapcraft/pull/1630>21:58
elopioI want to start experimenting with the adt-lxd.21:58
kyrofaelopio, sure thing-- done22:06
elopio:D22:07
mupPR snapcraft#1630 closed: tests: allow to select a suite when running autopkgtests <Created by elopio> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/1630>22:09
mupPR snapcraft#1637 closed: repo: add elementary to deb distros <Created by kyrofa> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/1637>22:09
kalikiana3 more to go til we can release :-D https://github.com/snapcore/snapcraft/pulls?q=is%3Aopen+is%3Apr+milestone%3A2.3522:18
kyrofakalikiana, still need autopkgtests to pass...22:23
kyrofaplainbox seems busted22:24

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!