[01:23] Hey folks, just wondering if snaps are supposed to be apart of your $PATH by default or if there's some sort of setup steps I'm missing before using CLI snaps [03:51] jaitaiwan: /etc/profile.d/apps-bin-path.sh on ubuntu (from snapd) [06:06] morning [07:22] PR snapd#4800 closed: cmd/snap: in changes and tasks, default to human-friendly times [07:30] good morning [07:31] jaitaiwan: hey, it's supposed to be automatic but if you just installed the snapd package you may need to reboot / log out for the changes to take effect [07:32] mvo: good morning [07:32] * zyga-ubuntu was up at 6AM but $KIDS needed more help today [07:33] zyga-ubuntu: hey, good morning [07:33] man, this bug is more elusive than we thought :/ [07:34] zyga-ubuntu: yeah, I was thinking it was caching [07:34] zyga-ubuntu: mvo: morning guys [07:34] mvo: did you get to a point where you can reproduce it on your machine? [07:34] hey mborzecki [07:34] zyga-ubuntu: I did manage that [07:35] mvo: and if so did you have to run all of main or was a subset sufficient? [07:35] mborzecki: hey! :) [07:35] zyga-ubuntu: I have to run all [07:35] mvo: I tried just the cgroup tests and I was "lucky" [07:35] zyga-ubuntu: I managed to get into a shell last night and changed the profile and reload and then things were good again [07:35] mvo: so just *one* test (with sub-tests) [07:35] zyga-ubuntu: I suspect its something subtle like the cache [07:36] ok, I can try disabling all of the cache [07:36] zyga-ubuntu: oh, just the cgroup tests, cool [07:36] and see if that helps [07:36] zyga-ubuntu: sounds great, how do you disable the caches? [07:36] zyga-ubuntu: I rm -f them in my PR but that is not enough [07:36] mvo: apparmor_parser explicitly controls this so we'd have to pass some options in a few places, the patch should be minor [07:36] mvo: the only exception is /etc/init.d/apparmor [07:37] but I _think_ we can ignore that [07:37] because it just runs once on boot [07:37] hmm hmm [07:37] but then again, some tests do reboot [07:37] I'll focus on the cgroup test [07:37] and on disabling cache [07:37] maybe it will show us something usefil [07:37] oh [07:38] zyga-ubuntu: is there magic to dump a profile from memory? i.e. to see if the in-memory state matches the on disk state? [07:38] and I have a small patch, though not a bug fix, related to what I did yesterday [07:38] yes [07:38] ish [07:38] you can dump the profile from memory [07:38] hold on, I will show you how [07:38] zyga-ubuntu: cool, when I am in the broken state I will try that [07:40] mvo: it's not easy or useful without some extra tools [07:40] mvo: you can go to /sys/kernel/security/apparmor/policy/raw_data/ [07:40] (raw_data is a magic symlink which behaves as a magic directory) [07:40] then you will see a lot of numbers [07:41] each time you insert a new profile you get a new number (including replaces I think) [07:41] then you can see the raw_data file inside the number directory [07:41] now [07:41] this doesn't match the cache I think [07:41] but there is a way to compile a profile and get the raw data [07:41] from apparmor_parser that is [07:41] so with some tooling we might compile all the profiles from disk [07:42] to get their raw_data [07:42] and compare that to the raw_data that the kernel knows about [07:42] and in the end try to match files to loaded profiles [07:42] I wrote a decompiler for the raw_data format a while ago [07:42] but did not manage to get the most interesting part of the policy [07:42] that is the set of regular exressions and matching permissions [07:43] I do get profile names and a lot of other things out though [07:43] the format is not documented well, I did this by following the kernel code that parses the binary profiles [07:43] so, that's that, not sure if useful [07:44] zyga-ubuntu: uh, ok [07:44] *fun* :-) [07:44] zyga-ubuntu: yeah [07:44] the format is not hard either [07:44] it's mostly tagged strutures and some tables [07:44] the devil is in the details [07:44] and the yet-not-understood (by me) format to which the DFAs are compiled [07:44] it's also coupled with some compression [07:46] mvo: though having such a decompiler would be very useful as it's our backbone in many ways [07:48] * mvo nods === pstolowski|afk is now known as pstolowski [07:53] mornings [07:54] hey hey [07:55] zyga-ubuntu: so you reproduced it without the core transition tests? I suspected those might be the culprits :/ [07:55] yes, I did [07:55] I'm running another pass now [07:55] I'm tweaking prepare/restore debugging [07:56] zyga-ubuntu: ta [08:00] PR snapd#4821 opened: tests: define MATCH from spread [08:01] zyga-ubuntu: I replied to your question about the system-key [08:01] zyga-ubuntu: let me know if that is reasonable, if not happy to have a quick HO to discuss [08:01] thanks, looking [08:03] zyga-ubuntu: my theory is essentially that during one of the tests we generate/load a snap.core.*.snap-confine profile from the real core in edge instead of from our modified core [08:03] zyga-ubuntu: and never undo this [08:03] hmm hmm [08:03] zyga-ubuntu: and then things fall apprt [08:03] can you explain when system key would be stale and we would not regenerate it? [08:03] zyga-ubuntu: and also its super confusing because we restore the profile *on-disk* (via tar) but of course do not reload [08:03] ahh [08:03] zyga-ubuntu: the core snap is 4422 (or whatever) [08:03] zyga-ubuntu: that is the one in edge [08:04] maybe as a part of restore we can do apparmor_parser -r /some/path/* [08:04] zyga-ubuntu: and also the one we "hack" [08:04] zyga-ubuntu: yes, that could be it [08:04] I just ran the cgroup tests and they didn't fail [08:04] your patch from yesterday (the 1st one) makes it less likely things break [08:04] zyga-ubuntu: the core transition tests will (most) definitely do that, use the "real" edge with the "wrong" snap-confine profile and load that [08:04] zyga-ubuntu: yeah [08:05] I'll run the core transition tests and the cgroup test in sequence now [08:08] mvo: so to get back to the PR with the system-key question [08:08] zyga-ubuntu: I suspect that we will see the error now in the 2.32 branch only, edge is up-to-date with core again so the apprmor profiles are essentially the same [08:08] zyga-ubuntu: sure [08:08] if I understand you correctly, you are saying that because we remove system key we trigger profile loads [08:08] zyga-ubuntu: correct [08:08] yeah, I work on the release branch all the time [08:09] mvo: I would strongly prefer if this was a bit less magic and more organized [08:09] zyga-ubuntu: we *could* also force this in restore as a big hammer [08:09] there's a lot of voodoo in our scripts [08:09] zyga-ubuntu: I think we are all in agereement about this :) [08:10] zyga-ubuntu: part of the problem is that the problem domain itself is confusing, I mean, re-exec make things confusing and we do some strange things in the suite [08:10] zyga-ubuntu: if we could rely on snapshots, that would be best [08:10] yeah, no question about that, a snapshot (though at a right time, this is tricky too) would be wonderful [08:11] zyga-ubuntu: or some way to ensure that prepare/restore really go back to a pristine state but given the amount of things that could change (apparmor profiles in memory, kernel bootloader vars, on-disk state) this is really hard [08:11] ahem [08:11] zyga-ubuntu: haha [08:11] * mvo hugs zyga-ubuntu [08:11] https://github.com/snapcore/spread/pull/47 [08:11] PR spread#47: Add support for project-wide measure-each stanza [08:11] zyga-ubuntu: do you feel better today? [08:11] mvo: yes, less coughing :) [08:11] zyga-ubuntu: I remember this one! [08:11] I wish we could just land it [08:11] could we dump the profiles, clean the state in prepare/restore each steps instead of just the selected tasks? [08:12] zyga-ubuntu: btw when I use less leaves from your tea its not so strong, I like this better [08:12] mborzecki: I think part of the problem is how our prepare/restore is organised, it's not really prepare and restore [08:12] prepare does restore internally [08:12] and restore is mostly a no-op [08:12] mborzecki: I think the profile restore is something to explore, I'm try to reproduce now again and then will add it [08:13] zyga-ubuntu: yeah, this part is also slightly terrible [08:13] mvo: yeah, I also use the same set of leaves throughout the day, it's easily good enough for 2-3 infusions [08:13] mvo: I'm fixing that now [08:13] zyga-ubuntu: \o/ [08:13] mvo: the MATCH pr is the first step now [08:13] chipaca had this idea some months ago [08:14] mvo: I'm making it so that prepare really prepares (applies tarball) and restore really cleans up [08:14] using the measure feature to ensure it's happening for real [08:14] though it will be a few PRs to review [08:14] mvo: what I found interesting is places that are not covered by the tarball [08:14] like various caches [08:14] like the apparmor cache you started cleaning now [08:15] afair apparmor has an in kernel db so even if you clean the cache, the rules may still be there right? [08:15] yes [08:15] totally [08:15] that is another thing we should clean [08:16] but it's not trivial as we really want to restore the right state [08:16] not discard it [08:16] part of the state is "competed" by the Apparmor init script [08:16] it will parse, compile, cache and load all the profiles [08:16] clean by rebooting :) [08:16] haha [08:16] yeah ;) [08:16] mvo: I ran core transition and security cgroups and it passed :/ [08:17] (I don't have your branch in yet) [08:17] hey chihchun_afk :-) [08:18] mvo: would the edge being really newer affect the failure rate in the release branch? [08:19] zyga-ubuntu: otoh reboot is pretty much instant with the cloud images running under qemu [08:19] is there a way to search for free software snaps? [08:21] itsfemme[m]: yes though it's not exposed in the UI yet AFAIK [08:22] got a URL? [08:22] no [08:22] it's just implemented on the store now [08:22] zyga-ubuntu: yeah, the fact that edge is now in sync (more or less) with master means even with the "wrong" profiles being loaded things are ok because the profile in edge is pretty much identical [08:22] zyga-ubuntu: I have some hopes in release/2.32 but its only on 22/232 and ok so far [08:22] zyga-ubuntu: I run one with just the cgroups now [08:22] mvo: can we do an experiment that would not affect master [08:23] mvo: publish 2.31 in some channel [08:23] and I can run tests against that channel [08:23] I want to nail the issue, not hide it till next time [08:24] zyga-ubuntu: again, we are in agreement, lets nail it [08:24] zyga-ubuntu: I would need a track to publish 2.31 (or a branch). I think I need store support for this [08:24] yes, [08:25] perhaps pedronis and noise][ can arrange that [08:29] PR snapd#4815 closed: tests: fix re-exec profile generation in tests on classic [08:34] I cannot create tracks, but for sure store can help with a track or branch [08:35] do i want to think about why downloading snaps is way slower inside a lxd than outside? [08:37] proxy settings not used? [08:37] how much slower is slower? === LtWorf_ is now known as LtWorf [08:40] mvo: do you want to keep https://github.com/snapcore/snapd/pull/4814/files ? [08:40] PR #4814: tests: fix re-exec profile generation in tests on classic (2.32) [08:41] zyga-ubuntu: for now, I want to see if it breaks and in what way [08:41] ok [08:41] zyga-ubuntu: I will set it to blocked if the tests pass [08:41] zyga-ubuntu: I think the first commit is actually correct and should land but the rest is dubious [08:41] zyga-ubuntu: also, the first commit is not sufficient (as we established) so :( [08:46] anyone wants to take a look at #4781? [08:46] PR #4781: wrappers: refactor desktop file sanitizer to support autostart files [08:48] zyga-ubuntu: yay, now I have the bug in the 2.32 PR [08:48] !! [08:48] where? [08:49] zyga-ubuntu: in the fix-reexec-snap-confine-profile-generateion (2.32 based) branch [08:49] zyga-ubuntu: in canonical-livepatch [08:49] zyga-ubuntu: the on-disk profile is correct AFAICT [08:49] zyga-ubuntu: but I get an error [08:50] if you reload the profile [08:50] is that fixing things? [08:50] I assume you have a shell now [08:50] zyga-ubuntu: just reloading does not help [08:50] ok, what is the denial you see [08:50] zyga-ubuntu: *but* changing the profile (adding a whitespace) and reloading [08:50] zyga-ubuntu: fixes it [08:50] exactly the same as before? [08:50] oooooooh [08:51] so caching? [08:51] zyga-ubuntu: yeah, looks very much like it [08:51] head.wall() [08:51] zyga-ubuntu: its "funny", I had this discussion back in the click days iirc, that the cache should use more data than just the mtime :/ [08:52] zyga-ubuntu: I suspect it is something like "a new aa profile is written; profile is cached; we restore to the previous aa profile in the restore step; the apparmor-parser sees that the mtime of the cache is nwewer than the mtime of the profile -> does nothing" [08:53] yeah [08:53] that's very likely [08:53] zyga-ubuntu: the question is how to nuke all apparmor caches [08:53] one sec [08:53] zyga-ubuntu: i.e. what to do short term, then what to do longer term [08:53] I have a patch for the nuclear option [08:54] zyga-ubuntu: bring it on! [08:56] pedronis: are you working on a backport of the refrsh hold PR that was recently merged? or shall I have a look? [08:56] pedronis: fwiw, 2.32 is blocked right now on this apparmor caching bug zyga-ubuntu and I are chasing [08:57] mvo: I will work on it, I forgot to squash it though, so it will be a bit annoying [08:58] mvo: it needs a follow up anyway, with the actual policy (which is what I'm working on right now, I had a variant yesterday but I had some ideas to simplify it a bit) [08:59] pedronis: cool, please let me know if I can help in any way [09:03] mvo: https://github.com/snapcore/snapd/pull/4822 [09:03] PR #4822: interfaces/apparmor: skip apparmor cache [09:03] mvo: just for consideration [09:03] I wonder what will happen [09:03] PR snapd#4822 opened: interfaces/apparmor: skip apparmor cache [09:04] zyga-ubuntu: lets run that against 2.32 and we will know :) [09:04] ah, sorry, I'll make a 2.32 variant [09:09] mvo: the backport is https://github.com/snapcore/snapd/pull/4823 [09:09] PR #4823: interfaces/apparmor: skip apparmor cache [09:09] PR snapd#4823 opened: interfaces/apparmor: skip apparmor cache [09:09] zyga-ubuntu: cool, lets see what the tests say [09:10] just in case the smoking gun is found [09:10] what do we do [09:10] fix apparmor parser caching ourselves? [09:10] zyga-ubuntu: its mostly test related, I think we fix the tests and "gently" remind the team that the apparmor caching could need some love [09:11] zyga-ubuntu: I suspect https://bugs.launchpad.net/snappy/+bug/1460152 [09:11] Bug #1460152: apparmor cache not updated when apparmor.d rules change (breaks 15.04/stable -> 15.04/edge updates) [09:11] mvo: I recall we had a spike on error.u.c [09:11] could that be related? [09:12] zyga-ubuntu: maybe, but I suspect its different reason, this looks tests related to me still (but maybe I'm wrong) [09:13] on a related note, can you look at https://github.com/snapcore/snapd/pull/4821 [09:13] PR #4821: tests: define MATCH from spread [09:13] it will help me land small refactors to prepare-restore next [09:22] zyga-ubuntu: sure, looking [09:23] zyga-ubuntu: why tail -n +2 ? [09:23] zyga-ubuntu: what is in the first two lines? [09:23] thank you [09:24] https://www.irccloud.com/pastebin/4E4oqQqV/ [09:29] zyga-ubuntu: aha, I see [09:30] zyga-ubuntu: replied in the PR [09:34] 
replied as well [09:34] ta === __chip__ is now known as Chipaca [09:42] mvo: I see an error [09:42] but it makes no sense [09:43] + MATCH gadget [09:43] + snap list [09:43] error: pattern not found, got: [09:43] Name Version Rev Tracking Developer Notes [09:43] core 16-2.31.2+git610.7a79b84 4229 edge canonical core [09:43] this is in https://github.com/snapcore/snapd/pull/4822 [09:43] PR #4822: interfaces/apparmor: skip apparmor cache [09:43] PR snapd#4768 closed: [RFC] snap userd autostart v2 [09:43] mvo: shall I restart? [09:43] those were in master [09:43] (not in the 2.32) [09:44] zyga-ubuntu: yeah, master is probably not it [09:45] does the error make any sense to you? [09:45] zyga-ubuntu: does apparmor have more caches except /etc/apparmor.d/cache - do you happen to know? [09:45] very unhappy with the tests state right now :-( [09:45] zyga-ubuntu: no [09:45] yes [09:45] it has two caches [09:45] one legacy [09:45] Chipaca: we are as well [09:45] and one "modern" [09:45] * Chipaca hugs everybody [09:45] one is in /var/apparmor/cache [09:45] Chipaca: we are head first diving into it and its not pretty [09:45] * Chipaca is still grumpy tho [09:45] Chipaca: rightfully so [09:45] another one is in /etc/apparmor.d/cache AFAIR [09:45] hey Chipaca :) [09:46] zyga-ubuntu: thanks [09:46] why are you grumpy? [09:46] zyga-ubuntu: tests breakage [09:46] Chipaca: hug the tests [09:46] :D [09:46] zyga-ubuntu: my huggage would be to feature flag errything [09:47] so maybe not the best approach [09:47] mvo: any ideas? [09:47] (caches in /etc are "fun") [09:48] mvo: shall we restart that PR/ [09:48] zyga-ubuntu: works for me [09:48] zyga-ubuntu: I'm trying different things in parallel but its seems to be all not quite fixing it :( [09:48] zyga-ubuntu: I think I am getting annoyed as well [09:49] Chipaca: master at least should be more happy now, no? tests there? [09:49] mvo: that's good, this is a great motivation technique [09:49] Chipaca: because we got a new core overnight [09:49] mvo: let's merge my MATCH pr (with changes if you wish) and let's reorganise prepare-restore [09:50] mvo: dunno master; #4820 has been red five times already, for everything from layouts to cgroups to network [09:50] PR #4820: cmd/snap: use timeutil.Human to show times in `snap refresh --time` [09:50] mvo: and the 2.32 PR failed now [09:50] Chipaca: oh, right - layout will still be a problem [09:50] cannot execute snap-update-ns: Permission denied [09:50] Chipaca: but cgroups should be ok now, did that run today? [09:50] :-( [09:50] I don't know what's going on now [09:50] zyga-ubuntu: meh, that is exactly the failure that I think comes from a stale cache [09:51] mvo: hold on, maybe the --skip-cache does something silly [09:51] * zyga-ubuntu looks [09:51] mvo: it might've been late yesterday, or today, one side or the other of the date line [09:51] -K, --skip-cache [09:51] Perform no caching at all: disables -W, implies -T. [09:51] zyga-ubuntu: also maybe for some reason aa thinks the profile has not changed? [09:51] Chipaca: ok, just for "fun", please restart [09:51] mvo: but without any caching how could it decide? [09:52] Chipaca: if it still fails in the cgroups I will consider start drinking [09:52] mvo: one more idea [09:52] mvo: i did (and then it failed on another thing) [09:52] zyga-ubuntu: I have no idea [09:52] actually, not sure [09:52] Chipaca: aha, what is this other thing? [09:52] I wanted to say that we could wrap apparmor_parser [09:52] zyga-ubuntu: ohhhh [09:52] oh? [09:52] zyga-ubuntu: you probably need my first commit [09:52] zyga-ubuntu: wait a sec [09:52] kk [09:52] just push to those PRs [09:53] mvo: a bunch of unrelated stuff like econnreset [09:53] mvo: and now it's failed yet again, this time in suite prepare [09:53] zyga-ubuntu: I think you need https://github.com/snapcore/snapd/pull/4814/commits/8c065f9f22547497ca0c925009fe1b448ea78045 or you will never have the right profile in the first place [09:53] PR #4814: tests: fix re-exec profile generation in tests on classic (2.32) [09:53] so that's six times [09:53] zyga-ubuntu: i.e. without that commit the snap-confine profile for core is always incorrect [09:53] Chipaca: in suite prepare? [09:53] * Chipaca gives up on this programming stuff and goes shopping [09:53] zyga-ubuntu: with that commit it becomes a caching issue (I think) [09:54] Chipaca: in prepare suite :( [09:54] Chipaca: what is the link to the log? [09:54] mvo: will you push or do you want me to cherry-pick? [09:54] https://api.travis-ci.org/v3/job/352504456/log.txt [09:55] + tar -C/ -xf /home/gopath/src/github.com/snapcore/snapd/snapd-state.tar.gz [09:55] tar: /home/gopath/src/github.com/snapcore/snapd/snapd-state.tar.gz: Cannot open: No such file or directory [09:55] I have no words [09:55] is there a gnome pissing into the milk [09:55] and messing up our tests [09:55] zyga-ubuntu: s/gnome/elephant/ [09:56] hmm GNOME? :) [09:57] zyga-ubuntu: if you could cherry-pick [09:57] sure [09:59] mvo: pushed, [09:59] do you think it's worth to try to wrap apparmor_parser [09:59] and do caching ourselves? [09:59] we can use apparmor_parser to get the raw data [10:00] and use any logic we wish to cache or discard the raw data (compiled profiles) [10:02] zyga-ubuntu: is the branch just to check if it helps? or is the plan to really turn off the cache? [10:02] pedronis: just to check [10:02] (hence the rfc) [10:02] if the caching system is broken we can work around it [10:02] I think we really could cache ourselves [10:02] since the parser gives us the primitives [10:03] the rfc? [10:03] the description is very not rfcs [10:04] but maybe I'm misunderstanding [10:04] mvo: are you still using that log, or can i nuke it? [10:04] Chipaca: nuke it [10:05] zyga-ubuntu: thats a nice idea, use our own wrapper at least in the tests to understand things better [10:05] ok, I'll get to it [10:06] zyga-ubuntu: but same deal I think all followup work needs https://github.com/snapcore/snapd/pull/4814/commits/8c065f9f22547497ca0c925009fe1b448ea78045 first [10:06] PR #4814: tests: fix re-exec profile generation in tests on classic (2.32) [10:06] (so that the initial profile is ok) [10:07] mvo: is this ok? https://github.com/snapcore/snapd/pull/4823 [10:07] PR #4823: interfaces/apparmor: skip apparmor cache (2.32) [10:07] zyga-ubuntu: yes, can't wait to see the result [10:07] zyga-ubuntu: its also super annoying, spread failed, my local run is still going strong :( [10:08] zyga-ubuntu: for 4814 [10:08] yeah, it's a day that keeps giving [10:12] Son_Goku: hey, I have a working F28 chroot now [10:12] thank you for that! [10:14] zyga-ubuntu: hm, another problem is that we have ~8 workers, so ideally I would like to know what tests run on exactly the worker that failed. but spread does not give me this info, does it? [10:14] no, it does not [10:14] I wish it did [10:14] a sequence of tests on the worker who failed [10:15] zyga-ubuntu: yeah, in the travis run it fails after ~70 tests so with 8 works that is not that many that run on each worker [10:16] mvo: I'll work on a wrapper for the cache [10:16] just a small prototype [10:16] to see where this goes [10:16] ok [10:19] can't we keep tab of the relevant raw_hash and see when it changes? [10:19] pedronis: can you explain please? [10:20] mvo: hi! could you please rebuild and push base-18 to the store again, so that it gets the latest glibc? [10:20] zyga-ubuntu: /sys/kernel/security/apparmor/policy/profiles/usr.lib.snapd.snap-confine.17/raw_hash [10:20] yeah but that's the hash of the (assuming, I didn't check) blob that comes out of the compiler [10:20] how does that help with the parser? [10:21] BjornT_: started a build now [10:21] BjornT_: https://code.launchpad.net/~mvo/+snap/base-18 <- sorry for the delay [10:21] zyga-ubuntu: I assume the parser is deterministic? [10:21] * zyga-ubuntu knocks on wood [10:21] I hope so [10:21] so we can track the values of that and when it changes [10:21] hold on [10:21] when what changes? [10:21] or also try hold of what we expect to be sane [10:22] is the idea designed to optimise the cache or to do something else? [10:22] zyga-ubuntu: when we load the profile for real and the profile is different that should change no? [10:22] pedronis: yeah, we can store the value when the suite starts and fail if the fails, then we can narrow down what test breaks it [10:22] zyga-ubuntu: to understand if we have loaded what we expect [10:22] pedronis: yes, I agree [10:22] mvo: thanks [10:22] fail if the value changes [10:22] sorry [10:23] (this assumes we understand what that value is) [10:23] there are more hashes there [10:23] I don't know what they do [10:23] (yet) [10:23] * zyga-ubuntu is experimenting [10:23] mvo: something like that, to make this a bit tractable [10:23] pedronis: for debugging yeah [10:23] yes, for debugging [10:23] though it's not super clear how to map that, the profiles are loaded into subsequent slots [10:24] and each loaded profile is really a set of profiles [10:24] but I think we can come up with something [10:25] I have an idea [10:25] that we can use to make the cache reliable [10:25] it's essentially the same problem as ccache [10:26] resolve all imports, use some hash of the full text as key [10:26] compile the pre-processed text [10:26] and store that in the cache under that key [10:26] that's it [10:26] ccache for apprmor [10:26] zyga-ubuntu: afaiu raw_hash is the sha1sum of raw_data [10:28] PR snapd#4824 opened: tests: fix re-exec profile generation in tests on classic [10:29] mvo: can you vote on 4821 [10:29] I can make it a stub if that's what you want [10:31] zyga-ubuntu: mvo: piece of good news afaict (here) : apparmor_parser -S of the snap_confine profile |sha1sum produces the same value as raw_hash [10:31] zyga-ubuntu: I'm a bit on the fence, if there is a use case for it, then lets keep it, otherwise I would say make it a stub (YAGNI and all that) [10:31] good [10:31] pedronis: \o/ [10:31] mvo: I can make it a stub [10:32] mvo: I need it for real stuff after that [10:33] mvo: zyga-ubuntu: I mean this: https://pastebin.ubuntu.com/p/4XGYSmk8zt/ [10:33] zyga-ubuntu: ok, if a stub is enough lets do that (if you don't mind) [10:33] yes [10:33] that's fine [10:33] why do we enable the services first and the do a daemon-reload? [10:33] I'll finish the parser wrapper and do it in a moment [10:33] mborzecki: huh? [10:33] that's odd [10:34] https://github.com/snapcore/snapd/blob/master/wrappers/services.go#L244 runs before daemon-reload [10:34] mvo: saw I think we could compute the hash in prepare somewhere and fail when it changes (and decide when is expected and what we need to do) [10:34] s/saw/so/ [10:37] PR snapd#4825 opened: overlord/snapstate: implement policy of holding refreshes for up to 6h since seeding on classic [10:39] BjornT_: core build failed with what looks like a snapcraft error: if ElfFile.is_elf(path): [10:39] File "/usr/lib/python3/dist-packages/snapcraft/internal/elf.py", line 152, in is_elf [10:39] with open(path, 'rb') as bin_file: [10:39] OSError: [Errno 6] No such device or address: '/build/base-18/prime/dev/tty' [10:39] sergiusens: -^ [10:41] PR snapd#4826 opened: wrappers: services which are socket or timer activated should not be started during boot [10:42] zyga-ubuntu: and no build slots right now, a bit frustrating [10:42] I have a wrapper [10:42] man [10:43] mvo: sorry, feel free to cancel the job for my PR, i'll restart it later [10:44] mvo: ah. we saw those errors when building locally. i think it's fixed already in snapcraft master, though. [10:48] mborzecki: no worries, I think cacneling will not even help [10:48] BjornT_: aha, ok [10:48] BjornT_: thank you! [10:49] zyga-ubuntu: https://forum.snapcraft.io/t/race-condition-between-snapd-and-nm-when-providing-permissions/4451 serial devices go though device cgroups too right? [10:49] yes [10:51] mvo: can you please look at https://github.com/snapcore/snapd/pull/4827 [10:51] PR #4827: cmd/snap-apparmor-parser: add a prototype apparmor parser [10:51] I will replace the real stuff with this now, ok? [10:51] and we can implement the cache in a real language next [10:51] PR snapd#4827 opened: cmd/snap-apparmor-parser: add a prototype apparmor parser [10:52] * Chipaca goes for a walk [11:02] zyga-ubuntu: looks good but I think we first need to figure out if that fixes our issues before replace the real thing (i.e. one step at a time :) [11:03] mvo: hold on, how do we know that if we replace the real thing [11:04] I meant to write it and use it to see if this makes anything better [11:04] we don't have to merge it [11:04] s/if we replace the real thing/if we *don't* replace the real thing/ [11:05] do we even know what is broken? [11:05] pedronis: mvo ran a test where the profile on disk was ok, in memory was bad, running the parser did nothing unless a trivial whitespace change was introduced [11:05] this hints at a bug in the cache [11:05] which is based solely on mtime [11:05] well, it would be good to have a reproducer then [11:05] to submit a bug [11:06] it also sounds very bad [11:06] upstream knows about this bug [11:06] we had this conversation a few times now [11:06] zyga-ubuntu, \o/ [11:06] is the mtime a problem for us in general or because the tests do strange things? [11:07] pedronis: I think our tests make the story more complicated [11:07] pedronis: and +100 for a reliable reproducer [11:08] yeah, but I want to say that pedronis is right we don't have a single, well defined problem [11:08] we had a problem with edge version [11:08] how "went away" (not really fixed) [11:08] we definitely have a problem with mtime and cache but it is unclear why [11:08] and we may have more problems we don't know about yet [11:08] I still think that adding logic to the tests that check what we expect to be loaded [11:08] the wrapper is meant to make the mtime problem void [11:08] pedronis: so what I ran into was that I got a denial from "canonical-livepatch status" about it being not able to run snap-device-helper. then I tried appamror_parser -r which did not help, then a whitespace change in the profile and then apparmor_parser -r and things were ok [11:08] vs what is laoded match would halep [11:08] help [11:08] pedronis: unfortunately super hard to get into the right (i.e. broken) state [11:09] mvo: I pushed the stub MATCH [11:09] and I'll merge when green [11:09] anyway if it's naively mtime based [11:09] but first, I need to take a break and take the dog for a walk [11:09] problem should occur only if we go back in time [11:09] zyga-ubuntu: ok [11:09] I suppose our revert might not work [11:09] anyway we could test this stuff out [11:10] pedronis: I have not looked in ages how it works, it used to be mtime based and the pattern I saw indicates it but I did not double check [11:10] mvo, hey, I didn't see any error reported in the forum on candidate [11:10] zyga-ubuntu: good idea wrt making the stub MATCH dummy [11:10] mvo, may I promote to stable? [11:10] zyga-ubuntu: if you visit that file again, maybe make it echo to stderr instead of stdout [11:10] mvo: ok, as I said I just fear that fixing things before understanding them will just add to the confusion [11:10] cachio: yay, then I would say lets just give a quick heads up in the store channel [11:10] cachio: and go [11:10] of which it seems we have plenty [11:10] pedronis: yeah [11:10] mvo, sure [11:11] pedronis: not disagreeing at all [11:11] brb, changing machines around [11:11] pedronis: we found one thing that is broken [11:12] pedronis: but there is more lurking, I added a crude hash compare now to one of my test PRs [11:12] pedronis: essentially what you suggested, maybe that gives us a clue [11:12] pedronis: it hooks into restore so hopefully it gives us a hint what test is changing things [11:13] mvo: -k seems also interesting for debugging [11:15] pedronis: indeed [11:18] anyway the man page is quite explicit about it being time based [11:18] fwiw [11:18] * mvo nods [11:32] mvo: zyga-ubuntu: any problem about merging things to master right now? (finally got green..) [11:33] mvo, well, core promoted to stable [11:33] niemeyer: it seems you changed spread to only build with a go newer than 1.6 and it's breaking autopackagetest [11:34] niemeyer: “go get -u github.com/snapcore/spread/cmd/spread” spits out “package context: unrecognized import path "context" (import path does not begin with hostname)” with 1.6 now [11:40] Chipaca: still Alf [11:40] Afk [11:41] But I think it is ok [11:41] zyga-ubuntu: https://i.imgur.com/ZTNhK7W.png [11:42] lol [11:42] Alf says “let’s eat this cat” [11:43] cachio: thanks [11:43] Chipaca: should be fine [11:44] ack [11:44] PR snapd#4820 closed: cmd/snap: use timeutil.Human to show times in `snap refresh --time` [11:45] mvo, niemeyer I'll be few minutes late today in the standup [11:45] * cachio afk [11:56] re [11:58] Chipaca: can you please look at https://api.travis-ci.org/v3/job/352784798/log.txt [11:59] it looks like logger unit tests panicked [11:59] ahahaha [11:59] zyga-ubuntu: yes i can look [11:59] ah [11:59] sorry, not logger [11:59] osutil [11:59] RunWithContext [11:59] hmm hmm [12:00] zyga-ubuntu: timeout [12:00] zyga-ubuntu: that's a timeout [12:00] zyga-ubuntu: *** Test killed with quit: ran too long (10m0s). [12:00] PR snapd#4788 closed: packaging/fedora: Merge changes from Fedora Dist-Git plus trivial fix [12:01] Chipaca: real bug or shall I restart [12:01] * zyga-ubuntu tries to figure out what the test is [12:01] TestRunRace? [12:04] zyga-ubuntu: yes [12:04] * zyga-ubuntu asks the wrong precise questions [12:04] zyga-ubuntu: this shouldn't happen, but if it happens again let me know; there should be a way to make it exponentially less likely to happen [12:04] I thought this was less-likely-enough :-) [12:05] can you explain what went wrong here? [12:05] we called RunWithContext [12:05] zyga-ubuntu: it times how long it takes /bin/false to run [12:05] zyga-ubuntu: then it calls /bin/false in a loop with a timeout of exactly that time [12:06] zyga-ubuntu: it _should_ fail to finish, at least once, after a little bit [12:06] zyga-ubuntu: and it also _should_ be able to succeed, at least once, after a little bit [12:06] so how come it did not? after 10 minutes [12:06] zyga-ubuntu: it was really (un)lucky the first time [12:06] zyga-ubuntu: or the vm got clamped as soon as it got into a tight loop [12:06] I see [12:07] ok, I'll reboot [12:07] so an easy way would be [12:07] if it happens even one more time [12:07] to do two things in each loop: once with dt/2, once with 2*dt [12:07] if that still sometimes hiccups, change it to 4, etc :-) [12:09] zyga-ubuntu: that test is meant to have some surety around it not failing if the deadline happens at just the same time the command finishes though, which is why it's written as it is [12:10] (that's why the only thing it reports as an error is an unexpected error) [12:11] the more i try to work on 18.04, the less productive it feels and the more i stick with 16.04 :-( [12:11] can no longer alt-space x to maximize a window, for example [12:12] * Chipaca working in the 18 live cd [12:13] Chipaca: how is it? I heard that snapd doesn't work but it was on old build [12:14] zyga-ubuntu: snapd works [12:14] zyga-ubuntu: snaps don't :-) [12:14] Chipaca: SHIP IT ;-) [12:14] zyga-ubuntu: yesterday I was getting the thing about snap-confine escalation yadda yadda [12:15] on the same ISO? [12:15] today i'm getting 'cannot create lock directory' [12:15] Chipaca: do you get an apparmor denial? [12:15] zyga-ubuntu: http://cdimage.ubuntu.com/daily-live/20180313/ vs http://cdimage.ubuntu.com/daily-live/20180312/ [12:15] are you getting a denial on /run/snapd/ns/ stuff? [12:15] aha [12:15] so daily [12:16] Chipaca: oh, you are working on the script? how nice [12:19] zyga-ubuntu: [ 2114.200898] audit: type=1400 audit(1520943269.861:58): apparmor="DENIED" operation="open" profile="/snap/core/4206/usr/lib/snapd/snap-confine" name="/upper/" pid=4392 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [12:19] mvo: trying to [12:19] Chipaca: that's interesting [12:19] mvo: I'm going to have to rebuild it at some point, do you know if there's an easy way of doing that? [12:19] Chipaca: can you look at /var/lib/snapd/apparmor/snap-confine [12:19] and can you please paste system-eky [12:19] *key [12:20] Chipaca: unsquash, dpkg --extract [12:20] Chipaca: _probably) [12:21] zyga-ubuntu: which system key? [12:21] Chipaca: in /var/lib/snapd/system-key [12:21] zyga-ubuntu: /var/lib/snapd/apparmor/snap-confine is empty [12:21] Chipaca: that's expected given the symptom [12:22] it says that there's no overlay workaround in place [12:22] zyga-ubuntu: /var/lib/snapd/system-key does not exist [12:22] if it's also not in the system-key then either you have an old snapdo [12:22] or the logic is broken [12:22] * zyga-ubuntu looks [12:22] 2.31.2 [12:22] that's too old :/ [12:22] you need master or 2.32 [12:22] I wasn't expecting it to work given just yesterday you were saying it was only fixed on master [12:22] I wonder why there's no system-key though [12:22] 2.31 should have it [12:27] * Chipaca takes a break from 18.04 and looks at lunch [12:27] lunch sounds nice [12:27] I'm preparing a prototype of snapd-side apparmor cache [12:28] zyga-ubuntu: cuban black beans sounds a'ight today, with it being dull outside [12:29] pedronis: if you have a bit of time I could use a look at #4790 (which purports to fix #1750527) [12:29] PR #4790: jsonutil/puritan: introducing puritan.String & etc [12:32] PR snapd#4828 opened: many: support holding refreshes by setting refresh.hold (2.32) [12:33] mvo: I made the backport ^ needs to go together with #4825 (once landed in master) [12:33] PR #4825: overlord/snapstate: implement policy of holding refreshes for up to 6h since seeding on classic [12:33] Chipaca: do you recall if `snap pack --check-skeleton` was to run ValidateContainer too? [12:33] mborzecki: the way we talked about it I imagined ValidateContainer growing a no-fail-on-missing mode [12:35] mborzecki: (or equivalently checking for and ignoring snap.ErrMissingPath when calling ValidateContainer from --check-skellie) [12:36] mborzecki: it'll still _log_ the error though (not sure if that's a pro or a con) [12:36] Chipaca: ok, i like the error checking approach better ;) [12:37] Chipaca: I'll try to look at 4790 later today, 4825 from me also needs a review [12:37] pedronis: thanks [12:37] I'm grabbing lunch now, will look at it after [12:43] PR snapd#4829 opened: store: Sections and WriteCatalogs need to strictly send device auth only if the device has a custom store [12:46] mvo: hey, +1 to merge 4821 now? [12:47] * zyga-ubuntu finishes lunch [13:00] zyga-ubuntu: sounds good [13:01] PR snapd#4821 closed: tests: define MATCH from spread [13:29] mvo: interesting, on a 2.32 branch that passed --skip-cache to apparmor_parser I got the denial again [13:29] and this PR includes your patch for re-exec profile g eneration [13:29] I will look at apprmor_parser now [13:29] maybe it ignores that flag [13:29] this is https://github.com/snapcore/snapd/pull/4823 [13:29] PR #4823: interfaces/apparmor: skip apparmor cache (2.32) [13:30] it definitely doesn't ignore the --skip-cache flag [13:31] tyhicks: hey [13:31] so something else is broken [13:31] I use --skip-cache regularly and I also know that there's quite a bit of test coverage for that in parser/tst/caching.py [13:31] tyhicks: I'll try the snapd-side-cache prototype next [13:31] hey zyga-ubuntu :) [13:31] thank you for confirming that [13:32] tyhicks: I wrote https://github.com/snapcore/snapd/pull/4827 as a shoot-in-the-dark (ish) on the issues we are seeing [13:32] PR #4827: cmd/snap-apparmor-parser: add a prototype apparmor parser [13:32] zyga-ubuntu: is there a short writeup on the issues you're seeing? [13:33] (I need to finish publishing a security update before I get too involved here) [13:33] tyhicks: we see an apparmor denial on snap-confine, the rest is hazy but in one case we confirmed that calling apparmor_parser --replace did _not_ fix the issue without making a no-op whitespace change. [13:34] tyhicks: it's probably an error related to just our testing process but it's keeping a few people busy as tests are broken. [13:37] zyga-ubuntu: have you been able to reproduce the issue manually, outside of your testing process, or are you stuck having to tickle the issue via the testing process? [13:37] it's inside the testing process [13:38] and it's not super reliable to reproduce because one factor changed [13:38] we'll try to restore that factor so that we can really understand and nail the issue [13:38] if we nail the issue I'll tell you what it was for sure [13:40] ok, let me know if I can answer any questions or help wade through apparmor_parser code [13:40] thank you [13:58] zyga-ubuntu: does http://paste.ubuntu.com/p/4TcqxcRdgh/ ring a bell? [13:59] yes [13:59] zyga-ubuntu: I see this on 14.04 and the profile does not allow running update-ns from core snap [13:59] it looks like we are in the system where we try to run snap-update-ns without profile transition [13:59] do you have a debug shell? [14:00] can you look at the profile for snap-confine from core [14:00] what's at the bottom? [14:00] zyga-ubuntu: I am [14:00] is is the rule -> snap-update-ns.* [14:00] it looks like this thing we had on Thursday [14:00] right? [14:00] zyga-ubuntu: http://paste.ubuntu.com/p/B2fXb62yy4/ [14:01] we didn't get to the bottom of that problem [14:01] this is the wrong profile :( [14:01] /var/lib/snapd/hostfs/usr/lib{,exec,64}/snapd/snap-update-ns Cxr -> snap_update_ns, [14:01] this is the old code, we changed that in 2.32 and master now [14:02] can you look at the profile in the core snap [14:02] is it the same? [14:02] mvo: to be clear, I'm saying this looks like a 2.31 profile [14:09] hi all, I need some help with snapcraft, not sure if this is the right channel [14:10] is there anyone online? === pstolowski is now known as pstolowski|afk [14:17] dan_it__: yesish [14:18] i'm off to pick up the kids [14:18] dan_it__: things are a little hectic today [14:28] PR snapd#4830 opened: many: generate and use per-snap snap-update-ns profile (2.32) [14:28] that's the backport [14:28] I'll work on layout fixes noww [14:32] zyga-ubuntu: ta [14:34] dan_it__: The forum is always a good place.. lots of eyes, and async so people respond as soon as they can [14:34] Chipaca: hangout? [14:34] omw [14:34] niemeyer: standup one? [14:34] Yeah [14:44] Hi [14:44] coffee [14:45] What variables are available in snapcraft.yaml [14:45] ? [14:46] And how to install fonts into package? [14:51] Muravey: we have a very very busy week, can you please look at the forum, there are (probably) people with the same desires as you now [14:51] Muravey: you can also open a new topic and ask your questions there [14:53] zyga-ubuntu: ok..sorry [14:53] Muravey: no worries, it's not your fault we're busy today, just ask your questions there as there are more than just the snapd and snapcraft teams participating there [14:53] the whole snapcraft community is there [14:53] making and using snaps [14:53] it's the best place to talk [15:16] mvo: 4830 is green now [15:16] mvo: or, green from 1st try [15:19] zyga-ubuntu: green on first run? woah [15:19] it's magic [15:19] also [15:19] it never uses stale profiles [15:20] actually, it would still use stale profile for the transition [15:20] zyga-ubuntu: does it keep changing what is loaded? it must do something different [15:20] it does change things but it would still be affected by a stale profile [15:22] because it cannot do a transition without confinement permission [15:25] zyga-ubuntu: then maybe it was a lucky PR [15:25] zyga-ubuntu: still strange [15:25] maybe [15:26] I suspect there's something to it though [15:26] yeah I think so too [15:26] zyga-ubuntu: maybe it is because now edge and 2.32 are close? [15:26] shall we land it? [15:26] zyga-ubuntu: no [15:26] zyga-ubuntu: please not [15:26] yes, I suspect something like that [15:26] ok [15:27] zyga-ubuntu: lets wait a wee bit, I guess we could land it and then use 2.31 as the testcase [15:28] 2.31 as a test case? what do you mean? [15:29] zyga-ubuntu: running release/2.31 with the edge core should also show the bug [15:29] interesting [15:29] does it? [15:29] zyga-ubuntu: i.e. we probably don't need 2.32 for this [15:30] zyga-ubuntu: I have not tried yet :) but I would be surprised if not [15:30] careful what you wish for :) [15:30] zyga-ubuntu: haha [15:30] oh man [15:30] 16:30 [15:30] zyga-ubuntu: well, if the bug shows itself for core snap-confine profile != our snap-confine profile [15:30] and I forgot about the coffee [15:30] zyga-ubuntu: then… [15:30] yes [15:30] zyga-ubuntu: heh, go for it! [15:30] it will be a second [15:32] zyga-ubuntu: no worries - fwiw, test-classic-firstboot is another one of the tests that will use a unmodified core and install it. all these tests will trigger the issue I bet. /me tries [15:35] mvo: keep me posted, I'm curious [15:36] zyga-ubuntu: trying this now, hopefully its a minimal testcase [15:38] zyga-ubuntu: aha, maybe 2.31 is not enough - we need the system-key to trigger the bug I think [15:38] * mvo tries that too [15:43] Issue snapcraft#2001 opened: node-engine does not support NodeJS 8.X.X [15:45] mvo: that's fixed on edge, the 2.40 release will be this week which fixes that [15:45] hey sergiusens how are you [15:45] sergiusens: cool, thank you [15:45] did you get home safely and without surprises? [15:46] zyga-ubuntu: doing well. I am home, no further surprises or incidents [15:58] niemeyer: what does 'spread -list' do, to take 2s to print out the list? [15:58] * Chipaca is too lazy to go look [15:59] niemeyer: if that could take an order of magnitude less, I'd write a tab-completer for it :-) [15:59] Chipaca: Generates the full matrix of tests, which includes reading and parsing everything [16:09] $GOPATH/bin/spread -v qemu:ubuntu-14.04-64:tests/main/classic-firstboot qemu:ubuntu-14.04-64:tests/main/interfaces-broadcom-asic-control [16:10] zyga-ubuntu: -^ [16:10] zyga-ubuntu: that works for me to reproduce, I'm testing on 16.04 now, also 4814 seems to be good now [16:11] zyga-ubuntu: slightly heavy handed though [16:16] mvo: completely left-field question: is there a way to tell apt to not use /var/cache/apt/archives? [16:18] cachio: yes, run "sudo apt install foo -o dir::cache=/tmp/my-cache [16:19] mvo, it was for Chipaca perhaps [16:19] Chipaca: you may need to create archives/partial in there [16:19] cachio: ups [16:19] Chipaca: -^ [16:19] heh [16:19] mvo: gotcha [16:19] Chipaca: why? [16:20] mvo: chatting with a guy just now and they were talking about how they have a load of space wasted in their vms because of that dir [16:20] mvo: an always-do-apt-clean would probably also work [16:21] Chipaca: yes [16:21] Chipaca: also unattended-upgrades cleans properly after itself [16:21] mvo: is there a way to do the latter? :-) [16:23] Chipaca: not right now it seems, would be very easy to have a "apt::tidy=1" option that does that automatically [16:23] Chipaca: in fact, "ulink" right after the deb was installed sounds prudent [16:25] mvo: thanks [16:26] Chipaca: sounds like a fun evening patch, please remind me, I think it will be a good way to forget this apparmor profile drama I'm in right now [16:26] * Chipaca hugs mvo [16:29] hi fireclawTheFox , hows the snap coming along? [16:31] Hey popey, haven't done much yesterday and just came back from work. But yeah so far it looks good everything works. Just need to see if I can lower the size again, I think it got a few things that doesn't need to ship. [16:32] ooh, we have done some of that optimisation in other snaps [16:32] mvo: re (dog), so does it mean you still see the issue but now have a better way of reproducing it? [16:33] zyga-ubuntu: mvo: FWIW my new laptop is nvidia (prime, so it has an intel mode as well), running 16.04, if you ever need to test something there (not 18 because lots of oem stuff) [16:33] !!! [16:33] wooot [16:33] perfect [16:33] first step [16:33] update to 18.04 :D [16:33] just kidding [16:33] zyga-ubuntu: i would, but even if i didn't hate it, i'd still not have some of the hardware [16:33] but some 2nd disk (usb is fine) to boot into 18.04 once in a while would help [16:34] ah, that i could do [16:34] Chipaca: I think you can reproduce the GLVND issue easily then [16:34] Chipaca: and you could beta-test fixes [16:34] i had to choose between a second disk and more battery (and i chose battery of course) [16:34] nah, I mean external disk [16:34] even a usb thumb drive [16:34] just for testing at home [16:34] no need to affect your daily experience [16:34] :+1: [16:35] you can install on it, if you are careful that is (don't overwrite your boot records) [16:35] are you on the nvidia driver now? [16:35] yes [16:35] and if so, which version is it? [16:35] does Minecraft snap work? [16:35] fireclawTheFox: I find doing "ncdu /snap//current" is a good way to see what's taking up space in the snap [16:35] zyga-ubuntu: 384.111 [16:36] popey: ah, thanks will try. [16:36] Chipaca: that's good, I think you are the missing link :-) [16:36] Chipaca: I wish you had this machine last week, I'd love to see it :) [16:36] minecraft snap installing [16:36] \o/ [16:36] zyga-ubuntu: it arrived at 11am on the monday [16:36] Chipaca: murphy strikes [16:36] zyga-ubuntu: i sat all through the sprint knowing this was waiting for me on my bed [16:36] popey: Well, basically I shouldn't need much to add aside of the things that are packed by the engine. I think I just needed some libs like pulse, and xlib stuff. [16:36] i had the jitters [16:37] Chipaca: do you own a copy of Minecraft? [16:37] (disclaimer: it's a fantastic procrastination helper) [16:37] zyga-ubuntu: 3 [16:37] really?! [16:38] zyga-ubuntu: i play with the boys [16:38] nice! :) [16:38] the minecraft launcher works [16:38] (we also own three haha [16:38] try logging in and see if it works [16:38] on 16.04 it should be OK [16:38] now downloading minecraft itself [16:38] so 16.04 is the baseline [16:38] yes it works [16:39] ok, sometime this week it'd be great to try if Minecraft works with nvidia on 18.04 [16:39] do you have a spare disk to install bionic on? [16:39] zyga-ubuntu: pendrives, yes [16:40] cool [16:40] actual disk, maybe? still usb though [16:40] I don't have a fix yet, I'll do one just after layouts (I need two days perhaps, for layouts) [16:40] pen drives are fine [16:41] zyga-ubuntu: also it'll be good to know if it works when in intel mode [16:41] I didn't even consider that [16:41] :-) [16:41] I don't know how the switching works [16:41] i need to log out and back in, so not that magic [16:42] it's not the old voodoo switch thing :-) [16:42] zyga-ubuntu: I thought so but its random and now its no longer reproducible and its all horrible [16:42] aaaah [16:42] so crazy [16:48] zyga-ubuntu: yes, I think I give up for today [16:48] zyga-ubuntu: well, almost, one more run [16:48] mvo: I'm working on something to finish this exercise of the day [16:48] but tomorrow I focus solely on symlink bug and on the writable leak bug [16:48] and nothing else [16:48] so :-) [16:48] unless you ask [16:49] zyga-ubuntu: ok, on the layouts bug, right? [16:49] mvo: so there is still no good reproducer? [16:49] did you ask for a way to use 2.31 as a baseline? [16:49] oh right, did that turn out anything? [16:51] pedronis: I thought I had one (and it all made sense): classic-firstboot installs the real core, that will generate a snap.core.xxx.usr.lib.snapd.snap-confine profile from the "real" core snap (which is out-of-sync with the profile from the branch). that gets loaded into the kernel. now the interfaces-broadcom-asic-control test runs, that runs snap-confine with the wrong profile and boom. except it exploded once and then did not explode even thou [16:51] gh the order is correct (i.e. first classic-firstboot test runs, then the interfaces test runs) [16:51] pedronis: 2.31 has no system-key yet so we always re-generate profiles there so its hidden there [16:52] so it was 2.31 but a old edge? [16:52] mvo: I think we want the track/channel [16:52] and to put 2.31 there [16:52] and test against that [16:52] pedronis: yeah, when we had 2.31 on edge briefly things were really wrong [16:53] zyga-ubuntu: yeah, I think that is a good plan [17:22] Chipaca: I did a pass on that branch, is not super easy to review [17:22] cachio: Any news on https://github.com/snapcore/snapd/pull/4778? [17:22] PR #4778: tests: moving debian 9 from linode to google backend [17:23] cachio: 11 days old, conflicts [17:23] pedronis: sorry... [17:23] pedronis: if it's any consolation mborzecki futzed it and it was a'ight [17:23] (then i broke it, and he futzed it again and found the breakage, so i unbroke it) [17:24] Chipaca: it still feels a bit too fiddly [17:24] pedronis: but also, thank you for the pass [17:26] Chipaca: I mean, I'm not looking forward about expanding this over more fields in the apis [17:26] pedronis: in what sense? [17:27] do we need to use these things in other places? [17:27] or is info the only source of things we print? [17:27] it also feels like all the fields that are not freeform [17:28] should have specific validators and not this [17:28] jjohansen: hey, I have a question about the kernel apparmor interface [17:29] pedronis: agreed on the latter; on the former I'm still not sure what you mean: currently it's checking everything that comes from the store that doesn't have its own type [17:29] pedronis: if you're asking whether we'd need to do this over the things we get from snap.yaml, I don't know [17:29] jjohansen: under what circumstances would /sys/kernel/security/apparmor/policy/profiles/$something/raw_{abi,data,sha1} be broken links? [17:29] I see that on my system and don't understand the cause [17:30] Chipaca: no, we have other store apis [17:30] do they need the same treatment? [17:30] it's a bit strange why we do this to urls for example [17:31] pedronis: yes they'd need the same treatment (or equivalent) (and i just realised what you mean) [17:31] I mean, this doesn't seem to scale to me [17:31] we need to decide to use it a bit less [17:31] or rethink how to make it scale [17:31] pedronis: how doesn't it scale? [17:32] Chipaca: we need a checker that we don't use string in our json [17:32] or something [17:32] ah [17:32] I can probably do that :-) [17:32] yes [17:32] is that sane though? [17:32] I understand the fix data on input [17:33] but make all our json painful because we might print it [17:33] is also strange [17:33] pedronis: I think the alternative would be perhaps to have our own json decoder, but I'm not sure that's a movement towards origin on the sanity line [17:34] pedronis: it seemed saner to clean everything than to answer, for each thing, 'is this going to be printed' [17:34] the latter seems a lot more bug-prone [17:35] popey: do you maybe know if there's a way to automatically install snap-xdg-open with the snap or a way to not have to install it at all? [17:35] What's the problem you want to solve? [17:35] Chipaca: people have invented language level solutions for this kind of problems (tainting), so it's hard [17:37] open a weblink from my application in a webbrowser [17:37] pedronis: I'm aware [17:37] Chipaca: also afaiu, yes, we need the same for snap.yaml [17:37] pedronis: sigh [17:38] Chipaca: I still think we might need to step back and rethink a bit the trade offs [17:39] popey: currently doing this using subprocess.Popen(["xdg-open_", self.website]) as last time I tried webbrowser.open from python didn't work on snaps. [17:39] xdg-open should work AIUI [17:39] Chipaca: there are things that definitely don't come from the user [17:40] sha3_384, the download urls for example [17:41] yeah, xdg-open does work, but afaik only when the deb package snap-xdg-open is installed, right? [17:41] fireclawTheFox: no, this is long gone now [17:41] i dont think that's needed anymore [17:41] fireclawTheFox: the whole setup is now part of snapd [17:41] ah ok, didn't know that [17:42] Chipaca: I mean, is there anything that is marked SimpleString that comes from the user? or doesn't have other validation? [17:42] pedronis: there shouldn't be [17:44] Chipaca: do we need SimpleString? [17:44] zyga-ubuntu: uh it shouldn't ever be broken links [17:44] wanna see? :) [17:44] this is on artful [17:44] under newer kernels its moved to symlinks, but they shouldn't be broken [17:44] Chipaca: anyway we can also start from the other end, what tooling we need to make sure we don't forget any string field? or we mark them somehow [17:44] https://usercontent.irccloud-cdn.com/file/oZ3RpjNO/Zrzut%20ekranu%202018-03-13%20o%2018.44.34.png [17:45] zyga-ubuntu: yes, if its broken it would be helpful to see how, so I can chase it down [17:45] pedronis: I don't know; what I understood from jdstrand was that I needed to check _everything_ [17:45] https://usercontent.irccloud-cdn.com/file/7ul07q75/Zrzut%20ekranu%202018-03-13%20o%2018.45.12.png [17:45] jjohansen: I'm not quite sure how I got to this state [17:45] this is on 4.13.0-36-generic [17:45] Chipaca: I don't know if you checked everything, I'm also not if we checked everything tomorrow [17:45] jjohansen: I was just looking at my system [17:46] pedronis: this does not check everything yet, no [17:46] Chipaca: I can add a string field, copy it, and nothing will fail [17:46] pedronis: you're right about needing a static checker if we're taking this route [17:47] jjohansen: I actually see plenty of broken links [17:47] jjohansen: if you have any snaps, can you do a quick check [17:47] jjohansen: maybe the way we remove loaded profiles is wrong [17:47] and something ends up staying behind [17:47] pedronis: but if I understand correctly, your position is that this route might be wrong [17:48] Chipaca: it seems very fiddly and given that we need static checking, we might make lighter weight at that point [17:48] Chipaca: is the fear only things that can be printed? [17:48] or was the worry something else [17:48] Chipaca: who is the attacker? [17:49] there's a lot of things that comes in and we print [17:49] there is also config [17:49] stuff [17:49] zyga-ubuntu: no, there shouldn't be a way to remove profiles wrong, there is the potential for a race of sorts because the symlink doesn't have the same hard reference, but that isn't something you should be seeing [17:50] find /sys/kernel/security/apparmor/policy/profiles -type l -exec sh -c "file -b {} | grep -q ^broken" \; -print [17:50] pedronis: what sort of lighter weight thing could it be? [17:50] zyga-ubuntu: the raw_data file should not be going away as long as that profile directory exists [17:51] this reports a load of broken symlinks for me [17:51] Chipaca: we could have a white list of things that are store produced [17:51] Chipaca: ^ can you run that and see if you get any output [17:51] (anyone with snaps on their system really) [17:51] so yeah there is a bug, I will have to hunt down [17:51] like sha3_384 or the download urls [17:51] and some other bits [17:51] zyga-ubuntu: no output [17:51] Chipaca: thank you [17:51] uptime? [17:51] for me I only see 3 hours [17:51] zyga-ubuntu: i can try on other boxes; on this one: [17:52] 17:52:01 up 20:36, 1 user, load average: 0.31, 0.24, 0.35 [17:52] thansk [17:52] mvo: ^ can you run the find command above [17:52] pedronis: ^ [17:52] hmm [17:52] zyga-ubuntu: i have a lot of output on my other box [17:52] (unless "broken" is localised) [17:52] !! thank you [17:52] You're welcome! But keep in mind I'm just a bot ;-) [17:52] 17:52:29 up 2 days, 21:14, 3 users, load average: 0.18, 0.39, 0.27 [17:52] Chipaca: which kernel? [17:52] jjohansen: where can I report this [17:52] nothing here [17:52] 4.13.0-36-generic on the one with lots of broken bts [17:53] 4.4.0-116-generic on the one without [17:53] ha [17:53] I have the same kernel as you then (the one with broken symlinks) [17:53] mvo: ^ a _maybe_ problem related to what we are seeing [17:53] 4.4.0 [17:54] mborzecki, and you perhaps? [17:54] Chipaca: I'm happy to have a HO on Thu [17:54] if we really need to do this for everything we really need a tool [17:54] of sorts [17:55] pedronis: to check, or to do? [17:55] i'd be fine with writing a checker, seems not that much work [17:56] Chipaca: to check, but I would also understand if we really need SimpleString once we have a checker [17:57] the exact attack worries are still a bit unclear to me [17:57] s/also/also like/ [17:57] pedronis: AIUI it's about decoupling things; we need it on the editable ones, but if it were already on everything we wouldn't have to worry about it at all so why not do it [17:58] if it had no costs that would make sense [17:58] but is not cost free [17:58] pedronis: do you mean in complexity? [17:59] yes [17:59] also speed apparently [17:59] you had to make it fast [17:59] to apply it everywhere [18:00] yes, i worried about this impacting that, as these were potentially big fields [18:02] anyway I should almost call it a day [18:02] we should chat again on Thu [18:02] (but maybe less than you might think, as the more arcane bits of this are from a de-escaper for terminal stuff i'm working on elsewhere) [18:02] ok [18:02] pedronis: you're off tomorrow? [18:02] yes [18:02] hmmm :-) [18:02] pedronis: enjoy [18:02] maybe I should too [18:02] mvo: you're taking friday off, right? [18:04] Chipaca: it's a bit unclear to me why anything but description and summary needs the heavy handed approach [18:04] zyga-ubuntu: if you want to file a bug against apparmor or even the ubuntu kernel in launchpad, and subscribe me, that would be the best place atm [18:04] thank you, I'll file one against artful kernel then [18:12] jjohansen: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1755563 [18:12] Bug #1755563: dangling symlinks to loaded apparmor policy [18:12] thanks [18:28] jjohansen: reproduced on bionic just now [18:29] jjohansen: I updated all the packages and got this (not snaps!) [18:29] /sys/kernel/security/apparmor/policy/profiles/libreoffice-senddoc.17/raw_data [18:30] I updated the bug report [18:32] thanks [18:41] I cannot reproduce this by simply installing the deb again, tried downgrading and upgrading, nothing [18:47] Bug #1755568 opened: Snaps are refreshed on metered connection [18:57] zyga-ubuntu: fwiw the one where i repro this is xenial [18:57] excellent [18:58] I updated the bug [18:58] jjohansen: it affects all supported kernels [19:11] PR snapd#4810 closed: osutil: DropPrivs morphs an *exec.Cmd to drop privileges [19:11] heh [19:11] that's not what i call 'tentative' :-) [19:12] niemeyer: the driver for that PR is superseded by the hash cache, so, eh [19:12] * Chipaca -> eol [19:13] It's tentative, in that there's a "reopen" button.. [19:13] Dear hit-and-run chipaca.. [19:24] PR snapd#4831 opened: tests: force profile re-generation via system-key [19:25] ^- *might* be enough to workaround the profile issue [19:48] PR snapd#4832 opened: tests: move fedora 27 to google backend [19:54] mvo: Nice [19:55] niemeyer: well, I was wrong for this particular bug before, so I will hold my breath until the tests pass :) [20:05] niemeyer: i didn't mean to hit-n-run! sorry [20:05] i mean, i didn't mean it as a hit [20:05] i definitely meant to run, because, dinner [20:06] PR snapd#4833 opened: tests: add a detector for kernel bug https://pad.lv/1755563 [20:06] niemeyer: but the order in which i saw things was: your comment about "tentative -1", then mup said "closed" -- I didn't get to see the 'tentatively closing' comment until just now [20:06] so it was funny [20:07] (but also i agreed, because the driver for the pr was gone) [20:07] * Chipaca handwaves more [20:09] Chipaca: No worries :) [20:09] Chipaca: Hope dinner was good [20:12] niemeyer: I am learning to make my mum's cheese roll [20:12] niemeyer: hell yeah it was good :-D [20:12] Wow.. I'm instantly hungry now :) [20:12] still things to tweak but getting better [20:13] Chipaca: to answer your earlier question - yes [20:13] wooooo [20:13] * mvo hugs Chipaca [20:13] mvo: wait, was that the "can i have a million euros" quesiton [20:13] mvo: or was it the "are you taking friday off" question [20:14] mvo: either way "woo!", but I need to prepare differently [20:14] with one of them there's going to be a massive party [20:14] with the other, i need a financial advisor [20:15] does https://amdflaws.com/ mean we will not see the security team for the next few weeks? [20:17] or was jdstrand going on holidays just a kind way to say he's busy ;-) [20:18] PR snapcraft#2002 opened: pluginhandler: add builtin functions to scriptlets [20:23] nice that in writing code I find other code that was tested improperly and wasn't actually testing the thing and we weren't actually doing the thing we were testing for [20:23] /o\ [20:26] mvo: Quickie if you're still around: is #4814 still relevant given #4831? [20:26] PR #4814: tests: fix re-exec profile generation in tests on classic (2.32) [20:26] PR #4831: tests: force profile re-generation via system-key [20:34] jjohansen: https://api.travis-ci.org/v3/job/353025489/log.txt and look for "kernel bug detected" [20:35] it looks like a race to me [20:37] zyga-ubuntu: hrmmm, okay I believe you, not that I didn't hours ago ;-) [20:38] jjohansen: is there anything I can provide that would help you? [20:38] zyga-ubuntu: not atm, but maybe some testing when I get a patch togeth [20:38] together [20:42] Pharaoh_Atem, hey [20:42] almost ready fedora 27 [20:42] I just see errors on snaps connecting with dbud [20:43] Pharaoh_Atem, any idea how to allow snaps connect through dbus? [20:43] Pharaoh_Atem, this is an example https://api.travis-ci.org/v3/job/353017992/log.txt [20:44] Pharaoh_Atem, the PR related is https://github.com/snapcore/snapd/pull/4832 [20:44] PR #4832: tests: move fedora 27 to google backend [20:47] cachio: dbus between snaps should already be allowed [20:47] ah [20:47] we only allow dbus comms with polkit [20:47] Pharaoh_Atem, I tested it and selinux is blocking that [20:48] so we probably need a policy update to allow snaps to communicate with each other [20:48] though I'm not exactly sure how to do that... [20:49] what I see is that is not aloowed niether connection between snaps nor using dbus-send [20:49] cachio: are you able to give me access to the machine that has the denials happening? [20:49] I need to trigger again the test, it will take 10 minutes [20:50] Pharaoh_Atem, tests triggered [21:03] slangasek: ping [21:03] Chipaca: hi there [21:03] slangasek: hiya [21:04] cachio: okay ;) [21:04] PR snapd#4834 opened: snap/squashfs: when installing from seed, try symlink before cp [21:04] slangasek: so today in the standup we came up with a better plan [21:04] slangasek: ^ that pr is the first (and probably more important) half of it :-D [21:04] slangasek: (this is about first boot from livecd being slow) [21:05] slangasek: second half is doing the hash check only once, which will speed things up; that'll be a bit more code, but the whole thing ended up being a lot more straightforward than we feared [21:05] ah, symlink, not hardlink? [21:06] slangasek: yup [21:06] seems more portable from my side, if it meets all your requirements on the snapd side [21:06] slangasek: we don't typically do that because we don't control the source (this codepath is used when sideloading for ex), but in the seed case we do [21:06] slangasek: yep, much happier with it [21:07] Chipaca: excellent. will we need any changes on the image mastering side to integrate this? [21:07] or will it be transparent to us? [21:07] slangasek: it should be entirely transparent [21:08] wfm [21:16] niemeyer: ping [21:28] PR snapcraft#2000 closed: elf: remove dead code [21:33] niemeyer: before I forget: the ping was about going over a rewrite of the login help with you before pushing it [21:35] niemeyer: also ping about quoting :-) [21:56] PR snapd#4814 closed: tests: fix re-exec profile generation in tests on classic (2.32) [21:57] Chipaca: thanks for the PR! [21:58] niemeyer: I closed 4814 and will close all related ones, 4831 is hopefully enough [21:58] PR snapd#4824 closed: tests: fix re-exec profile generation in tests on classic [22:00] PR snapd#4822 closed: interfaces/apparmor: skip apparmor cache [22:00] PR snapd#4823 closed: interfaces/apparmor: skip apparmor cache (2.32) [22:00] PR snapd#4827 closed: cmd/snap-apparmor-parser: add a prototype apparmor parser [22:03] * mvo <- bed [22:37] niemeyer: also question: is it ok to talk about 'tracking channel' and 'channel tip' in user docs [22:38] Chipaca: Yo [22:38] niemeyer: hi [22:38] niemeyer: taking the opportunity to improve the docs for find, install and refresh, via changing the help for login [22:38] niemeyer: but some of these things are hard to explain :-) [22:39] Chipaca: Tracking channel sounds fine, although there's a minor conflict with the term "channel track" which is slightly sad but okayish [22:39] Chipaca: Channel tip feels too developer-oriented [22:39] niemeyer: https://pastebin.ubuntu.com/p/D758SWrydd/ [22:40] Chipaca: Expanding it to "current revision in the channel" feels more clear to someone that doesn't have as much baggage, I guess [22:40] Chipaca: Reading... [22:40] yeah, s/tip/current revision/ works [22:42] niemeyer: this comes about because I tweaked the 'login' help: https://pastebin.ubuntu.com/p/BQGfmJQxXw/ [22:42] Chipaca: The --beta/etc references doesn't seem necessary there.. --channel is generally more interesting to teach after we introduced tracks and branches as it's the general form that works across all cases [22:42] niemeyer: so then i had to change the 'find' help: https://pastebin.ubuntu.com/p/YKWRbmrnNt/ :-) [22:43] niemeyer: yeah, was in the middle of dropping that [22:43] as in the help output it can be obvious [22:43] Chipaca: "has access to" [22:43] * Chipaca adds a 'to' to 'has access' [22:43] heh [22:43] :) [22:44] Chipaca: Mentioning --revision is nice since it's a bit of an edge case, but I think we could do a trick there by mentioning the constraint before mentioning the option, so that people don't stop reading in excitement mid-way through [22:44] E.g. [22:44] hmm, interesting, that might make refresh read easier too [22:44] For revisions that are currently installed in the local system and for personal snaps, --revision will blah blah blah... [22:45] Or similar [22:45] ok, will tinker a bit more, then push and call it a night [22:46] Chipaca: That last sentence is a bit cryptic.. I mean, I get what you mean, but I suspect most people won't [22:46] niemeyer: the one about security options? [22:46] Yeah [22:47] I suggest just dropping it and letting the error messages play their role.. [22:47] niemeyer: perhaps. I'll try to make it obvious in context (ie when the options are spelled out below) [22:47] that's fair [22:47] i mean, it's what we do today :-) [22:47] Yeah :) [22:47] i could drop the whole 'only one at a time' thing in fact [22:47] it's very discoverable and fast [22:48] * Chipaca tinkers [23:04] niemeyer: ah, one more thing [23:04] niemeyer: i had a hard time understanding your comment about 'temporarely', and i don't know if you were making a joke or not :-) [23:07] Chipaca: haha [23:07] Chipaca: Yeah, it was a joke :P [23:07] phew :) [23:07] That's such a great typo, though [23:07] Is there a place we can submit candidate words? [23:08] niemeyer: you'll need a wig and a posh accent [23:10] niice [23:10] https://forum.snapcraft.io/t/explain-permissions-better-or-people-will-freak-out/4456 [23:10] I didn't know we had that in place already :-) [23:10] robert_ancell: that you ^?