[03:36] hmm, there's something very wrong with auto-connected interfaces here [03:36] "snap install lxd" on a completely clean system (no core snap yet) leads to none of our interfaces being connected [03:36] removing the snap and reinstalling it then gets things connected properly [03:47] https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 [03:48] this is obviously rather critical [04:15] confirmed to affect all snaps on initial install (when no core snap is present) [04:56] morning [04:58] mborzecki: good morning, looks like you're the first snap person around today, you may want to take a look at https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 [05:01] stgraber: looking [05:16] stgraber: working locally with latest snapd, even if i purge everything and start with clean state, i'm trying cloud image now [05:17] mborzecki: 16.04 cloud image should be affected, that's effectively what MAAS uses [05:18] mborzecki: it may or may not be related to re-exec, so latest snapd on your system may get you a different result, not sure [05:20] stgraber: yeah, something is off: https://paste.ubuntu.com/p/6P4JpMhgPf/ [05:21] that's on cloud image [05:21] yup, matches what I'm seeing here and what our users have been reporting [05:23] and that's my host: https://paste.ubuntu.com/p/hDzWMgY2zb/ [05:48] stgraber: left a note, i think the approach needs to be discussed with mvo/pstolowski/pedronis [05:48] stgraber: refreshing the core snap first should workaround the problem [05:49] mvo: speaking of mvo :P [05:49] mvo: morning [05:50] mborzecki: goooood morning [05:50] mborzecki: what did I miss :) [05:50] ? [05:51] mvo: stgraber reported a problem with auto-connect https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 [05:51] mvo: i looked around and my observation is that we're missing the 'auto-connect' task because the task list is created using the old 'snapd' [05:53] mborzecki: yeah, I was just reading the forum [05:53] mborzecki: I think your analysis makes sense, that is most likely the issue [05:54] mvo: i looked in state.json and there's no auto-connect when old snapd creates the task list [05:54] mborzecki: yeah, this makes sense. so its the race when you don't have a core snap and it restarts itself [05:55] mvo: do you think we could somehow patch it when core updates? [05:57] mborzecki: yeah, I think we need to add compat code into the "old" task (setup-profiles) [05:58] mborzecki: I mean, something like (in setup-profiles): scan the task list and if the task is missing either inject it or run it [05:58] mvo: otoh, maybe we should rewrite the whole pending task list if core is updated in the process [05:58] mborzecki: we will need pawel here [05:59] mborzecki: yeah, its a bit of a generic problem [06:01] PR snapd#4972 opened: cmd/snap-mgmt: remove timers, udev rules, dbus policy files [06:20] mborzecki: I followed up in the forum, lets discuss with pawel once he is here [06:20] mvo: ok [06:30] good morning [06:31] * zyga catches up with IRC [06:31] stgraber: ack, [06:32] oh, good analysis indeed mborzecki [06:32] I would love if we could formalize the reexec protocol limitations somewhere [06:32] zyga: hey, morning [06:32] so we don't run into this willy nilly again [06:32] mvo: do you consider this a release blocker? [06:41] zyga: I think we've satisfied everyone's requirements as far as my internet check PR goes [06:41] zyga: also, could you please change leap version in: https://github.com/canonical-docs/snappy-docs/pull/380 [06:41] PR canonical-docs/snappy-docs#380: Use address --refresh on openSUSE [06:42] from 42.2 to 42.3 [06:42] I would have submitted my own, but then it would conflict with yours [06:42] Caelum: yes, I will do that now [06:42] thank you [06:43] Caelum: I prepared an update that enables apparmor but I need to send a mail to suse security team and discuss some topics [06:43] it's pretty complex how all those things interact [06:43] nice [06:43] the goal is to land an apparmor-enabled version into the system:snappy and start the discussion with the security team on how to add snapd to factory proper [06:44] that would be awesome [06:44] zyga: good morning! yes, I think this is a blocker [06:45] zyga: I wonder if we need to re-think re-exec, maybe something like: if there is a new core, always *only* referesh that then re-exec and do the rest [06:45] mvo: yes, I think this makes a lot of sense [06:46] mvo: or introduce a formal protocol where snapd "new" can re-interpret partially finished tasks [06:46] mvo: but whatever we do needs to work all the way back [06:46] or we would always have to adjust old versions [06:46] or provide updates/backports [06:47] I think doing this on the "new" side would be conceptually harder but easier to deploy [06:47] we could also do both sides so in the future it is easier in general [06:47] Caelum: PR updated [06:49] mvo: in this case, do we have enough information to synthesize new tasks on snapd startup? [06:49] mvo: cook old snapd state, start overlord, look at state; [06:49] zyga: awesome thank you! [06:49] mvo: the new state should include the desired set of tasks [06:49] zyga: I think so, in this case I think we can just insert a auto-connect task [06:49] Caelum: I'll poke people to merge it soon [06:49] Caelum: thank *you* :-) [06:50] yes, that's my thinking [06:50] zyga: and then we need to discuss a more general approach [06:50] it should be simple enough for this one-off case [06:50] zyga: its definitely making things very complicated (re-exec) [06:50] zyga: yeah [06:50] but yes, definitely a topic for discussion [06:50] mvo: it's so much simpler to test and evaluate in distros where we don't support it [06:50] mvo: so I agree totally [06:50] zyga: I want to wait for pawel but hopefully its easy, we have code for task injecton already [06:50] mvo: alternative, make shallow snapd [06:51] mvo: that doesn't do stuff [06:51] just reexecs [06:51] sounds good, I'll get back to fstab [06:51] zyga: yeah, but even with the shallow one we will need to deal with ugprades [06:51] zyga: i.e. when to re-exec on upgrades [06:52] zyga: but yeah, a much bigger topic that does not fit well into irc :) [06:52] shallow one would not contain any logic apart from "fetch core in super generic way, re-exec" [06:52] zyga: sure, that would fix step 0. but what if we have full snapd and refresh to full snapd+1 with new tasks [06:53] ohh [06:53] yes [07:12] morning [07:15] moin moin [07:15] o/ pstolowski [07:17] hey pawel [07:17] we have some important work for you [07:17] pstolowski: thanks for your reply in the forum! [07:18] zyga: the auto-connect issue? [07:19] indeed [07:19] mvo: https://github.com/snapcore/snapd/pull/4973 [07:19] PR #4973: osutil: fix fstab parser to allow for # in field values [07:19] I'll make a backport after this lands [07:19] PR snapd#4973 opened: osutil: fix fstab parser to allow for # in field values [07:20] zyga: ta [07:24] mborzecki: https://github.com/snapcore/snapd/pull/4938 updated [07:24] PR #4938: release-tools: add repack-debian-tarball.sh [07:29] 4969 needs a second review [07:32] zyga: yeah, i find `find .. -exec` utterly confusing syntax wise and replace it with `| xargs` where possible (also xargs has the nice -P switch which i abuse when possible) [07:37] mborzecki: but find ... -exec works for any number of files; xargs will hit the kernel argument size limit eventually [07:40] pstolowski: 4968 + 1 but add a test please [07:42] zyga: will do, thanks === pbek_ is now known as pbek [07:43] mvo is https://github.com/snapcore/snapd/pull/4951 a 2.32 backport candidate [07:43] PR #4951: interfaces/desktop-legacy: allow access to gnome-shell screenshot/screencast [07:44] it needs gustao review [07:44] zyga: it is, but blocked right now [07:46] it seems we need a new tests that is not about upgrades but about starting installing from stable deb [07:48] pedronis: indeed [07:57] PR snapd#4974 opened: ifacestate: injectTasks helper [07:58] mvo zyga pedronis can you please take a look at ^ ? it's a helper that I initially meant to propose with interface hooks, but it's going to be useful now for auto-connect fix [07:59] ack [07:59] * zyga sees a curious error: [07:59] moin moin [07:59] https://www.irccloud.com/pastebin/pVrMuty3/ [07:59] Chipaca: hey [08:00] pstolowski: did you see zyga's comment (somewhere, on the forum maybe?) about making sure core is installed before doing anything further? wouldn't this solve the autoconnect thing? [08:00] Chipaca: this is about core [08:02] Chipaca: it's installed first, is just that the next install is not setup properly [08:02] pedronis: because the tasks are created by the old snapd? [08:02] yes [08:03] the other issue is a bit different [08:03] k [08:03] is about trying to run hooks while core is not active [08:03] that needs to wait in some form === fnordahl_ is now known as fnordahl [08:04] Chipaca: to be clear, I should probably write this on the forum, but first doesn't mean a lot in our world, we either need to setup dependecies or wait [08:05] you can always start an install and a different change at the same time [08:05] mvo: https://github.com/snapcore/snapd/pull/4973 updated [08:05] PR #4973: osutil: fix fstab parser to allow for # in field values [08:05] zyga: ta! [08:05] Chipaca: https://github.com/snapcore/snapd/pull/4938 needs your re-review [08:05] PR #4938: release-tools: add repack-debian-tarball.sh [08:06] oh [08:06] zyga: no it doesn't [08:06] you just did [08:06] =) [08:06] thanks :) [08:06] despite you telling me your comment was *fine* when it wasn't :-) === tinwood_ is now known as tinwood [08:06] sorry ^_^ [08:06] I changed this a few times locally and I forgot what it was then [08:07] 'twas fine, just confusing (but you got it sorted in the end) [08:07] with that merged we will just have 6 more scripts from mvo's stash to port ;) [08:07] pstolowski: are you looking also into writing the spread? start with current stable deb, and install a snap that needs autoconnections , probably needs the fakestore to use the latest code for snapd [08:08] zyga: *cough* [08:08] zyga: I'm looking at this as well now [08:08] well, ok, 9 [08:11] mvo: I have a small README file to add there next [08:12] wow, it's really sprint this time, +16 in the shade and sunny [08:12] zyga: thanks, added a small comment [08:12] mborzecki: it's bound to be 22 next monday [08:13] I cannot wait for coding sessions in the park :) [08:14] zyga: mvo: you both make good points about cpuinfo [08:14] zyga: mvo: note however I am doing what apport does, there [08:14] hah, can't wait to start bicycling again [08:15] zyga: mvo: and what it does is fine, i think: if it doesn't look like the x86 ones, it ships the whole file [08:15] (and the non-x86 ones i've seen so far are saner in this sense) [08:15] Chipaca: yeah, I think its fine. also x86 will be the 99% [08:15] (for now at least) [08:16] mvo: non-x86 is where a lot more bugs happen though :-) proportionally [08:18] mvo: did you see intel shares drop 10% [08:18] mvo: when apple announced macs will move to arm in 2020 [08:19] well, not arm specifically but probably [08:19] (some people think it might be riscV as well since it is free) [08:19] and the core design is custom anyway, just instruction set was used [08:21] pstolowski: can we close https://github.com/snapcore/snapd/pull/4965 [08:21] PR #4965: ifacestate: injectTasks helper [08:22] zyga: yes, done [08:23] PR snapd#4965 closed: ifacestate: injectTasks helper [08:23] zyga: pushed an update to #4972 [08:23] PR #4972: cmd/snap-mgmt: remove timers, udev rules, dbus policy files [08:24] siigh [08:24] is it the 90s again [08:24] except there's no NeXT [08:24] mborzecki: thanks, looking [08:25] apple going off and doing its own CPUs, not doing anything really new, fans still being fans [08:25] mborzecki: ack [08:25] thank you :) [08:25] mvo: that's 2.32.3 right? [08:25] mborzecki: did you shellcheck everything new? [08:25] Chipaca: it's the 90s but we have linux [08:26] zyga: i had linux in the 90s [08:26] mborzecki: yes [08:26] but linux now sucks less ;-) [08:26] zyga: yeah, it will complain about -exec and recommend -execdir (but that's expected) [08:26] zyga: i've been on linux since 1994, and i switched because it sucked less [08:27] mborzecki, zyga hm, if shellcheck does not like, whats the advantage over find|xargs? [08:27] mborzecki: interesting, I didn't know about execdir [08:27] can we just use execdir then [08:27] it looks very sane [08:27] force push? [08:27] mvo: xargs has size limits [08:27] (trying to keep the commit count low) [08:27] there's -n [08:27] though [08:27] and you can use + form to make fewer calls to rm even :) [08:28] mborzecki: we can squash it on merge but I'm fine with force push too [08:28] heh, -n, + bikeshedding :P [08:28] yeah, this feels like we are deep in bikesheed land :) [08:29] it's shell ;) [08:29] it's the perfect candidate [08:29] zyga: what's that about size limits? [08:30] PR snapd#4975 opened: osutil: fix fstab parser to allow for # in field values (2.32) [08:31] Chipaca: xargs wasn't smart and could run into issues with the size of cmdline [08:31] that's why I traditionally don't like using it [08:32] mvo: hi! do you know why core18 wasn't pushed to the store, even though it was built successfully for amd64? does it wait for all architectures to build? [08:32] https://www.irccloud.com/pastebin/srWqXthX/ [08:32] Chipaca: ^ [08:33] zyga: Chipaca: iirc xargs will call the command a number of times if it hits a limit, but i'm not sure if that applies to all versions in the wild [08:33] (or anything else than gnu) [08:33] BjornT_: let me check [08:33] zyga: ah [08:34] BjornT_: I manually published it now, but let me try to figure out why it wasn't done so automatically [08:35] mborzecki: that's intrinsic to xargs, yes, but see zyga's pastebin [08:35] mvo: can we override gustavo's -1 since the issue is fixed now [08:35] https://github.com/snapcore/snapd/pull/4930/files [08:35] PR #4930: skip test that requires internet when not present [08:35] Chipaca: ack [08:35] how do I use adt-buildvm-ubuntu-cloud to get a bionic image for spread? [08:35] in xenial [08:36] (this is from man xargs) [08:36] (it looks for a .disk1.img that's not there) [08:36] zyga: looking [08:36] Chipaca: adt-buildvm-ubuntu-cloud -r bionic does not work? [08:37] mvo: thanks! i did confirm that the built core18 snap worked together with maas and snapcraft 2.40, btw [08:37] mvo: it tries “Downloading https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64-disk1.img” and then crashes with a 404 [08:37] 404 -> traceback, because python [08:37] Chipaca: iirc cloudimg' don't have .disk1.img [08:37] but 404 nontheless :-) [08:37] nonetheless* [08:37] Chipaca: oh, indeed, iirc you need to do: "sudo apt install -t xenail-backports apport" [08:37] Chipaca: to get the non-ancient version of apport [08:37] Chipaca: eh, adt [08:37] _apport_? for cloud images? [08:37] Chipaca: silly me [08:37] ah :-) [08:37] Chipaca: autopkgtest [08:38] Chipaca: but let me quickly double check [08:38] Chipaca: yes, please try that [08:38] (the version of *autopkgtest* from xenial-backports) [08:38] E: The value 'xenail-backports' is invalid for APT::Default-Release as such a release is not available in the sources [08:38] Chipaca: without my typo :( [08:38] * Chipaca fixes the typo and tries again [08:38] * Chipaca is copy-pasting commands that start with 'sudo' [08:39] mvo: I _trusted_ you! /o\ [08:39] Chipaca: sorry for letting you down! [08:39] I'm used to it by now :-p [08:39] anyway, that worked, thanks! [08:39] * mvo weeps a bit in the corner [08:39] Chipaca: happy that it worked, this should give you an image [08:40] yep yep, it's doing its thing [08:40] damn, forgot to start the pomodoro timer [08:42] mborzecki: for what specifically? [08:42] mvo: work :P [08:43] mvo: there's a nice gnome shell extension http://gnomepomodoro.org/ [08:43] mborzecki: do you use a physical one or software? if software, which one [08:43] mborzecki: heh - you answered already [08:43] hello [08:44] I think snap or apparmor broke my lxd install. Is this the correct place to look for help? [08:44] hey timp [08:44] yes, what happeend? [08:44] happened [08:44] zyga: thanks :) [08:44] mvo: forces breaks on you, reminds me to do some stretching, squats etc [08:44] yesterday, my lxd worked fine. Today it does not. (I did an apt upgrade yesterday, that may be related. No snap refresh though) [08:44] $ lxc list [08:44] mborzecki: zyga: not sure why I'm looking at this, but freebsd's find also has -execdir (but its xargs doesn't have --show-limits) [08:44] cat: /proc/self/attr/current: Permission denied [08:44] /snap/lxd/6492/commands/lxc: 6: exec: aa-exec: Permission denied [08:44] timp: snaps refresh automatically [08:44] dmesg shows this: [08:44] [ 1140.618728] audit: type=1400 audit(1522829695.691:122): apparmor="DENIED" operation="open" profile="snap.lxd.lxc" name="/proc/8365/attr/current" pid=8365 comm="cat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 [08:45] [ 1140.619047] audit: type=1400 audit(1522829695.691:123): apparmor="DENIED" operation="exec" profile="snap.lxd.lxc" name="/usr/sbin/aa-exec" pid=8352 comm="lxc" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0 [08:45] timp: can you check if "snap changes" has anything about lxd? [08:45] yes, it did: [08:45] 93 Done 2018-04-04T07:44:00Z 2018-04-04T07:45:55Z Auto-refresh snaps "lxd", "aws-cli", "core" [08:45] timp: and can you paste "snap interfaces | grep lxd" [08:45] timp: ok [08:45] :lxd-support lxd [08:45] :network lxd,nextcloud-client,simplenote,telegram-sergiusens [08:45] :system-observe lxd [08:45] lxd:lxd [08:46] hmm all looks good here [08:46] let me look deeper [08:47] thanks [08:47] isn't there a discussion about in the forum [08:47] I can 'cat /proc/self/attr/current', so that permission denied is weird. I don't know how AppArmor works though. [08:47] oh, perhaps [08:48] is this relevant: curl https://assertions.staging.ubuntu.com/v1/assertions/account-key?account-id=canonical [08:48] mborzecki: what version of systemd do you have over there? (on arch i mean :-) ) [08:48] sorry [08:48] is this relevant: https://forum.snapcraft.io/t/2-0-lxd-snap-fails-on-sytems-with-partial-apparmor-support/4707 [08:48] Chipaca: 238 [08:48] ahh [08:48] timp: can you paste "snap version" [08:49] snap 2.32.1 [08:49] snapd 2.32.1 [08:49] series 16 [08:49] ubuntu 16.04 [08:49] kernel 4.13.0-37-generic [08:49] I'm on xenial [08:50] timp: thank you [08:50] so this is not a partial apparmor support issue [08:50] ok, I'll look at what I was meant to before [08:50] what do you mean with partial apparmor support issue? [08:50] zyga: mvo: force pushed to 4972 [08:50] timp: debian and ubuntu have different set of supported apparmor features because ubuntu still carries a patch with non-upstreamed extensions [08:51] timp: can you please pastebin /var/lib/snapd/apparmor/profiles/snap.lxd.lxc [08:52] * zyga -> brb [08:53] zyga: sure, http://paste.ubuntu.com/p/hrrQFkXYSx/ [08:54] https://forum.snapcraft.io/t/snapped-lxd-has-stopped-working-aa-exec-doesnt-exist-in-the-snap/2356 seems similar, but there's no solution there [08:55] hmm, this is unfortunate. I was doing all my work in containers so now I'm blocked. [08:57] so, [08:57] $ snap revert lxd [08:57] lxd reverted to 3.0.0 [08:57] tim@tim-XPS-13-9350:~$ lxc list [08:57] that fixes it for now. [08:57] looks like the lxd update broke stuff [08:57] timp, you might want to ask on #lxc-dev as well [08:58] re [08:58] timp: looking [08:59] timp: it looks like the profile is wrong [08:59] it doesn't contain contributions from lxd-support [09:00] hmm, 'snap refresh' tells me that there are no updates for lxd, even though I just did snap revert lxd [09:00] can you snap disconnect lxd:lxd-support [09:00] and then re-connect it [09:00] and see if the profile is different [09:00] (copy the current profile out) [09:01] note that after 'snap revert lxd', it is working again. And I cannot reproduce the problem any more. [09:01] right because you are on a different revision [09:01] you can look at the same file again [09:01] and the header will say @{SNAP_REVISION} = ... [09:01] snap.lxd.lxc? [09:01] yes [09:02] in the pastebin it was 6492 [09:02] this is with the working version: http://paste.ubuntu.com/p/vgJcv7dPRm/ [09:02] hmm, also 6492? [09:03] right, this one is correct [09:03] the previous one looked like basic, unconnected (no interfaces) application [09:03] ah, so what I did is 'snap revert lxd'. And then 'snap refresh lxd'. Now I again have 6492, and it still works. [09:03] pstolowski: ^ maybe some other bug? [09:04] it looks like a bug in snapd, more than a bug in lxd IMO [09:04] hmm hmm [09:04] can you add this to a forum thread or a bug report [09:04] I don't want to lose it here [09:04] on LP? [09:04] yes, on snapd [09:04] ok [09:04] this a much more tested path than the snap install lxd (without anything) [09:05] though [09:06] timp: please include: snap version; snap changes; snap interfaces (from what you did, not from what is is now when it works) [09:06] and the two profiles you made [09:06] (the pastebins) [09:06] I think it shows that we didn't really include the lxd-support snippets at all [09:07] pstolowski: the bug on the forum feels related now [09:07] the reporter there doens't have lxd-support [09:07] mvo: didn't you publish the latest one? the one you published is still on glibc 2.26. [09:07] mvo: this is the one i used when i confirmed it worked with maas: https://code.launchpad.net/~mvo/+snap/core18/+build/182036/+files/core18_very-unstable_amd64.snap [09:08] * zyga afk again [09:11] zyga, pedronis: thanks for the help. I reported the bug here: https://bugs.launchpad.net/snapd/+bug/1761115 [09:11] Bug #1761115: After lxd snap upgrade, it lxc stopped working [09:11] timp: thank you [09:11] BjornT_: indeed, it was an older one [09:13]