| stgraber | hmm, there's something very wrong with auto-connected interfaces here | 03:36 |
|---|---|---|
| stgraber | "snap install lxd" on a completely clean system (no core snap yet) leads to none of our interfaces being connected | 03:36 |
| stgraber | removing the snap and reinstalling it then gets things connected properly | 03:36 |
| stgraber | https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 | 03:47 |
| stgraber | this is obviously rather critical | 03:48 |
| stgraber | confirmed to affect all snaps on initial install (when no core snap is present) | 04:15 |
| mborzecki | morning | 04:56 |
| stgraber | mborzecki: good morning, looks like you're the first snap person around today, you may want to take a look at https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 | 04:58 |
| mborzecki | stgraber: looking | 05:01 |
| mborzecki | stgraber: working locally with latest snapd, even if i purge everything and start with clean state, i'm trying cloud image now | 05:16 |
| stgraber | mborzecki: 16.04 cloud image should be affected, that's effectively what MAAS uses | 05:17 |
| stgraber | mborzecki: it may or may not be related to re-exec, so latest snapd on your system may get you a different result, not sure | 05:18 |
| mborzecki | stgraber: yeah, something is off: https://paste.ubuntu.com/p/6P4JpMhgPf/ | 05:20 |
| mborzecki | that's on cloud image | 05:21 |
| stgraber | yup, matches what I'm seeing here and what our users have been reporting | 05:21 |
| mborzecki | and that's my host: https://paste.ubuntu.com/p/hDzWMgY2zb/ | 05:23 |
| mborzecki | stgraber: left a note, i think the approach needs to be discussed with mvo/pstolowski/pedronis | 05:48 |
| mborzecki | stgraber: refreshing the core snap first should workaround the problem | 05:48 |
| mborzecki | mvo: speaking of mvo :P | 05:49 |
| mborzecki | mvo: morning | 05:49 |
| mvo | mborzecki: goooood morning | 05:50 |
| mvo | mborzecki: what did I miss :) | 05:50 |
| mvo | ? | 05:50 |
| mborzecki | mvo: stgraber reported a problem with auto-connect https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 | 05:51 |
| mborzecki | mvo: i looked around and my observation is that we're missing the 'auto-connect' task because the task list is created using the old 'snapd' | 05:51 |
| mvo | mborzecki: yeah, I was just reading the forum | 05:53 |
| mvo | mborzecki: I think your analysis makes sense, that is most likely the issue | 05:53 |
| mborzecki | mvo: i looked in state.json and there's no auto-connect when old snapd creates the task list | 05:54 |
| mvo | mborzecki: yeah, this makes sense. so its the race when you don't have a core snap and it restarts itself | 05:54 |
| mborzecki | mvo: do you think we could somehow patch it when core updates? | 05:55 |
| mvo | mborzecki: yeah, I think we need to add compat code into the "old" task (setup-profiles) | 05:57 |
| mvo | mborzecki: I mean, something like (in setup-profiles): scan the task list and if the task is missing either inject it or run it | 05:58 |
| mborzecki | mvo: otoh, maybe we should rewrite the whole pending task list if core is updated in the process | 05:58 |
| mvo | mborzecki: we will need pawel here | 05:58 |
| mvo | mborzecki: yeah, its a bit of a generic problem | 05:59 |
| mup | PR snapd#4972 opened: cmd/snap-mgmt: remove timers, udev rules, dbus policy files <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4972> | 06:01 |
| mvo | mborzecki: I followed up in the forum, lets discuss with pawel once he is here | 06:20 |
| mborzecki | mvo: ok | 06:20 |
| zyga | good morning | 06:30 |
| * zyga catches up with IRC | 06:31 | |
| zyga | stgraber: ack, | 06:31 |
| zyga | oh, good analysis indeed mborzecki | 06:32 |
| zyga | I would love if we could formalize the reexec protocol limitations somewhere | 06:32 |
| mborzecki | zyga: hey, morning | 06:32 |
| zyga | so we don't run into this willy nilly again | 06:32 |
| zyga | mvo: do you consider this a release blocker? | 06:32 |
| Caelum | zyga: I think we've satisfied everyone's requirements as far as my internet check PR goes | 06:41 |
| Caelum | zyga: also, could you please change leap version in: https://github.com/canonical-docs/snappy-docs/pull/380 | 06:41 |
| mup | PR canonical-docs/snappy-docs#380: Use address --refresh on openSUSE <Created by zyga> <https://github.com/canonical-docs/snappy-docs/pull/380> | 06:41 |
| Caelum | from 42.2 to 42.3 | 06:42 |
| Caelum | I would have submitted my own, but then it would conflict with yours | 06:42 |
| zyga | Caelum: yes, I will do that now | 06:42 |
| Caelum | thank you | 06:42 |
| zyga | Caelum: I prepared an update that enables apparmor but I need to send a mail to suse security team and discuss some topics | 06:43 |
| zyga | it's pretty complex how all those things interact | 06:43 |
| Caelum | nice | 06:43 |
| zyga | the goal is to land an apparmor-enabled version into the system:snappy and start the discussion with the security team on how to add snapd to factory proper | 06:43 |
| Caelum | that would be awesome | 06:44 |
| mvo | zyga: good morning! yes, I think this is a blocker | 06:44 |
| mvo | zyga: I wonder if we need to re-think re-exec, maybe something like: if there is a new core, always *only* referesh that then re-exec and do the rest | 06:45 |
| zyga | mvo: yes, I think this makes a lot of sense | 06:45 |
| zyga | mvo: or introduce a formal protocol where snapd "new" can re-interpret partially finished tasks | 06:46 |
| zyga | mvo: but whatever we do needs to work all the way back | 06:46 |
| zyga | or we would always have to adjust old versions | 06:46 |
| zyga | or provide updates/backports | 06:46 |
| zyga | I think doing this on the "new" side would be conceptually harder but easier to deploy | 06:47 |
| zyga | we could also do both sides so in the future it is easier in general | 06:47 |
| zyga | Caelum: PR updated | 06:47 |
| zyga | mvo: in this case, do we have enough information to synthesize new tasks on snapd startup? | 06:49 |
| zyga | mvo: cook old snapd state, start overlord, look at state; | 06:49 |
| Caelum | zyga: awesome thank you! | 06:49 |
| zyga | mvo: the new state should include the desired set of tasks | 06:49 |
| mvo | zyga: I think so, in this case I think we can just insert a auto-connect task | 06:49 |
| zyga | Caelum: I'll poke people to merge it soon | 06:49 |
| zyga | Caelum: thank *you* :-) | 06:49 |
| zyga | yes, that's my thinking | 06:50 |
| mvo | zyga: and then we need to discuss a more general approach | 06:50 |
| zyga | it should be simple enough for this one-off case | 06:50 |
| mvo | zyga: its definitely making things very complicated (re-exec) | 06:50 |
| mvo | zyga: yeah | 06:50 |
| zyga | but yes, definitely a topic for discussion | 06:50 |
| zyga | mvo: it's so much simpler to test and evaluate in distros where we don't support it | 06:50 |
| zyga | mvo: so I agree totally | 06:50 |
| mvo | zyga: I want to wait for pawel but hopefully its easy, we have code for task injecton already | 06:50 |
| zyga | mvo: alternative, make shallow snapd | 06:50 |
| zyga | mvo: that doesn't do stuff | 06:51 |
| zyga | just reexecs | 06:51 |
| zyga | sounds good, I'll get back to fstab | 06:51 |
| mvo | zyga: yeah, but even with the shallow one we will need to deal with ugprades | 06:51 |
| mvo | zyga: i.e. when to re-exec on upgrades | 06:51 |
| mvo | zyga: but yeah, a much bigger topic that does not fit well into irc :) | 06:52 |
| zyga | shallow one would not contain any logic apart from "fetch core in super generic way, re-exec" | 06:52 |
| mvo | zyga: sure, that would fix step 0. but what if we have full snapd and refresh to full snapd+1 with new tasks | 06:52 |
| zyga | ohh | 06:53 |
| zyga | yes | 06:53 |
| pstolowski | morning | 07:12 |
| kalikiana | moin moin | 07:15 |
| kalikiana | o/ pstolowski | 07:15 |
| zyga | hey pawel | 07:17 |
| zyga | we have some important work for you | 07:17 |
| mvo | pstolowski: thanks for your reply in the forum! | 07:17 |
| pstolowski | zyga: the auto-connect issue? | 07:18 |
| zyga | indeed | 07:19 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/4973 | 07:19 |
| mup | PR #4973: osutil: fix fstab parser to allow for # in field values <Created by zyga> <https://github.com/snapcore/snapd/pull/4973> | 07:19 |
| zyga | I'll make a backport after this lands | 07:19 |
| mup | PR snapd#4973 opened: osutil: fix fstab parser to allow for # in field values <Created by zyga> <https://github.com/snapcore/snapd/pull/4973> | 07:19 |
| mvo | zyga: ta | 07:20 |
| zyga | mborzecki: https://github.com/snapcore/snapd/pull/4938 updated | 07:24 |
| mup | PR #4938: release-tools: add repack-debian-tarball.sh <Created by zyga> <https://github.com/snapcore/snapd/pull/4938> | 07:24 |
| mvo | 4969 needs a second review | 07:29 |
| mborzecki | zyga: yeah, i find `find .. -exec` utterly confusing syntax wise and replace it with `| xargs` where possible (also xargs has the nice -P switch which i abuse when possible) | 07:32 |
| zyga | mborzecki: but find ... -exec works for any number of files; xargs will hit the kernel argument size limit eventually | 07:37 |
| zyga | pstolowski: 4968 + 1 but add a test please | 07:40 |
| pstolowski | zyga: will do, thanks | 07:42 |
| === pbek_ is now known as pbek | ||
| zyga | mvo is https://github.com/snapcore/snapd/pull/4951 a 2.32 backport candidate | 07:43 |
| mup | PR #4951: interfaces/desktop-legacy: allow access to gnome-shell screenshot/screencast <Created by jdstrand> <https://github.com/snapcore/snapd/pull/4951> | 07:43 |
| zyga | it needs gustao review | 07:44 |
| mvo | zyga: it is, but blocked right now | 07:44 |
| pedronis | it seems we need a new tests that is not about upgrades but about starting installing from stable deb | 07:46 |
| mvo | pedronis: indeed | 07:48 |
| mup | PR snapd#4974 opened: ifacestate: injectTasks helper <Created by stolowski> <https://github.com/snapcore/snapd/pull/4974> | 07:57 |
| pstolowski | mvo zyga pedronis can you please take a look at ^ ? it's a helper that I initially meant to propose with interface hooks, but it's going to be useful now for auto-connect fix | 07:58 |
| zyga | ack | 07:59 |
| * zyga sees a curious error: | 07:59 | |
| Chipaca | moin moin | 07:59 |
| zyga | https://www.irccloud.com/pastebin/pVrMuty3/ | 07:59 |
| zyga | Chipaca: hey | 07:59 |
| Chipaca | pstolowski: did you see zyga's comment (somewhere, on the forum maybe?) about making sure core is installed before doing anything further? wouldn't this solve the autoconnect thing? | 08:00 |
| pedronis | Chipaca: this is about core | 08:00 |
| pedronis | Chipaca: it's installed first, is just that the next install is not setup properly | 08:02 |
| Chipaca | pedronis: because the tasks are created by the old snapd? | 08:02 |
| pedronis | yes | 08:02 |
| pedronis | the other issue is a bit different | 08:03 |
| Chipaca | k | 08:03 |
| pedronis | is about trying to run hooks while core is not active | 08:03 |
| pedronis | that needs to wait in some form | 08:03 |
| === fnordahl_ is now known as fnordahl | ||
| pedronis | Chipaca: to be clear, I should probably write this on the forum, but first doesn't mean a lot in our world, we either need to setup dependecies or wait | 08:04 |
| pedronis | you can always start an install and a different change at the same time | 08:05 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/4973 updated | 08:05 |
| mup | PR #4973: osutil: fix fstab parser to allow for # in field values <Created by zyga> <https://github.com/snapcore/snapd/pull/4973> | 08:05 |
| mvo | zyga: ta! | 08:05 |
| zyga | Chipaca: https://github.com/snapcore/snapd/pull/4938 needs your re-review | 08:05 |
| mup | PR #4938: release-tools: add repack-debian-tarball.sh <Created by zyga> <https://github.com/snapcore/snapd/pull/4938> | 08:05 |
| zyga | oh | 08:06 |
| Chipaca | zyga: no it doesn't | 08:06 |
| zyga | you just did | 08:06 |
| Chipaca | =) | 08:06 |
| zyga | thanks :) | 08:06 |
| Chipaca | despite you telling me your comment was *fine* when it wasn't :-) | 08:06 |
| === tinwood_ is now known as tinwood | ||
| zyga | sorry ^_^ | 08:06 |
| zyga | I changed this a few times locally and I forgot what it was then | 08:06 |
| Chipaca | 'twas fine, just confusing (but you got it sorted in the end) | 08:07 |
| zyga | with that merged we will just have 6 more scripts from mvo's stash to port ;) | 08:07 |
| pedronis | pstolowski: are you looking also into writing the spread? start with current stable deb, and install a snap that needs autoconnections , probably needs the fakestore to use the latest code for snapd | 08:07 |
| mvo | zyga: *cough* | 08:08 |
| mvo | zyga: I'm looking at this as well now | 08:08 |
| zyga | well, ok, 9 | 08:08 |
| zyga | mvo: I have a small README file to add there next | 08:11 |
| mborzecki | wow, it's really sprint this time, +16 in the shade and sunny | 08:12 |
| mvo | zyga: thanks, added a small comment | 08:12 |
| zyga | mborzecki: it's bound to be 22 next monday | 08:12 |
| zyga | I cannot wait for coding sessions in the park :) | 08:13 |
| Chipaca | zyga: mvo: you both make good points about cpuinfo | 08:14 |
| Chipaca | zyga: mvo: note however I am doing what apport does, there | 08:14 |
| mborzecki | hah, can't wait to start bicycling again | 08:14 |
| Chipaca | zyga: mvo: and what it does is fine, i think: if it doesn't look like the x86 ones, it ships the whole file | 08:15 |
| Chipaca | (and the non-x86 ones i've seen so far are saner in this sense) | 08:15 |
| mvo | Chipaca: yeah, I think its fine. also x86 will be the 99% | 08:15 |
| mvo | (for now at least) | 08:15 |
| Chipaca | mvo: non-x86 is where a lot more bugs happen though :-) proportionally | 08:16 |
| zyga | mvo: did you see intel shares drop 10% | 08:18 |
| zyga | mvo: when apple announced macs will move to arm in 2020 | 08:18 |
| zyga | well, not arm specifically but probably | 08:19 |
| zyga | (some people think it might be riscV as well since it is free) | 08:19 |
| zyga | and the core design is custom anyway, just instruction set was used | 08:19 |
| zyga | pstolowski: can we close https://github.com/snapcore/snapd/pull/4965 | 08:21 |
| mup | PR #4965: ifacestate: injectTasks helper <Blocked> <Created by stolowski> <https://github.com/snapcore/snapd/pull/4965> | 08:21 |
| pstolowski | zyga: yes, done | 08:22 |
| mup | PR snapd#4965 closed: ifacestate: injectTasks helper <Blocked> <Created by stolowski> <Closed by stolowski> <https://github.com/snapcore/snapd/pull/4965> | 08:23 |
| mborzecki | zyga: pushed an update to #4972 | 08:23 |
| mup | PR #4972: cmd/snap-mgmt: remove timers, udev rules, dbus policy files <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4972> | 08:23 |
| Chipaca | siigh | 08:24 |
| Chipaca | is it the 90s again | 08:24 |
| Chipaca | except there's no NeXT | 08:24 |
| mvo | mborzecki: thanks, looking | 08:24 |
| Chipaca | apple going off and doing its own CPUs, not doing anything really new, fans still being fans | 08:25 |
| zyga | mborzecki: ack | 08:25 |
| zyga | thank you :) | 08:25 |
| mborzecki | mvo: that's 2.32.3 right? | 08:25 |
| zyga | mborzecki: did you shellcheck everything new? | 08:25 |
| zyga | Chipaca: it's the 90s but we have linux | 08:25 |
| Chipaca | zyga: i had linux in the 90s | 08:26 |
| mvo | mborzecki: yes | 08:26 |
| zyga | but linux now sucks less ;-) | 08:26 |
| mborzecki | zyga: yeah, it will complain about -exec and recommend -execdir (but that's expected) | 08:26 |
| Chipaca | zyga: i've been on linux since 1994, and i switched because it sucked less | 08:26 |
| mvo | mborzecki, zyga hm, if shellcheck does not like, whats the advantage over find|xargs? | 08:27 |
| zyga | mborzecki: interesting, I didn't know about execdir | 08:27 |
| zyga | can we just use execdir then | 08:27 |
| zyga | it looks very sane | 08:27 |
| mborzecki | force push? | 08:27 |
| zyga | mvo: xargs has size limits | 08:27 |
| mborzecki | (trying to keep the commit count low) | 08:27 |
| pedronis | there's -n | 08:27 |
| pedronis | though | 08:27 |
| zyga | and you can use + form to make fewer calls to rm even :) | 08:27 |
| mvo | mborzecki: we can squash it on merge but I'm fine with force push too | 08:28 |
| mborzecki | heh, -n, + bikeshedding :P | 08:28 |
| mvo | yeah, this feels like we are deep in bikesheed land :) | 08:28 |
| zyga | it's shell ;) | 08:29 |
| zyga | it's the perfect candidate | 08:29 |
| Chipaca | zyga: what's that about size limits? | 08:29 |
| mup | PR snapd#4975 opened: osutil: fix fstab parser to allow for # in field values (2.32) <Created by zyga> <https://github.com/snapcore/snapd/pull/4975> | 08:30 |
| zyga | Chipaca: xargs wasn't smart and could run into issues with the size of cmdline | 08:31 |
| zyga | that's why I traditionally don't like using it | 08:31 |
| BjornT_ | mvo: hi! do you know why core18 wasn't pushed to the store, even though it was built successfully for amd64? does it wait for all architectures to build? | 08:32 |
| zyga | https://www.irccloud.com/pastebin/srWqXthX/ | 08:32 |
| zyga | Chipaca: ^ | 08:32 |
| mborzecki | zyga: Chipaca: iirc xargs will call the command a number of times if it hits a limit, but i'm not sure if that applies to all versions in the wild | 08:33 |
| mborzecki | (or anything else than gnu) | 08:33 |
| mvo | BjornT_: let me check | 08:33 |
| Chipaca | zyga: ah | 08:33 |
| mvo | BjornT_: I manually published it now, but let me try to figure out why it wasn't done so automatically | 08:34 |
| Chipaca | mborzecki: that's intrinsic to xargs, yes, but see zyga's pastebin | 08:35 |
| zyga | mvo: can we override gustavo's -1 since the issue is fixed now | 08:35 |
| zyga | https://github.com/snapcore/snapd/pull/4930/files | 08:35 |
| mup | PR #4930: skip test that requires internet when not present <Created by rkitover> <https://github.com/snapcore/snapd/pull/4930> | 08:35 |
| mborzecki | Chipaca: ack | 08:35 |
| Chipaca | how do I use adt-buildvm-ubuntu-cloud to get a bionic image for spread? | 08:35 |
| Chipaca | in xenial | 08:35 |
| zyga | (this is from man xargs) | 08:36 |
| Chipaca | (it looks for a .disk1.img that's not there) | 08:36 |
| mvo | zyga: looking | 08:36 |
| mvo | Chipaca: adt-buildvm-ubuntu-cloud -r bionic does not work? | 08:36 |
| BjornT_ | mvo: thanks! i did confirm that the built core18 snap worked together with maas and snapcraft 2.40, btw | 08:37 |
| Chipaca | mvo: it tries “Downloading https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64-disk1.img” and then crashes with a 404 | 08:37 |
| Chipaca | 404 -> traceback, because python | 08:37 |
| mborzecki | Chipaca: iirc cloudimg' don't have .disk1.img | 08:37 |
| Chipaca | but 404 nontheless :-) | 08:37 |
| Chipaca | nonetheless* | 08:37 |
| mvo | Chipaca: oh, indeed, iirc you need to do: "sudo apt install -t xenail-backports apport" | 08:37 |
| mvo | Chipaca: to get the non-ancient version of apport | 08:37 |
| mvo | Chipaca: eh, adt | 08:37 |
| Chipaca | _apport_? for cloud images? | 08:37 |
| mvo | Chipaca: silly me | 08:37 |
| Chipaca | ah :-) | 08:37 |
| mvo | Chipaca: autopkgtest | 08:37 |
| mvo | Chipaca: but let me quickly double check | 08:38 |
| mvo | Chipaca: yes, please try that | 08:38 |
| mvo | (the version of *autopkgtest* from xenial-backports) | 08:38 |
| Chipaca | E: The value 'xenail-backports' is invalid for APT::Default-Release as such a release is not available in the sources | 08:38 |
| mvo | Chipaca: without my typo :( | 08:38 |
| * Chipaca fixes the typo and tries again | 08:38 | |
| * Chipaca is copy-pasting commands that start with 'sudo' | 08:38 | |
| Chipaca | mvo: I _trusted_ you! /o\ | 08:39 |
| mvo | Chipaca: sorry for letting you down! | 08:39 |
| Chipaca | I'm used to it by now :-p | 08:39 |
| Chipaca | anyway, that worked, thanks! | 08:39 |
| * mvo weeps a bit in the corner | 08:39 | |
| mvo | Chipaca: happy that it worked, this should give you an image | 08:39 |
| Chipaca | yep yep, it's doing its thing | 08:40 |
| mborzecki | damn, forgot to start the pomodoro timer | 08:40 |
| mvo | mborzecki: for what specifically? | 08:42 |
| mborzecki | mvo: work :P | 08:42 |
| mborzecki | mvo: there's a nice gnome shell extension http://gnomepomodoro.org/ | 08:43 |
| mvo | mborzecki: do you use a physical one or software? if software, which one | 08:43 |
| mvo | mborzecki: heh - you answered already | 08:43 |
| timp | hello | 08:43 |
| timp | I think snap or apparmor broke my lxd install. Is this the correct place to look for help? | 08:44 |
| zyga | hey timp | 08:44 |
| zyga | yes, what happeend? | 08:44 |
| zyga | happened | 08:44 |
| timp | zyga: thanks :) | 08:44 |
| mborzecki | mvo: forces breaks on you, reminds me to do some stretching, squats etc | 08:44 |
| timp | yesterday, my lxd worked fine. Today it does not. (I did an apt upgrade yesterday, that may be related. No snap refresh though) | 08:44 |
| timp | $ lxc list | 08:44 |
| Chipaca | mborzecki: zyga: not sure why I'm looking at this, but freebsd's find also has -execdir (but its xargs doesn't have --show-limits) | 08:44 |
| timp | cat: /proc/self/attr/current: Permission denied | 08:44 |
| timp | /snap/lxd/6492/commands/lxc: 6: exec: aa-exec: Permission denied | 08:44 |
| zyga | timp: snaps refresh automatically | 08:44 |
| timp | dmesg shows this: | 08:44 |
| timp | [ 1140.618728] audit: type=1400 audit(1522829695.691:122): apparmor="DENIED" operation="open" profile="snap.lxd.lxc" name="/proc/8365/attr/current" pid=8365 comm="cat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 | 08:44 |
| timp | [ 1140.619047] audit: type=1400 audit(1522829695.691:123): apparmor="DENIED" operation="exec" profile="snap.lxd.lxc" name="/usr/sbin/aa-exec" pid=8352 comm="lxc" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0 | 08:45 |
| zyga | timp: can you check if "snap changes" has anything about lxd? | 08:45 |
| timp | yes, it did: | 08:45 |
| timp | 93 Done 2018-04-04T07:44:00Z 2018-04-04T07:45:55Z Auto-refresh snaps "lxd", "aws-cli", "core" | 08:45 |
| zyga | timp: and can you paste "snap interfaces | grep lxd" | 08:45 |
| zyga | timp: ok | 08:45 |
| timp | :lxd-support lxd | 08:45 |
| timp | :network lxd,nextcloud-client,simplenote,telegram-sergiusens | 08:45 |
| timp | :system-observe lxd | 08:45 |
| timp | lxd:lxd | 08:45 |
| zyga | hmm all looks good here | 08:46 |
| zyga | let me look deeper | 08:46 |
| timp | thanks | 08:47 |
| pedronis | isn't there a discussion about in the forum | 08:47 |
| timp | I can 'cat /proc/self/attr/current', so that permission denied is weird. I don't know how AppArmor works though. | 08:47 |
| zyga | oh, perhaps | 08:47 |
| pedronis | is this relevant: curl https://assertions.staging.ubuntu.com/v1/assertions/account-key?account-id=canonical | 08:48 |
| Chipaca | mborzecki: what version of systemd do you have over there? (on arch i mean :-) ) | 08:48 |
| pedronis | sorry | 08:48 |
| pedronis | is this relevant: https://forum.snapcraft.io/t/2-0-lxd-snap-fails-on-sytems-with-partial-apparmor-support/4707 | 08:48 |
| mborzecki | Chipaca: 238 | 08:48 |
| zyga | ahh | 08:48 |
| zyga | timp: can you paste "snap version" | 08:48 |
| timp | snap 2.32.1 | 08:49 |
| timp | snapd 2.32.1 | 08:49 |
| timp | series 16 | 08:49 |
| timp | ubuntu 16.04 | 08:49 |
| timp | kernel 4.13.0-37-generic | 08:49 |
| timp | I'm on xenial | 08:49 |
| zyga | timp: thank you | 08:50 |
| zyga | so this is not a partial apparmor support issue | 08:50 |
| zyga | ok, I'll look at what I was meant to before | 08:50 |
| timp | what do you mean with partial apparmor support issue? | 08:50 |
| mborzecki | zyga: mvo: force pushed to 4972 | 08:50 |
| zyga | timp: debian and ubuntu have different set of supported apparmor features because ubuntu still carries a patch with non-upstreamed extensions | 08:50 |
| zyga | timp: can you please pastebin /var/lib/snapd/apparmor/profiles/snap.lxd.lxc | 08:51 |
| * zyga -> brb | 08:52 | |
| timp | zyga: sure, http://paste.ubuntu.com/p/hrrQFkXYSx/ | 08:53 |
| timp | https://forum.snapcraft.io/t/snapped-lxd-has-stopped-working-aa-exec-doesnt-exist-in-the-snap/2356 seems similar, but there's no solution there | 08:54 |
| timp | hmm, this is unfortunate. I was doing all my work in containers so now I'm blocked. | 08:55 |
| timp | so, | 08:57 |
| timp | $ snap revert lxd | 08:57 |
| timp | lxd reverted to 3.0.0 | 08:57 |
| timp | tim@tim-XPS-13-9350:~$ lxc list | 08:57 |
| timp | that fixes it for now. | 08:57 |
| timp | looks like the lxd update broke stuff | 08:57 |
| ackk | timp, you might want to ask on #lxc-dev as well | 08:57 |
| zyga | re | 08:58 |
| zyga | timp: looking | 08:58 |
| zyga | timp: it looks like the profile is wrong | 08:59 |
| zyga | it doesn't contain contributions from lxd-support | 08:59 |
| timp | hmm, 'snap refresh' tells me that there are no updates for lxd, even though I just did snap revert lxd | 09:00 |
| zyga | can you snap disconnect lxd:lxd-support | 09:00 |
| zyga | and then re-connect it | 09:00 |
| zyga | and see if the profile is different | 09:00 |
| zyga | (copy the current profile out) | 09:00 |
| timp | note that after 'snap revert lxd', it is working again. And I cannot reproduce the problem any more. | 09:01 |
| zyga | right because you are on a different revision | 09:01 |
| zyga | you can look at the same file again | 09:01 |
| zyga | and the header will say @{SNAP_REVISION} = ... | 09:01 |
| timp | snap.lxd.lxc? | 09:01 |
| zyga | yes | 09:01 |
| zyga | in the pastebin it was 6492 | 09:02 |
| timp | this is with the working version: http://paste.ubuntu.com/p/vgJcv7dPRm/ | 09:02 |
| timp | hmm, also 6492? | 09:02 |
| zyga | right, this one is correct | 09:03 |
| zyga | the previous one looked like basic, unconnected (no interfaces) application | 09:03 |
| timp | ah, so what I did is 'snap revert lxd'. And then 'snap refresh lxd'. Now I again have 6492, and it still works. | 09:03 |
| zyga | pstolowski: ^ maybe some other bug? | 09:03 |
| zyga | it looks like a bug in snapd, more than a bug in lxd IMO | 09:04 |
| zyga | hmm hmm | 09:04 |
| zyga | can you add this to a forum thread or a bug report | 09:04 |
| zyga | I don't want to lose it here | 09:04 |
| timp | on LP? | 09:04 |
| zyga | yes, on snapd | 09:04 |
| timp | ok | 09:04 |
| pedronis | this a much more tested path than the snap install lxd (without anything) | 09:04 |
| pedronis | though | 09:05 |
| zyga | timp: please include: snap version; snap changes; snap interfaces (from what you did, not from what is is now when it works) | 09:06 |
| zyga | and the two profiles you made | 09:06 |
| zyga | (the pastebins) | 09:06 |
| zyga | I think it shows that we didn't really include the lxd-support snippets at all | 09:06 |
| zyga | pstolowski: the bug on the forum feels related now | 09:07 |
| zyga | the reporter there doens't have lxd-support | 09:07 |
| BjornT_ | mvo: didn't you publish the latest one? the one you published is still on glibc 2.26. | 09:07 |
| BjornT_ | mvo: this is the one i used when i confirmed it worked with maas: https://code.launchpad.net/~mvo/+snap/core18/+build/182036/+files/core18_very-unstable_amd64.snap | 09:07 |
| * zyga afk again | 09:08 | |
| timp | zyga, pedronis: thanks for the help. I reported the bug here: https://bugs.launchpad.net/snapd/+bug/1761115 | 09:11 |
| mup | Bug #1761115: After lxd snap upgrade, it lxc stopped working <snapd:New> <https://launchpad.net/bugs/1761115> | 09:11 |
| zyga | timp: thank you | 09:11 |
| mvo | BjornT_: indeed, it was an older one | 09:11 |
| mvo | BjornT_: pushed a new one to edge, | 09:13 |
| mvo | BjornT_: and I think wgrant helped me fix the auto-upload issue, so hopefully this will be fixed now (testing this right now) | 09:14 |
| ackk | mvo, hi, do you know why I'm getting this error when trying to switch to the channel? https://pastebin.canonical.com/p/KywXTYgC6y/ | 09:15 |
| BjornT_ | mvo: nice, thanks | 09:16 |
| mvo | ackk: this looks like snapd crashed or something, what do you see in "journalctl -u snapd"? anything that indicates a panic? | 09:20 |
| ackk | mvo, no panics, but repeated errors like: | 09:21 |
| ackk | Jan 08 15:11:47 maas systemd[1]: snapd.service: Failed to set invocation ID on control group /system.slice/snapd.service, ignoring: Operation not permitted | 09:21 |
| ackk | Jan 08 15:11:47 maas systemd[1]: snapd.service: Failed to reset devices.list: Operation not permitted | 09:21 |
| ackk | mvo, fwiw I can install/remove snaps | 09:21 |
| ackk | it's just this operation that seems to fail | 09:21 |
| ackk | mvo, removing the base and reinstalling from the store works | 09:21 |
| mvo | ackk: ok, I will try to reproduce this, the EOF looks suspicious | 09:22 |
| ackk | mvo, btw it seems snapd doesn't check if you have snaps that depend on a base you remove? | 09:23 |
| ackk | I just removed core18 and maas which depends on it was installed | 09:23 |
| Chipaca | ackk: mvo: yeah that EOF usually means snapd went away mid-request (typically because of a crash) | 09:24 |
| ackk | Chipaca, but snapd seems to work fine for everythingelse | 09:24 |
| Chipaca | ackk: systemd'll restart it unless it does it too often | 09:24 |
| Chipaca | and yes, snapd does not do proper dependency management (yet) | 09:25 |
| ackk | Chipaca, weird, I thought I saw an error when trying to remove a snap before | 09:25 |
| Chipaca | ackk: it'll block core, gadget and kernel removes | 09:25 |
| ackk | oh I see | 09:26 |
| pedronis | we don't have that check for bases in use | 09:26 |
| Chipaca | basically things you can't come back from :) | 09:26 |
| Chipaca | pedronis: nor for default providers | 09:26 |
| pedronis | afaik | 09:26 |
| pedronis | well default providers are a bit of different issue | 09:26 |
| Chipaca | we probably should at least warn / prompt | 09:26 |
| pedronis | they are optional | 09:26 |
| Chipaca | pedronis: about as optional as bases | 09:27 |
| pedronis | not that we know what thât means | 09:27 |
| pedronis | Chipaca: well, | 09:27 |
| Chipaca | pedronis: try running gnome-calculator without gnome-16.04-whatevs | 09:27 |
| pedronis | it dpends | 09:27 |
| pedronis | Chipaca: they are installed as if they were optional | 09:27 |
| Chipaca | yes | 09:27 |
| Chipaca | and we don't have, afaik, a way of expressing optionalness | 09:27 |
| Chipaca | optionality | 09:27 |
| Chipaca | optionionness | 09:28 |
| pedronis | which mostly means we don't make a fuss | 09:28 |
| pedronis | if they don't install | 09:28 |
| pedronis | the snap uses them might | 09:28 |
| pstolowski | zyga: ack. i'm working on auto-connect PR atm, I can look at this in a bit | 09:29 |
| zyga | ok | 09:29 |
| pedronis | pstolowski: are you looking into the spread test for that? afaict is likely a very untested area atm | 09:30 |
| zyga | pstolowski, mvo: https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850/16 | 09:32 |
| zyga | it's not just that we didn't connect | 09:32 |
| zyga | we didn't even _load_ interfaces somehow | 09:32 |
| zyga | this is very very nasty | 09:32 |
| pstolowski | pedronis: not sure how to spread-test this particular case yet (other than by removing autoconnect task with jq, but that's not a good test). for regular auto-connect with current snapd we do have spread test(s) | 09:32 |
| pedronis | pstolowski: we need to start from the deb in the archive | 09:33 |
| pstolowski | pedronis: we can, but that's a moving piece no? | 09:34 |
| pedronis | pstolowski: it is | 09:34 |
| pedronis | but not different than what we do in the current upgrade test | 09:34 |
| pstolowski | let me see | 09:35 |
| pedronis | as I wrote probably a bit annoying because it will need the fakestore | 09:35 |
| pedronis | because we hardcode how we get core | 09:35 |
| * Chipaca -> break | 09:42 | |
| pstolowski | pedronis: i think in the existing tests we install whatever version of snapd we have in distro (or whatever we build locally); now we would need an old version that doesn't have the feature | 09:43 |
| pstolowski | zyga: yes, this looks rather bad.. | 09:44 |
| pedronis | pstolowski: xenial has 2.29 | 09:45 |
| pstolowski | pedronis: ah, indeed, and it will never get updated, you're right | 09:46 |
| pedronis | well, it will | 09:46 |
| pstolowski | hmm | 09:47 |
| pedronis | but is good enough for testing the current problem | 09:47 |
| pedronis | as it's happening | 09:47 |
| pstolowski | ok, fair enough | 09:48 |
| mvo | BjornT_, ackk, wgrant auto-upload of core18 works now, thanks for reporting and helping with the fix | 09:48 |
| pstolowski | would be great to have a test that's stays valid for longer time | 09:48 |
| ackk | mvo, great, thank you! | 09:52 |
| zyga | Chipaca: https://github.com/snapcore/snapd/pull/4973 needs a 2nd review for the release | 09:57 |
| mup | PR #4973: osutil: fix fstab parser to allow for # in field values <Created by zyga> <https://github.com/snapcore/snapd/pull/4973> | 09:57 |
| zyga | Chipaca: fstab, wanna take it? | 09:57 |
| pedronis | pstolowski: well, we have unit tests too | 09:58 |
| zyga | mvo: can I remove the snappy.upstream part when we merge the other script that makes the input tarball | 09:59 |
| zyga | that branch is green and it doesn't hurt much | 09:59 |
| pedronis | pstolowski: also hopefully at some point soon we will get epochs and testing jumping so much between revision will be a bit less needed | 10:00 |
| mvo | zyga: ok, fwiw, I am working on git-buildpackage right onw | 10:00 |
| zyga | Thanks! | 10:01 |
| mup | PR snapd#4938 closed: release-tools: add repack-debian-tarball.sh <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4938> | 10:01 |
| mup | PR snapd#4972 closed: cmd/snap-mgmt: remove timers, udev rules, dbus policy files <Created by bboozzoo> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4972> | 10:02 |
| zyga | mborzecki: can you please provide a backport for this branch for 2.32 ^ | 10:02 |
| Chipaca | zyga: looking | 10:04 |
| mup | PR snapd#4976 opened: cmd/snap-mgmt: remove timers, udev rules, dbus policy files (2.32) <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4976> | 10:08 |
| zyga | mborzecki: wrong target branch | 10:08 |
| mborzecki | zyga: should be fixed now | 10:09 |
| mup | PR snapd#4977 opened: debian: add gbp.conf script to build snapd via `gbp buildpackage` <Created by mvo5> <https://github.com/snapcore/snapd/pull/4977> | 10:16 |
| === Beret- is now known as Beret | ||
| pedronis | mvo: what's the status of your review of #4900, got half through it yesterday and then new emergencies today? | 10:18 |
| mup | PR #4900: many: use the new install/refresh API by switching snapstate to use store.SnapAction <Critical> <Squash-merge> <Created by pedronis> <https://github.com/snapcore/snapd/pull/4900> | 10:18 |
| mvo | pedronis: yes, continuing now | 10:24 |
| zyga | is the store having a slow ceph day again? | 10:29 |
| === oSoMoN_ is now known as oSoMoN | ||
| pedronis | not that I know of, a little bit of load with the release yesterday | 10:42 |
| zyga | mvo: can I merge the backport https://github.com/snapcore/snapd/pull/4975 | 10:48 |
| mup | PR #4975: osutil: fix fstab parser to allow for # in field values (2.32) <Created by zyga> <https://github.com/snapcore/snapd/pull/4975> | 10:48 |
| mup | PR snapd#4973 closed: osutil: fix fstab parser to allow for # in field values <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4973> | 10:48 |
| zyga | the master branch had 2 +1s and I just merged it | 10:48 |
| zyga | pedronis: can you please do a 2nd review on release-critical https://github.com/snapcore/snapd/pull/4969 fix from mvo | 10:48 |
| mup | PR #4969: interfaces: make system-key more robust against invalid fstab entries <Squash-merge> <Created by mvo5> <https://github.com/snapcore/snapd/pull/4969> | 10:48 |
| zyga | jdstrand: hey, can you please enqueue https://github.com/snapcore/snapd/pull/4545 for re-review | 10:50 |
| mup | PR #4545: interfaces/x11: allow X11 slot implementations <Created by gerboland> <https://github.com/snapcore/snapd/pull/4545> | 10:50 |
| * zyga needs to plan an errand today :/ | 10:52 | |
| zyga | are we doing the standup earlier? (now?) | 10:52 |
| zyga | mvo: when is the standup today | 10:52 |
| pedronis | 2pm we agreed | 10:53 |
| zyga | ah, that's good then | 10:53 |
| zyga | I can do the errand after that | 10:53 |
| zyga | thanks | 10:53 |
| cachio | mvo, hey | 11:24 |
| mvo | hey cachio | 11:25 |
| cachio | what time are we making the standup today? | 11:25 |
| mvo | cachio: in 35min | 11:25 |
| mvo | cachio: hope that works for you | 11:25 |
| mvo | cachio: if not its ok to skip | 11:25 |
| cachio | mvo it doesn't work today | 11:26 |
| cachio | mvo, I have to go to my son's school | 11:26 |
| cachio | are we making all wednesday at this time? | 11:27 |
| cachio | every wednesday? | 11:27 |
| mborzecki | zyga: mvo: pushed updates to #4942 | 11:27 |
| mup | PR #4942: cmd/snap: user session application autostart v3 <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4942> | 11:27 |
| zyga | thanks | 11:27 |
| mvo | mborzecki: ta | 11:27 |
| mborzecki | standup is @ 2pm? | 11:27 |
| mvo | cachio: no problem, we need to discuss tomorrow/next week what time to pick then | 11:28 |
| zyga | yes | 11:28 |
| mborzecki | ack | 11:28 |
| mvo | cachio: we also need a time that works for gustavo | 11:28 |
| mvo | cachio: and for you obviously | 11:28 |
| cachio | mvo, it works for me but not today | 11:29 |
| cachio | but I'll block the calendar | 11:29 |
| mup | PR snapd#4930 closed: skip test that requires internet when not present <Created by rkitover> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4930> | 11:30 |
| mvo | cachio: ok, thank you | 11:30 |
| cachio | mvo, is it gonna by just on wednesdays? or every day? | 11:31 |
| mvo | cachio: only (some) wednesdays | 11:32 |
| cachio | mvo, | 11:32 |
| cachio | ok | 11:32 |
| cachio | mvo, when is comming the new sru? | 11:33 |
| zyga | pstolowski: https://github.com/snapcore/snapd/pull/4968 is broken on tests FYI | 11:39 |
| mup | PR #4968: RFC: ifacemgr: remove stale connections on startup <Created by stolowski> <https://github.com/snapcore/snapd/pull/4968> | 11:39 |
| mup | PR snapd#4975 closed: osutil: fix fstab parser to allow for # in field values (2.32) <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4975> | 11:40 |
| pstolowski | zyga: yep i know, i had to park this for a while due to autoconnect | 11:40 |
| zyga | ack | 11:40 |
| zyga | thanks | 11:40 |
| pstolowski | zyga: i have fixes and will also add new test | 11:41 |
| jdstrand | zyga (cc timp): that was the exact issue I saw yesterday. apparmor denial on attr/current and aa-exec. no interfaces listed for lxd with snap interfaces. I had to stop/start snapd to get them back then do 'sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.lxd.* | 11:45 |
| zyga | jdstrand: thank you for confirming that, I wonder what could be the cause | 11:45 |
| zyga | jdstrand: wich core channel were you on then? | 11:45 |
| zyga | I was suspecting some window when snapd restarts while there's an unmounted lxd snap | 11:45 |
| zyga | jdstrand: it feels like the place when we read snap.Info is flaky in that the error is non-fatal | 11:46 |
| zyga | and we carry on knowing *nothing* about that snap | 11:46 |
| jdstrand | zyga (cc timp): oh I forgot. I did: sudo systemctl stop snapd ; sudo systemctl start snapd # now interfaces show up) ; sudo snap disconnect lxd:lxd-support ; sudo snap connect lxd:lxd-support # now the interfacec works | 11:51 |
| * cachio afk | 11:51 | |
| zyga | yeah, that's that | 11:51 |
| jdstrand | ie, I tried apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.lxd.* before the disconnect/connect and realized it didn't have the policy in place | 11:52 |
| jdstrand | zyga: I'm on beta still | 11:52 |
| jdstrand | 16-2.32.2 | 11:52 |
| zyga | yep | 11:53 |
| zyga | that all hints to missing snap info | 11:53 |
| zyga | and missing interfaces | 11:53 |
| jdstrand | zyga: it may not be related, but lxd-support is one of the few interfaces with an installation constraint | 11:55 |
| zyga | oh, what is that constraint? | 11:57 |
| zyga | are you referring to the policy? | 11:57 |
| jdstrand | zyga: base declaration | 11:58 |
| jdstrand | zyga: allow-installation: false | 11:59 |
| zyga | right | 11:59 |
| jdstrand | zyga: so a snap from the store must have a snap declaration to install the snap (--dangerous installs do not) | 12:00 |
| jdstrand | I dono't know if it is related | 12:00 |
| jdstrand | don't | 12:00 |
| zyga | jdstrand: can you look at journalctl snapd.service | 12:00 |
| zyga | there's a log if we drop an interface | 12:00 |
| zyga | if it was that we'd have a trace | 12:00 |
| jdstrand | but it is the only snap I know that has this | 12:00 |
| zyga | that'd be a nice hit to what the issue is | 12:01 |
| zyga | it may be that we did read it but then skipped it | 12:02 |
| jdstrand | zyga: I experienced the problem yeserday. That put's us at something happening between 18:00 on Apr 2 through 10:25 Apr 3. I've pasted since Mar27 snapd start up through today here: https://paste.ubuntu.com/p/ySSrxXRgyK/ | 12:05 |
| jdstrand | puts* | 12:05 |
| zyga | thank you | 12:05 |
| zyga | I'll review it after standup | 12:06 |
| jdstrand | zyga: there is some weird stuff in there | 12:06 |
| zyga | woah | 12:06 |
| zyga | Apr 03 10:25:29 localhost snapd[27535]: 2018/04/03 10:25:29.156759 helpers.go:214: cannot connect plug "lxd-support" from snap "lxd", no such plug | 12:06 |
| jdstrand | Mar 27 12:44:27 localhost snapd[2848]: 2018/03/27 12:44:27.304321 stateengine.go:101: state ensure error: cannot refresh snap-declaration for "lxd": Get https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/J60k4JY0HppjwOjW8dZdYc8obXKxujRu?max-format=2: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving | 12:06 |
| zyga | pstolowski: ^ we cannot merge your fix for stale connections before we get to the bottom of this | 12:06 |
| zyga | or we'll mess up real state | 12:06 |
| zyga | thank you | 12:06 |
| jdstrand | zyga: note that the snap declaration wasn't pulled down days before | 12:06 |
| jdstrand | I don't know if later fetches worked out ok | 12:07 |
| jdstrand | at 10:25 on Apr 3, that is when I stopped/started lxd manually | 12:08 |
| jdstrand | actually, maybe that was 11:37 | 12:08 |
| * jdstrand checks irc backscroll | 12:09 | |
| jdstrand | ah, it was 11:37 | 12:09 |
| jdstrand | the thing at 10:25 was not me. 2708 Done 2018-04-03T15:24:34Z 2018-04-03T15:26:04Z Auto-refresh snaps "chromium", "lxd", "core" | 12:10 |
| jdstrand | that is in UTC and corresponds to the 10:25 restart | 12:11 |
| zyga | jdstrand: did you see any snaps listed as broken at that time? | 12:15 |
| zyga | jdstrand: we are discussing this issue now and we understand how it happens mechanically (we should be albe to reproduce that) but we have no idea (yet) why it happens | 12:16 |
| jdstrand | zyga: no | 12:19 |
| jdstrand | https://forum.snapcraft.io/t/snapped-lxd-has-stopped-working-aa-exec-doesnt-exist-in-the-snap/2356/27 | 12:20 |
| jdstrand | I lined up all the times | 12:20 |
| jdstrand | it looks like at 10:24 *both* core and lxd were updated | 12:20 |
| jdstrand | refreshed* | 12:20 |
| zyga | yes, its' something we suspect | 12:20 |
| jdstrand | then at 11:33 I noticed things went awry | 12:21 |
| zyga | if snapd starts when snap (or snaps) are unmounted we would hit this | 12:21 |
| zyga | it was just lxd for you, right? | 12:21 |
| jdstrand | and 11:37 I stopped/started manually and 11:39 diconnected/connected | 12:21 |
| zyga | (only a subset of the refreshed snaps could be affected) | 12:21 |
| jdstrand | zyga: it was what I noticed. it seems chromium could've been affected (it was refreshed at the same time) | 12:21 |
| zyga | but wasn't? | 12:22 |
| jdstrand | I don't use it regularly | 12:22 |
| * jdstrand uses firefox | 12:22 | |
| jdstrand | so I don't know | 12:22 |
| zyga | we need to look at ordering and at things like ensuring that when systemd says "it's mounted" it really really is | 12:22 |
| zyga | thank you | 12:22 |
| jdstrand | I can say the firefox snap wasn't misbehaving | 12:22 |
| popey | I have a core system which booted yesterday but today hangs | 12:26 |
| popey | https://usercontent.irccloud-cdn.com/file/d1gWsl07/IMG_20180404_132619.jpg | 12:27 |
| popey | Where should I file this? (can I revert back to previous core [given I can't boot it]) ? | 12:27 |
| * kalikiana lunch | 12:32 | |
| popey | mvo: ^ do you know where I should file this? | 12:33 |
| jdstrand | zyga: fyi, I added a few more comments | 12:33 |
| mborzecki | zyga: snap switch && snap refresh ? | 12:35 |
| mvo | popey: what happens if you power-cycle it? it always hangs at this point? | 12:35 |
| zyga | jdstrand: we have a way to loop through this code to hit the issue ifff it is a timing bug | 12:35 |
| popey | mvo: hangs at the same point | 12:35 |
| zyga | mborzecki: yes, on a pair of core + lxd | 12:35 |
| mvo | popey: the forum is a good place for a report - also here, because OMG and we want to know about it | 12:35 |
| mvo | popey: what is strange is that there is only kernel message, nothing from userspace AFAICT | 12:35 |
| mvo | popey: which indicates a kernel problem, can you get to the grub prompt? | 12:35 |
| mvo | popey: also even if its a kernel problem the rollback magic should have saved us/you :( | 12:36 |
| mvo | popey: so … | 12:36 |
| popey | oh man, 4th boot now it's moving | 12:36 |
| mvo | popey: *cough* | 12:36 |
| mvo | popey: magic! :) :/ :( | 12:36 |
| popey | :S | 12:36 |
| zyga | logger.Noticef("cannot retrieve info for snap %q: %s", snapName, err) | 12:36 |
| zyga | jdstrand: can you grep in your full journal for any hint of that | 12:36 |
| mvo | popey: moving in what direction? successful boot? or just hangs differently? | 12:36 |
| popey | its successfully booted now | 12:36 |
| popey | <shrug emoji> | 12:37 |
| zyga | mvo: if we had support for hardware watchdog we would be able to recover such issues automatically (what popey ran into) | 12:37 |
| popey | it totally hung 3 times, but this time it went through, no idea why. | 12:37 |
| zyga | jdstrand: I also totally missed this | 12:38 |
| zyga | Mar 27 12:44:27 localhost snapd[2848]: 2018/03/27 12:44:27.303347 autorefresh.go:327: Cannot prepare auto-refresh change: cannot refresh snap-declaration for "lxd": Get https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/J60k4JY0HppjwOjW8dZdYc8obXKxujRu?max-format=2: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving | 12:38 |
| zyga | if there's no assertion, we should not even attempt to continue refreshing | 12:38 |
| mvo | zyga: ohhhh | 12:38 |
| mvo | zyga: that is an interessting hint | 12:38 |
| zyga | Apr 03 10:25:29 localhost snapd[27535]: 2018/04/03 10:25:29.156224 helpers.go:214: cannot connect plug "gsettings" from snap "chromium", no such plug | 12:38 |
| zyga | but it seems the issue did affect both chromium and lxd | 12:39 |
| zyga | so unclear if this is just the assertion refresh | 12:39 |
| zyga | Apr 03 10:25:29 localhost snapd[27535]: 2018/04/03 10:25:29.156232 helpers.go:214: cannot connect plug "home" from snap "chromium", no such plug | 12:39 |
| zyga | for instance home is not gated | 12:39 |
| zyga | so not having an assertion would not change anything | 12:39 |
| zyga | I will try to reproduce this manually now | 12:42 |
| zyga | mvo: I just confirmed that logger.Noticef does *not* work | 12:43 |
| zyga | probably because at that stage, logger is just not setup yet | 12:43 |
| zyga | Chipaca: ^ | 12:43 |
| Chipaca | zyga: ? | 12:44 |
| zyga | mvo: I stopped snapd, stopped a mount unit of a random sanp, started snapd | 12:44 |
| zyga | what was logged was | 12:44 |
| zyga | kwi 04 14:43:28 fyke snapd[32408]: 2018/04/04 14:43:28.481566 helpers.go:106: invalid snap version: cannot be empty | 12:44 |
| Chipaca | zyga: logger is set up in snapd/main.go's init() | 12:44 |
| zyga | ha | 12:45 |
| Chipaca | zyga: so I call shenanigans :-) | 12:45 |
| Chipaca | in fact it's the first line of that init :-) | 12:45 |
| Chipaca | of course, that init runs after all other inits, because of the deps tree | 12:45 |
| Chipaca | but that's still pretty honking early :-) | 12:45 |
| zyga | grepping for that message on my system I see a case of snapd starting and not having snaps mounted | 12:45 |
| zyga | Chipaca: I will patch it to just print | 12:46 |
| zyga | but I suspect something is wrong there | 12:46 |
| popey | jdstrand: when you get a mo. can I have an exception for https://dashboard.snapcraft.io/snaps/emoj/revisions/19/ ? | 12:47 |
| popey | (it doesn't need a desktop file, but needs x11 interface) | 12:47 |
| pedronis | mvo: I applied feedback or answered in 4900 | 12:53 |
| zyga | ah | 12:54 |
| zyga | mvo: we say "cannot read snap %q: ..." and assign that to info.Broken | 12:54 |
| mvo | pedronis: thanks, I'm on the remaining bits now | 12:54 |
| zyga | we don't return an error | 12:54 |
| zyga | slightly funny to see this: https://github.com/snapcore/snapd/pull/4900#discussion_r179108664 | 13:04 |
| mup | PR #4900: many: use the new install/refresh API by switching snapstate to use store.SnapAction <Critical> <Squash-merge> <Created by pedronis> <https://github.com/snapcore/snapd/pull/4900> | 13:04 |
| mborzecki | Apr 04 13:04:33 ubuntu snapd[1037]: 2018/04/04 13:04:33.934552 store.go:1459: DEBUG: Hashsum error on download: sha3-384 mismatch for "core": got 86d2a911e843c48c8f49c459b5b7259a5ca786ad4e05b1f98a353082aee6dc79c0cc08ca9c09a3c603675a3acead29d9 but expected 7b354270492a85a54b44ad78c000e6a61ca38a49d5ab57d79a2e9d5eca20db9af89a1186b7b14d78ae232ec2f4824372 | 13:06 |
| mborzecki | that's unexpected | 13:06 |
| pedronis | zyga: maybe it should be moved lower? I'm open to input, in a meeting now though | 13:06 |
| zyga | pedronis: I was just adding similar logging | 13:06 |
| mborzecki | off to pick up the kids | 13:08 |
| mup | Issue snapcraft#2048 opened: Thank you <Created by jsamr> <https://github.com/snapcore/snapcraft/issue/2048> | 13:09 |
| pstolowski | pedronis, mvo : does this sound sensible/doable for autoconnect spread test: 1.snapd installed from distro deb; 2.snap download core; 3.unsquashfs core, inject new snapd, squash back; 4.snap download lxd; 5. point fake store at both modified core & lxd (plus do assertion magic which i'm not yet clear about); 6. snap install lxd? | 13:11 |
| mvo | pstolowski: in a meeting right now (and so is pedronis) sounds sane | 13:14 |
| mup | PR snapd#4978 opened: overlord,interfaces: be more vocal about broken snaps and read errors <Created by zyga> <https://github.com/snapcore/snapd/pull/4978> | 13:14 |
| zyga | mvo: extra logging I'd like to add to this release: https://github.com/snapcore/snapd/pull/4978 | 13:15 |
| mup | PR #4978: overlord,interfaces: be more vocal about broken snaps and read errors <Created by zyga> <https://github.com/snapcore/snapd/pull/4978> | 13:15 |
| zyga | also refuse adding broken snaps to repo | 13:15 |
| zyga | pstolowski: ^ | 13:16 |
| zyga | I think you are the only one to review now :) | 13:16 |
| pedronis | pstolowski: we already build that core that way at some point | 13:16 |
| pstolowski | pedronis: i know, I just need to prevent it from getting mounted immediately (it's update_core_snap_for_classic_reexec) | 13:18 |
| * zyga -> walk | 13:19 | |
| pstolowski | zyga: looks sane, quick questions: why "[stderr]"? where does Stderr go? just journal? | 13:23 |
| kalikiana | re | 13:24 |
| zyga | Yes | 13:24 |
| zyga | To error part of it | 13:24 |
| zyga | The [stderr] is to see if we are doing something wrong with o logger | 13:29 |
| mup | Bug #1761187 opened: ~/.snap/ not available for root on some systems <Snappy:New> <https://launchpad.net/bugs/1761187> | 13:30 |
| jdstrand | zyga: journalctl|grep 'cannot retrieve info for snap' returns nothing | 13:39 |
| zyga | Ack | 13:39 |
| zyga | I will setup a loop | 13:40 |
| zyga | Just afk now | 13:40 |
| mup | Bug #1761187 changed: ~/.snap/ not available for root on some systems <Snappy:New> <https://launchpad.net/bugs/1761187> | 13:42 |
| Chipaca | mup: you just said that | 13:43 |
| mup | Chipaca: I apologize. I'm a program with a limited vocabulary. | 13:43 |
| pstolowski | lol | 13:47 |
| cachio | lol | 13:48 |
| pstolowski | mup: Chipaca is making fun of you! | 13:49 |
| mup | pstolowski: I really wish I understood what you're trying to do. | 13:49 |
| cachio | :) | 13:49 |
| pstolowski | yeah me too | 13:50 |
| pedronis | Chipaca: can you look #4978 | 14:08 |
| mup | PR #4978: overlord,interfaces: be more vocal about broken snaps and read errors <Created by zyga> <https://github.com/snapcore/snapd/pull/4978> | 14:08 |
| pedronis | zyga: I think that code is logging too much, CurrentInfo will call readInfoAnyway | 14:09 |
| zyga | I can trim some, yeah | 14:12 |
| mvo | mborzecki: 4976 fails, I guess you know about this already(?) | 14:15 |
| mvo | zyga: any new data on the issue? | 14:17 |
| zyga | mvo: re, I was afk (kids/lunch/dog) | 14:18 |
| zyga | mvo: I did some digging and it seems we don't even return an error | 14:18 |
| zyga | mvo: we return snap info with Broken field set | 14:18 |
| zyga | mvo: I provided a patch that adds verbosity and barrages broken snaps from entering the repository | 14:18 |
| mvo | zyga: ok, I see the PR | 14:19 |
| mvo | zyga: thank you! | 14:19 |
| zyga | mvo: I haven't setup a loop that would reproduce the refresh cycle issue yet | 14:19 |
| pedronis | pstolowski: I think we can start with lxd, lxd is a bit strange in the sense that is both our typical case, but a bit fragile on its own | 14:19 |
| zyga | mvo: I will setup the loop soon, if I hit a case when it reproduces we will have some confirmation | 14:19 |
| mvo | zyga: ta | 14:19 |
| zyga | mvo: while that runs I will review tasks we do on refresh to look for issues | 14:19 |
| mvo | zyga: why the logger.Noticef() and the fmt.Fprintf() ? is the notice not sufficient? | 14:20 |
| pstolowski | pedronis: ok, that's what i'm basing my spread test on | 14:20 |
| zyga | mvo: maybe, I think so but wanted to double check | 14:20 |
| zyga | mvo: I will drop the fmt.Fprintf calls if logger works | 14:20 |
| zyga | better safe than sorry | 14:20 |
| pedronis | zyga: we don't read snaps that early, if there's some logging there should be the rest | 14:27 |
| jdstrand | popey: ok | 14:27 |
| popey | jdstrand: thanks | 14:28 |
| Chipaca | grah, grah, grah, everything is terrible | 14:36 |
| Chipaca | pedronis: looking | 14:36 |
| mvo | Chipaca: also 4969 if you have spare cycles for reviews | 14:37 |
| Chipaca | mvo: I do | 14:38 |
| Chipaca | for #1761193 I'm tempted to use 'cat' to read the auth.json file... :-/ | 14:39 |
| mup | Bug #1761193: ~/.snap/ not available for root on some systems <snapd:New> <https://launchpad.net/bugs/1761193> | 14:39 |
| === Eleventh_Doctor is now known as Pharaoh_Atem | ||
| Chipaca | as in ‘if running under sudo, do 'su -c "cat the/file" $SUDO_USER'’ | 14:39 |
| Chipaca | the alternative is to leave goroutines stuck, which might not be so terrible | 14:40 |
| Chipaca | as this is in client, they're short-lived | 14:40 |
| zyga | FYI this test always fails on my laptop | 14:44 |
| zyga | ... value *errors.errorString = &errors.errorString{s:"cannot obtain bus name 'io.snapcraft.Launcher'"} ("cannot obtain bus name 'io.snapcraft.Launcher'") | 14:44 |
| zyga | FAIL: cmd_userd_test.go:62: userdSuite.TestUserd | 14:44 |
| zyga | pedronis, mvo: updated https://github.com/snapcore/snapd/pull/4978/files | 14:45 |
| mup | PR #4978: overlord,interfaces: be more vocal about broken snaps and read errors <Created by zyga> <https://github.com/snapcore/snapd/pull/4978> | 14:45 |
| zyga | (force pushed for cherry pick) | 14:46 |
| zyga | dropped fprintfs | 14:46 |
| zyga | and one log | 14:46 |
| mvo | ta! | 14:46 |
| zyga | at the very least we will (maybe) get better logs next time | 14:47 |
| zyga | mvo: is .3 scheduled for today? | 14:48 |
| mvo | zyga: depends on when we get the fixes, but yes, would love to do it today | 14:49 |
| mvo | zyga: the "exec: aa-exec: Permission denied" is also a bit alarming | 14:49 |
| didrocks | kyrofa: hey! I hope you are well :) I'm really puzzled about bug #1761127 and can't even find a workaround | 14:51 |
| mup | Bug #1761127: On Travis (not a real vte), releasing to a branch name during snapcraft push prints a stacktrace <Snapcraft:New> <https://launchpad.net/bugs/1761127> | 14:51 |
| diddledan | is the libGL problem on Bionic (possibly related to nVidia binary drivers) with snaps being worked upon? : libGL error: No matching fbConfigs or visuals found | 14:52 |
| diddledan | I think it's because of the move to a unified loader? *looks for the right deb* .. `libglvnd0` | 14:53 |
| zyga | mvo: that is because aa-exec is not allowed by default template | 14:54 |
| zyga | mvo: when lxd does't have the lxd-support interface at all this would happen | 14:55 |
| pstolowski | mvo, zyga the spread test is painful, i'm learning through experimentation and trial-and-error | 14:55 |
| zyga | diddledan: can you try beta? it should be fixed there | 14:55 |
| pedronis | pstolowski: can I help somehow? | 14:55 |
| zyga | mvo: the fedora issue should be gone now | 14:55 |
| mvo | zyga: right, I can reproduce this here on 17.10 by purging snapd and then installing it | 14:55 |
| mvo | zyga: and then snap install lxd | 14:56 |
| zyga | mvo: oh? can you be more specifc | 14:56 |
| zyga | ah | 14:56 |
| mvo | pstolowski: yeah, its hard | 14:56 |
| zyga | but isn't that the auto-connect issue | 14:56 |
| zyga | or are you saying that you can reproduce by pruging, installing lxd and then see lxd without any interface at all | 14:56 |
| zyga | (as said by snap interfaces) | 14:56 |
| pedronis | pstolowski: we have other tests using the fakestore but probably not exactly how you need it | 14:56 |
| diddledan | zyga: ok, the beta fixes a game I've packaged (used for quick test) but the solus-runtime-based steam still fails | 14:57 |
| * diddledan wonders where ikey got to | 14:57 | |
| pstolowski | pedronis: yes, and there are differences, so not easy to tell what's crucial and what not; if you can spot what might be missing still https://pastebin.ubuntu.com/p/6Jd6rRZJGk/ then this could save me some iterations | 14:57 |
| zyga | diddledan: that is probably something for ikey and jdstrand | 14:57 |
| mvo | zyga: I did "apt purge snapd; apt install snapd/artful-updates; snap install lxd; lxc": /snap/lxd/6492/commands/lxc: 6: exec: aa-exec: Permission denied | 14:58 |
| mvo | zyga: lxd-support is not connected | 14:58 |
| zyga | mvo: does it exist? | 14:58 |
| pedronis | pstolowski: mvo: I fear we have a big issue | 14:58 |
| pedronis | with the tests | 14:58 |
| mvo | zyga: it does | 14:58 |
| zyga | if it exists, that's the bug that is well understood now that I believe pawel is fixing | 14:58 |
| zyga | ok | 14:58 |
| mvo | pedronis: what exactly? | 14:58 |
| * diddledan pokenprods jdstrand :-p | 14:58 | |
| zyga | in our case the bug is that the plug is gone, as evidenced on the forum... | 14:58 |
| mvo | zyga: ok, if its the same bug its all good | 14:58 |
| pedronis | pstolowski: mvo: we want to use the deb but also the fakestore, but by definition that's not possible | 14:58 |
| mup | PR snapd#4979 opened: overlord,interfaces: be more vocal about broken snaps and read errors (2.32) <Created by zyga> <https://github.com/snapcore/snapd/pull/4979> | 14:59 |
| mvo | pedronis: uhhh, indeed | 14:59 |
| pstolowski | pedronis: oh why | 14:59 |
| pedronis | pstolowski: mvo: the official deb will never trust the fakestore | 14:59 |
| pedronis | (that's a feature) | 14:59 |
| pedronis | but is annoying here | 14:59 |
| zyga | mvo: I sent a backport of the logging PR | 14:59 |
| === phoenix_firebrd is now known as phoenix_firebrd_ | ||
| zyga | and will try to reproduce this in a loop | 15:00 |
| mvo | pedronis: yeah, oh well | 15:00 |
| mvo | zyga: ta! | 15:00 |
| mvo | pstolowski: can you push what you have? without the spread test? | 15:00 |
| mvo | pstolowski, pedronis I think we understand the issue, so getting this into edge might be good as a first pass. | 15:01 |
| pedronis | mvo: pstolowski: at most we can write a daily test that uses the the deb and the snap in edge | 15:02 |
| pedronis | there's still value to that | 15:02 |
| pedronis | but cannot use what's on masrer | 15:03 |
| pstolowski | mvo: i can, let me just finish unit test | 15:03 |
| mvo | pstolowski: sure | 15:03 |
| mup | PR snapd#4974 closed: ifacestate: injectTasks helper <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/4974> | 15:04 |
| mvo | pstolowski: once that has landed we can test locally by modifying release/2.29 to install the edge core by default and running it locally. then we do "snap install lxd" (with no core yet) and we get the (fixed) core from edge and the re-exec should work | 15:04 |
| mvo | pstolowski: its not great but as pedronis said we have no way currently to test this for real | 15:04 |
| kyrofa | Hey there didrocks! I'll take a look this morning, see if we can figure it out | 15:04 |
| pstolowski | mvo: sounds good | 15:05 |
| didrocks | kyrofa: thanks! As it only happens on Travis, it's a little bit painful it seems. FTR, I tried with snapcraft release as well and as a simple edge/test in https://travis-ci.org/ubuntu/communitheme-snap-helpers/builds/362179292, but same issue. Same command with local "docker run" works perfectly | 15:06 |
| pedronis | pstolowski: sorry for making you chase impossible tests | 15:06 |
| didrocks | (but basically, our Travis instruction don't work on non branch releases) | 15:06 |
| pedronis | mvo: to write a daily we would need at least to have an envvar to change which channel we get core if it's not there | 15:07 |
| pstolowski | pedronis: np | 15:07 |
| kyrofa | didrocks, looks like you're successfully pushing, but then the store is giving us some error that we don't know how to parse | 15:07 |
| kyrofa | didrocks, how did you install snapcraft? | 15:08 |
| mup | PR snapd#4980 opened: Revert "spread.yaml: switch Fedora 27 tests to manual (#4962)" <Created by zyga> <https://github.com/snapcore/snapd/pull/4980> | 15:08 |
| kyrofa | Oh, the docker image | 15:08 |
| didrocks | kyrofa: docker run snapcore/snapcraft | 15:08 |
| didrocks | yep | 15:08 |
| didrocks | kyrofa: so, push is working, but release to a branchname don't | 15:08 |
| didrocks | release to "edge" works | 15:09 |
| didrocks | "edge/test" doesn't | 15:09 |
| kyrofa | didrocks, but it works locally? | 15:09 |
| kyrofa | How weird | 15:09 |
| didrocks | (but works locally, with a docker image) | 15:09 |
| didrocks | yeah :/ | 15:09 |
| didrocks | I think it's a vte vs non real vte thing… | 15:09 |
| kyrofa | didrocks, how are you authenticating? | 15:09 |
| didrocks | kyrofa: encrypted file creds, decrypted on build | 15:09 |
| didrocks | as told, replacing "edge/test" by "edge" only works, so, cred issues are out | 15:10 |
| kyrofa | didrocks, i.e. using export-login? | 15:10 |
| kyrofa | Or your own real creds? | 15:10 |
| didrocks | snapcraft enable-ci travis created an encrypted file and pushed env var to travis | 15:10 |
| kyrofa | didrocks, ah, then it is a creds issue | 15:10 |
| didrocks | oh? | 15:10 |
| didrocks | why edge would work when edge/test doesn't? | 15:11 |
| kyrofa | didrocks, enable-ci travis creates credentials that can only push to edge | 15:11 |
| didrocks | ahhhhhh | 15:11 |
| kyrofa | didrocks, you want export-login | 15:11 |
| zyga | Caelum: the instructions are merged now | 15:11 |
| didrocks | kyrofa: let me look on instructions for this | 15:11 |
| kyrofa | didrocks, use snapcraft export-login | 15:11 |
| kyrofa | didrocks, you can tune your creds how you like | 15:11 |
| didrocks | kyrofa: I think there is still the "it should tell you without printing a stacktrace" bug ;) | 15:11 |
| kyrofa | didrocks, oh totally, that we can fix | 15:12 |
| kyrofa | didrocks, snapcraft doesn't know what the creds can and can't do, so we rely on the store to tell us. But we get all sorts of different error formats from them :P | 15:12 |
| didrocks | fun ;) | 15:12 |
| didrocks | kyrofa: ok, I think I need to export that in a file, then, encrypt it before pushing | 15:13 |
| didrocks | shouldn't --channels=edge including --channels=edge/* ? | 15:13 |
| didrocks | include* | 15:13 |
| kyrofa | didrocks, to the length of my knowledge, the store ACLs don't support wildcards | 15:13 |
| kyrofa | cprov may know better | 15:13 |
| didrocks | so, basically, I need to give unrestricted access with my creds | 15:14 |
| didrocks | to create branch-on-the-go | 15:14 |
| kyrofa | didrocks, you can at least limit it to the specific snap | 15:14 |
| didrocks | (basically, based on the PR name) | 15:14 |
| didrocks | yeah | 15:14 |
| kyrofa | didrocks, also limit it only to uploads | 15:14 |
| didrocks | is the list of acls somewhere? | 15:14 |
| didrocks | (the help only mention acls option) | 15:15 |
| kyrofa | didrocks, https://dashboard.snapcraft.io/docs/api/macaroon.html#post--dev-api-acl- | 15:15 |
| didrocks | awesome! | 15:15 |
| kyrofa | I've been meaning to get those into snapcraft itself | 15:15 |
| kyrofa | cprov very helpfully documented them fo rus | 15:15 |
| didrocks | kyrofa: thanks a lot, I would never have figured out this restriction with this stacktrace! ;) | 15:15 |
| didrocks | at least, I'm unblocked now | 15:15 |
| kyrofa | didrocks, my pleasure! Should be easy to reproduce, I'll triage the bug and we'll get it fixed, thanks for the thorough report as always :) | 15:16 |
| didrocks | kyrofa: yw :-) | 15:16 |
| kyrofa | didrocks, by the way, are you building snaps per PR or something? | 15:19 |
| kyrofa | How do you plan on dealing with forks? | 15:20 |
| kyrofa | didrocks, by the way, if you don't want to encrypt, you can `travis env set SNAP_TOKEN "$(cat my-exported-login)"` and then use `echo "$SNAP_TOKEN" | snapcraft login --with -` | 15:22 |
| didrocks | kyrofa: yes, that's my plan. Basically, only PR made by core contributors will be built/testable this way | 15:23 |
| kyrofa | Gotcha | 15:23 |
| zyga | another bug getting in the way https://www.irccloud.com/pastebin/Q6aoFA0A/ | 15:23 |
| didrocks | I'm still ensure the build script can at least build on push for people to test | 15:23 |
| didrocks | kyrofa: excellent! I'll use this :) | 15:23 |
| zyga | bug.sh https://www.irccloud.com/pastebin/bzEWq7pw/ | 15:23 |
| zyga | not a happy day | 15:24 |
| kyrofa | didrocks, you might also like this: https://github.com/travis-ci/dpl/pull/800 | 15:25 |
| mup | PR travis-ci/dpl#800: Add snap provider <Created by kyrofa> <https://github.com/travis-ci/dpl/pull/800> | 15:25 |
| mvo | zyga: uhhh | 15:25 |
| zyga | I think we must add some code that waits for core restart with rest of setup | 15:25 |
| didrocks | kyrofa: oh, sounds that will enable me to reduce a lot what I'm doing right now | 15:26 |
| zyga | pedronis: ^ not sure what you would suggest for this, I'm looking at the snapd restart code now | 15:26 |
| zyga | mvo: FYI, I ran bug.sh exactly once :/ | 15:27 |
| === phoenix_firebrd_ is now known as phoenix_firebrd | ||
| zyga | mvo: holly cow | 15:34 |
| zyga | reproduced the bigger bug | 15:34 |
| zyga | 2nd run :| | 15:34 |
| zyga | I have lxd without lxd-support now | 15:34 |
| zyga | lxd-support bug trivially reproduced https://www.irccloud.com/pastebin/rootNBq3/ | 15:35 |
| zyga | details of the change refreshing core and lxd together | 15:36 |
| zyga | https://www.irccloud.com/pastebin/AxjIVck6/ | 15:36 |
| zyga | note that spawn and ready times are more interesting than task order | 15:37 |
| === phoenix_firebrd is now known as phoenix_firebrd_ | ||
| pedronis | zyga: there's no easy way to achieve that | 15:39 |
| pedronis | zyga: also there are no errors there? | 15:41 |
| pedronis | how does we get no interfaces | 15:42 |
| pedronis | and no errors | 15:42 |
| zyga | no, no errors | 15:42 |
| zyga | because the snap.Info is returned | 15:42 |
| zyga | just has Broken field set | 15:42 |
| zyga | :/ | 15:42 |
| pedronis | but that doesn't relate to core | 15:42 |
| zyga | we never return error if meta/snap.yaml cannot be found | 15:42 |
| zyga | pedronis: perhaps, I don't know yet | 15:42 |
| zyga | I ran it together to force a restart | 15:42 |
| zyga | this was done on my bionic laptop | 15:43 |
| mvo | zyga: yeah, I also have the lxd without lxd-support issue, its trivial to reproduce for me as well. but isn't that what pstolowski is fixing right now? | 15:44 |
| zyga | mvo: no | 15:45 |
| zyga | mvo: this is from stable core | 15:45 |
| zyga | mvo: and again, _threre is no plug_ | 15:45 |
| zyga | this is not a connect issue | 15:45 |
| zyga | this issue is that we have nothing to connect to | 15:45 |
| pedronis | do we think mount has happened but not? | 15:46 |
| pedronis | like we don't wait on mount enough | 15:46 |
| mvo | zyga: is https://www.irccloud.com/pastebin/raw/rootNBq3 the pastebin? if so, there is ":lxd-support -" in the snap intefaces? or am I confused? | 15:47 |
| zyga | https://www.irccloud.com/pastebin/F2qfHSP1/ | 15:47 |
| zyga | this is key | 15:47 |
| zyga | there's no plug at all | 15:47 |
| zyga | there must be a plug on the lxd snap | 15:47 |
| zyga | the slot you pasted is the implicit slot on core | 15:48 |
| zyga | which is there even if we cannot read core's snap.yaml | 15:48 |
| pstolowski | zyga: and you think this happens because we're not waiting for core install? | 15:48 |
| zyga | no, I don't know why yet | 15:48 |
| pedronis | what's the task task that does mount again | 15:48 |
| mvo | zyga: aha, thanks! that was what I was missing | 15:49 |
| zyga | more clear evidence of this bug (shorter) https://www.irccloud.com/pastebin/tmigy3Hi/ | 15:49 |
| zyga | pedronis: it is mount-snap | 15:49 |
| pedronis | zyga: there's no Mount at all in your changes | 15:49 |
| pedronis | afaict | 15:49 |
| zyga | whaaaa | 15:50 |
| zyga | is it another bug where old core vs new core taks are different? | 15:50 |
| pedronis | we didn't change it in a while | 15:51 |
| zyga | ??? https://www.irccloud.com/pastebin/XaWi8TtG/ | 15:51 |
| pedronis | but maybe we aren't setting up tasks correctly anymore | 15:51 |
| zyga | why is that conditional? | 15:51 |
| pedronis | that I don't know | 15:51 |
| pedronis | but in you case the condition should be false | 15:51 |
| pedronis | or so I hope | 15:51 |
| pedronis | but I don't see Mount in your pastebin | 15:52 |
| zyga | // check if we already have the revision locally (alters tasks) | 15:52 |
| zyga | revisionIsLocal := snapst.LastIndex(targetRevision) >= 0 | 15:52 |
| mvo | zyga: 2016 - thats a long time ago | 15:52 |
| zyga | "locally" is confusing here | 15:52 |
| zyga | it looks like switch specific artefact | 15:52 |
| pedronis | yes, it means it already there on disk | 15:52 |
| zyga | so perhaps red herring somehow | 15:52 |
| zyga | because they are both mounted | 15:52 |
| mvo | zyga: yeah, it will already be mounted | 15:53 |
| mup | PR snapd#4981 opened: ifacestate: inject autoconnect <Created by stolowski> <https://github.com/snapcore/snapd/pull/4981> | 15:53 |
| zyga | but good hunch | 15:53 |
| pedronis | but if they are mounted | 15:53 |
| pedronis | how can we have no plugs? | 15:53 |
| pstolowski | mvo: the PR: https://github.com/snapcore/snapd/pull/4981 | 15:53 |
| mup | PR #4981: ifacestate: inject autoconnect <Created by stolowski> <https://github.com/snapcore/snapd/pull/4981> | 15:53 |
| zyga | part of journal after affected snapd restart https://www.irccloud.com/pastebin/Rp3GCuu2/ | 15:54 |
| zyga | we add snaps first, reload connections next | 15:55 |
| zyga | whatever could cause this is beyond me now | 15:55 |
| zyga | though this code doesn't log the fancy broken snaps | 15:55 |
| zyga | so ... maybe we can merge that PR and rebuild core | 15:55 |
| zyga | for a low-tech reproducer | 15:55 |
| mvo | zyga: yeah, I think that will definitely help | 15:57 |
| mup | PR snapd#4978 closed: overlord,interfaces: be more vocal about broken snaps and read errors <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4978> | 15:57 |
| mup | PR snapd#4979 closed: overlord,interfaces: be more vocal about broken snaps and read errors (2.32) <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4979> | 15:57 |
| zyga | mvo: what do we need now? | 15:57 |
| mvo | Chipaca: 4845 is probably interessting for you | 15:57 |
| zyga | mvo: ppa build trigger + core build trigger? | 15:57 |
| pedronis | pstolowski: I don't understand that code, a change can be about many snaps | 15:57 |
| mvo | Chipaca: (mark asked about when we get version info for c-n-f :) | 15:57 |
| pstolowski | zyga: i've just got state.json from the user who reported this originally.. nothing suspicions there and since we're able to reproduce we won't learn anything new from it I guess | 15:57 |
| pedronis | pstolowski: we need to add an autoconnect for each snap, not just for the first | 15:57 |
| mvo | zyga: yeah, let me do this | 15:57 |
| mvo | zyga: vendor-sync first | 15:57 |
| zyga | pstolowski: yeah, I think it's all too easy to reproduce | 15:57 |
| zyga | mvo: thank you | 15:58 |
| pstolowski | pedronis: you're right... | 15:58 |
| zyga | mvo: I can work on this overnight but I need to break now, I'm on the phone all the time so if you have ideas I can come over | 15:58 |
| zyga | I just need to manage kids for a while | 15:58 |
| mvo | zyga: sure | 15:58 |
| mvo | zyga: thanks for all your help | 15:58 |
| mvo | zyga: I merge this now and once the new edge is ready I can re-run your script. | 15:59 |
| pedronis | pstolowski: I left a comment in the PR | 16:00 |
| zyga | mvo: please backport https://github.com/snapcore/snapd/pull/4969 | 16:00 |
| mup | PR #4969: interfaces: make system-key more robust against invalid fstab entries <Squash-merge> <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4969> | 16:00 |
| zyga | ohhh shit | 16:01 |
| zyga | forgot to squash | 16:01 |
| mup | PR snapd#4969 closed: interfaces: make system-key more robust against invalid fstab entries <Squash-merge> <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4969> | 16:01 |
| Chipaca | mvo: +1 | 16:01 |
| zyga | mvo: sorry, I didn't mean to :( | 16:01 |
| mup | PR snapd#4976 closed: cmd/snap-mgmt: remove timers, udev rules, dbus policy files (2.32) <Created by bboozzoo> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4976> | 16:02 |
| pedronis | also there's strange branch on the main repo: revert-4969-system-key-robustness | 16:02 |
| zyga | I think that was me clicking on the revert trying to undo | 16:03 |
| zyga | we can remove it | 16:03 |
| mvo | zyga: no worries, I can cherry-pick one-by-one, its not that terrible | 16:05 |
| mvo | zyga: go and take care of the kids :) | 16:05 |
| zyga | yeah, I'm going now | 16:05 |
| zyga | ttyl | 16:05 |
| mup | PR snapd#4982 opened: interfaces: make system-key more robust against invalid fstab entries (2.32) <Created by mvo5> <https://github.com/snapcore/snapd/pull/4982> | 16:07 |
| mup | PR snapd#4983 opened: osutil/sys, client: add sys.RunAsUidGid, use it for auth.json <Created by chipaca> <https://github.com/snapcore/snapd/pull/4983> | 16:11 |
| Chipaca | mvo: ^ your old friend 'drop privileges' | 16:11 |
| Chipaca | p.s. when are we moving to 1.10 :-) | 16:12 |
| zyga | Just observation from a phone: it cannot be a mount issue as I had both snaps mounted | 16:12 |
| zyga | So something else rejected the lxd snap (validation perhaps) | 16:13 |
| Chipaca | huh, launchpad giving me 'address unreachable' | 16:13 |
| Chipaca | and it's back | 16:14 |
| === phoenix_firebrd_ is now known as phoenix_firebrd | ||
| pedronis | unless we have grown something that unmounts things | 16:40 |
| pstolowski | pedronis: addressed your feedback + made a small addition to injectTasks helper, not strictly a bug, but I think it's good if extraTasks wait for main task - only affected tests at the moment that had very few tasks and no one waited for mainTask | 16:46 |
| cmars | hi, where should i open a snapd bug? | 16:51 |
| Chipaca | cmars: if it's snapd, https://bugs.launchpad.net/snapd/+filebug | 16:53 |
| Chipaca | cmars: if it might be snapd, but it might be something else, https://bugs.launchpad.net/snappy/+filebug | 16:53 |
| Chipaca | and then we assign it as appropriate | 16:53 |
| cmars | Chipaca: thanks! | 16:53 |
| Chipaca | cmars: what did you break _now_? :-p | 16:53 |
| cmars | Chipaca: asking for a friend :) | 16:54 |
| Chipaca | :-) | 16:54 |
| * kalikiana wrapping up for the day | 16:56 | |
| Chipaca | holy potato, that's a lovely stacktrace | 17:09 |
| pstolowski | zyga: i've run your bug.sh and got https://pastebin.ubuntu.com/p/SJTK4qP5Dc/ | 17:09 |
| zyga | that's another bug | 17:10 |
| zyga | I also got it | 17:10 |
| zyga | run it again | 17:10 |
| pstolowski | ah, ok | 17:10 |
| mvo | zyga, pstolowski fwiw, i triggered the new core build, we should have more debug info soon (~15-30min) | 17:13 |
| zyga | that's very good, thank you | 17:13 |
| pstolowski | ty | 17:13 |
| pedronis | pstolowski: I noticed something weird, at the beginning of doSetupProfiles we do addImplicSlot but I don't see that in doAutoconnect | 17:28 |
| pedronis | do we need that or not | 17:30 |
| === grumble is now known as Guest10721 | ||
| === rumble is now known as grumble | ||
| pedronis | pstolowski: wasn't the idea at some point that we wanted to move that somewhere else | 17:31 |
| mup | PR snapd#4982 closed: interfaces: make system-key more robust against invalid fstab entries (2.32) <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4982> | 17:35 |
| mup | PR snapd#4983 closed: osutil/sys, client: add sys.RunAsUidGid, use it for auth.json <Created by chipaca> <Closed by chipaca> <https://github.com/snapcore/snapd/pull/4983> | 17:35 |
| zyga | mvo: so, weird - new core, no error logged, still same bug | 17:35 |
| zyga | plus, for chipaca, I got this error: kwi 04 19:35:14 t470 snapd[23332]: 2018/04/04 19:35:14.803094 stateengine.go:101: state ensure error: cannot decode new commands catalog: got unexpected HTTP status code 403 via GET to "https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic" | 17:35 |
| Chipaca | zyga: that's probably fine | 17:36 |
| Chipaca | i mean, i don't know why the store is 403'ing that endpoint, but it's not fatal | 17:37 |
| pedronis | Ensure errors should block further ensure step | 17:37 |
| pedronis | should not | 17:37 |
| zyga | ah, error is when I go back to stable | 17:37 |
| zyga | so expecte d:/ | 17:37 |
| * Chipaca had a panic attack for 7 seconds there | 17:37 | |
| zyga | let's try the other way around now | 17:37 |
| pedronis | Chipaca: yes, that 403 is a bit weird | 17:38 |
| zyga | so | 17:38 |
| zyga | stable -> edge | 17:38 |
| zyga | confirmed new core is there | 17:38 |
| zyga | but no messages | 17:38 |
| zyga | WTF | 17:39 |
| zyga | maybe it isn't broken? | 17:39 |
| zyga | I'll enable debugging | 17:39 |
| pedronis | mvo: 4900 is green, also I removed the strange logging given that zyga's PR added it at a lower level | 17:42 |
| mup | Bug #1761253 opened: Installing a network-bind snap along with core fails results in incorrect permissions <Snappy:New> <https://launchpad.net/bugs/1761253> | 17:49 |
| === sarnold_ is now known as sarnold | ||
| * zyga adds debugging and repacks core | 17:52 | |
| zyga | mvo: I added some debugging and ... at the time snapd starts there's no lxd snap | 17:55 |
| zyga | I get it now | 17:55 |
| zyga | pedronis: it's funny really | 17:55 |
| zyga | pedronis: the bug is here | 17:55 |
| pedronis | it's disabled | 17:55 |
| zyga | snaps, err := snapstate.ActiveInfos(m.state) | 17:55 |
| zyga | we skip inactive snaps | 17:56 |
| pedronis | that's correct | 17:56 |
| pedronis | but when we re-activate them | 17:56 |
| pedronis | we don't do the right thing | 17:56 |
| pedronis | I suppose | 17:56 |
| pedronis | well "correct" | 17:57 |
| zyga | hmm | 17:57 |
| pedronis | disable removes profiles | 17:57 |
| pedronis | but doesn't remove from repo | 17:57 |
| zyga | log of the issue https://www.irccloud.com/pastebin/nxyEebVA/ | 17:58 |
| zyga | so after snapd starts up, ignoring lxd, it proceeds with the 2nd half of lxd setup | 17:58 |
| zyga | kwi 04 19:54:42 t470 snapd[7142]: 2018/04/04 19:54:42.808109 taskrunner.go:403: DEBUG: Running task 2249 on Do: Make snap "lxd" (6508) available to the system | 17:59 |
| zyga | I will review that code next | 17:59 |
| pedronis | we setup profiles | 17:59 |
| pedronis | before ? | 17:59 |
| pedronis | but the snap is not active | 17:59 |
| mvo | pedronis: thank you! | 17:59 |
| pedronis | it's again an issue that active is conflating too many things | 18:00 |
| pstolowski | pedronis: addimplicitslots is only relevant for repo.AddSnap call below | 18:00 |
| zyga | pedronis: where? I don't see it yet | 18:01 |
| pedronis | zyga: we setup-profiles already so the snap has profiles but is not active | 18:01 |
| mvo | zyga: nice catch! about the inactive ones | 18:01 |
| pedronis | so when we restart we don't load it | 18:01 |
| zyga | pedronis: I think in this run lxd was security setup before the restart | 18:01 |
| zyga | because I don't see any tasks for it | 18:01 |
| pedronis | in the repo | 18:01 |
| zyga | but it wasn't active yet | 18:01 |
| pedronis | so it has profiles but is not active | 18:01 |
| zyga | so it wasn't added | 18:01 |
| pedronis | but we don't add it to the repo | 18:01 |
| pedronis | when is made active | 18:01 |
| zyga | and no plugs/slots | 18:02 |
| zyga | and thus no connections | 18:02 |
| zyga | we should add inactive snaps but don't build their profiles IMO | 18:02 |
| zyga | pedronis: ^ | 18:02 |
| zyga | because if core hadn't restarted it would be still in the repo | 18:02 |
| zyga | just inactive | 18:02 |
| zyga | well, inactive in the state, in the repo there's no such notion | 18:03 |
| zyga | well, normally it's already there | 18:03 |
| zyga | we cannot blindly add it | 18:03 |
| pedronis | does remove remove the snaps from the repo? | 18:03 |
| zyga | disable removes profiles, enable adds them | 18:03 |
| zyga | but neither affects the repo | 18:03 |
| pedronis | but that is profiles | 18:03 |
| pedronis | not repo | 18:03 |
| pedronis | also I'm talking about remove | 18:03 |
| zyga | pedronis: AFAIR it does not, let me lookg | 18:03 |
| zyga | *look | 18:04 |
| mvo | pstolowski, pedronis 4981 looks good to me know, WDYT? | 18:04 |
| pedronis | zyga: I wonder why are we seeing thins only now though, we should have had this form of problem since a long while | 18:05 |
| zyga | more paralellism, restart at the "unlucky" time | 18:06 |
| pedronis | that part of parallelism was always there | 18:06 |
| pedronis | just lxd and snapd sending out a new release close to each other? | 18:06 |
| zyga | maybe | 18:06 |
| pedronis | anyway afaict we AddSnap when we setup profiles and at start | 18:08 |
| pedronis | so in theory something should be added if it has profiles | 18:08 |
| pedronis | sadly that != active | 18:08 |
| zyga | pedronis: setup profiles adds the snap to the repo | 18:09 |
| zyga | and removes any old snap from the repo | 18:09 |
| zyga | remove profiles removes the snap from the repo | 18:09 |
| pedronis | we have no memory of the setup profiles add though | 18:09 |
| pedronis | only later the get active | 18:10 |
| zyga | no becaues that's just in memory | 18:10 |
| zyga | that's not the state | 18:10 |
| pedronis | I know | 18:10 |
| pedronis | that's our problem | 18:10 |
| zyga | the pre-restart remembered it | 18:10 |
| zyga | yes | 18:10 |
| zyga | I agree, | 18:10 |
| zyga | mvo: I think we know what's the issue now but ideas on how to address it are welcome | 18:10 |
| zyga | pedronis: so what is that state that lxd is in | 18:11 |
| zyga | it's not active yet | 18:11 |
| pedronis | is not active | 18:11 |
| zyga | but we setup profiles for it | 18:11 |
| pedronis | but had profiles setup | 18:11 |
| zyga | what do we call that state | 18:11 |
| pedronis | but that's not a state tracked in state | 18:11 |
| zyga | pending? | 18:11 |
| zyga | right, I'm trying to invent a new name | 18:11 |
| pedronis | we don't track that | 18:11 |
| zyga | but all old snapds won't track that | 18:11 |
| zyga | so ... | 18:11 |
| zyga | a bit of a chicken and egg | 18:11 |
| pedronis | removeProfiles | 18:11 |
| pedronis | also remove from repo | 18:12 |
| zyga | yes | 18:12 |
| pedronis | so active is almost right except when not | 18:12 |
| zyga | setup ensures we have the latest snap in the repo and generates profiles | 18:12 |
| zyga | remove ensures we don't know about it and removes profiles | 18:12 |
| zyga | haha, yes :) | 18:12 |
| zyga | pedronis: one more super fun fact | 18:13 |
| zyga | pedronis: has-profiles flag is not per revision | 18:13 |
| pedronis | can we add the same snap again? | 18:13 |
| pedronis | what happens in that case | 18:13 |
| zyga | add no, it would clash | 18:13 |
| zyga | remove and add yes | 18:13 |
| zyga | wel | 18:13 |
| zyga | well adding a snap that's empty is a no-op | 18:13 |
| zyga | we only really track interfaces | 18:13 |
| pedronis | anyway it doesn't help | 18:14 |
| pedronis | because our problem is reloading connections | 18:14 |
| zyga | part of the logic remembers revisions (via snap info) so that's why we remove/add snap on setup-profiels | 18:14 |
| zyga | *profiles | 18:14 |
| zyga | pedronis: I think my suggestion is still correct | 18:14 |
| zyga | pedronis: load all snaps into repo | 18:14 |
| zyga | pedronis: but setup security only for those that are active | 18:14 |
| zyga | this would fix this issue | 18:14 |
| zyga | (only on snapd startup, that is) | 18:15 |
| zyga | then we have the right state to reconnect | 18:15 |
| zyga | and to setup stuff later | 18:15 |
| zyga | though... if inactive, which is "latest" | 18:15 |
| zyga | which revision to pick? | 18:15 |
| pedronis | sounds like we need ProfileRevision in SnapState | 18:16 |
| pedronis | we have Current but that a guess | 18:16 |
| pedronis | if we are in the middle of other ops | 18:16 |
| pedronis | otoh is probably what we use anyway | 18:17 |
| zyga | can we add ProfileRevision in a way that will work on refreshes from snapd that have no idea about it | 18:17 |
| pedronis | well we use it only if not Active | 18:17 |
| pedronis | by definition if Active ProfileRevison == Current | 18:17 |
| pstolowski | zyga: if you add all snaps to repo and not just active, that will make their slots/plugs available to the system, no? | 18:18 |
| zyga | pstolowski: we cannot add all, we need to add one of each | 18:18 |
| zyga | pstolowski: which revision should we add? | 18:18 |
| zyga | pstolowski: and besides, it's still a bit broke | 18:18 |
| pedronis | it will also create problem with autoconnect | 18:19 |
| pstolowski | zyga: ah, i see what you mean | 18:19 |
| pedronis | we shouldn't atuoconnect to disabled snaps | 18:19 |
| zyga | pstolowski: even if we add "all" then snap interfaces will tell you about snaps that are disabled | 18:19 |
| zyga | pstolowski: that's not supposed to happen | 18:19 |
| zyga | pstolowski: what we need to is to ensure that lxd can be installed the same way if core wasn't restarting | 18:19 |
| zyga | track something we don't | 18:19 |
| pstolowski | zyga: yes, exactly | 18:19 |
| pstolowski | (about interfaces whose snaps disabled) | 18:20 |
| zyga | kwi 04 20:21:35 t470 snapd[11725]: 2018/04/04 20:21:35.357788 snapstate.go:1696: DEBUG: skipping inactive snap lxd with snap state &{app [0xc420497400 0xc420497480 0xc420497500] false 6492 edge {false false false false false false false false false false false} map[lxc:0xc42031b380] false true 0} | 18:21 |
| zyga | some explicit debugging to confirm this | 18:22 |
| zyga | kwi 04 20:26:08 t470 snapd[16117]: 2018/04/04 20:26:08.840805 snapstate.go:1696: DEBUG: skipping inactive snap lxd with snap state &snapstate.SnapState{SnapType:"app", Sequence:[]*snap.SideInfo{(*snap.SideInfo)(0xc4200c7180), (*snap.SideInfo)(0xc4200c7200), (*snap.SideInfo)(0xc4200c7280)}, Active:false, Current:snap.Revision{N:6492}, Channel:"edge", Flags:snapstate.Flags{DevMode:false, JailMode:false, Classic:false, | 18:26 |
| zyga | TryMode:false, Revert:false, RemoveSnapPath:false, IgnoreValidation:false, Required:false, SkipConfigure:false, Unaliased:false, Amend:false}, Aliases:map[string]*snapstate.AliasTarget{"lxc":(*snapstate.AliasTarget)(0xc420322360)}, AutoAliasesDisabled:false, AliasesPending:true, UserID:0} | 18:26 |
| zyga | with field names | 18:26 |
| pedronis | zyga: I see two possible fixes, either we do something like ProfileRevision in snapstate use that and Active and Current to decide which snaps to add to the repo. or we teach doLinkSnap to AddSnap if it's not there yet but then it would need to load the missing connections as well | 18:27 |
| pedronis | and it breaks the ifacestate/snapstate separation | 18:27 |
| * zyga looks at doLinkSnap code | 18:27 | |
| pedronis | it's already big and in the wrong package | 18:28 |
| pedronis | the appeal is not needing more state | 18:28 |
| pedronis | zyga: there's no cheap way to look at disk and be sure if a snap has profiles and for which revision? | 18:29 |
| pedronis | (anyway that's not our usual approach) | 18:29 |
| zyga | pedronis: no, we'd have to be "creative" but it's terrible | 18:29 |
| jdstrand | stgraber: I wonder if your issues are at all related to https://forum.snapcraft.io/t/snapped-lxd-has-stopped-working-aa-exec-permission-denied/2356/34 | 18:29 |
| zyga | jdstrand: btw: we understand that bug very well now | 18:29 |
| zyga | it's unfortunate coincidence but easy to trigger | 18:29 |
| zyga | not a new bug in any way | 18:29 |
| jdstrand | zyga: you are saying that stgraber's issue is a different, new one? | 18:30 |
| zyga | jdstrand: we understand issues with missing interfaces that cause lxd issue with aa-exec | 18:30 |
| pedronis | jdstrand: there are probably 3 different bugs | 18:30 |
| zyga | pedronis: 3? | 18:31 |
| jdstrand | zyga: to be clear stgraber's was related to fresh install and seem to be hitting all of a sudden. mine was refresh. | 18:31 |
| zyga | jdstrand: yes | 18:31 |
| zyga | ah, that's another bug perhaps | 18:31 |
| pedronis | counting this one as well: https://forum.snapcraft.io/t/2-0-lxd-snap-fails-on-sytems-with-partial-apparmor-support/4707 | 18:31 |
| zyga | deb -> snap missing task #1st bug | 18:31 |
| pedronis | that hits fresh installs | 18:32 |
| zyga | core, lxd refresh -> #2nd bug | 18:32 |
| zyga | core, lxd refresh (hook cannot talk to snapd) -> #3rd bug | 18:32 |
| zyga | core, lxd refresh (missing interfaces on lxd) -> #2nd bug | 18:32 |
| zyga | that's what we konw | 18:32 |
| pedronis | ah, so 4 with something to do with aa-exec | 18:32 |
| zyga | ah, the partial apparmor is another one | 18:32 |
| zyga | man, fun day :) | 18:33 |
| zyga | pedronis: reading doLinkSnap now | 18:33 |
| jdstrand | zyga: this is the one I was referring to: https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850 | 18:33 |
| pedronis | we havea PR to fix #1 | 18:33 |
| zyga | jdstrand: that's #1 | 18:33 |
| jdstrand | ok | 18:33 |
| jdstrand | I don't think we need to do anything about https://forum.snapcraft.io/t/2-0-lxd-snap-fails-on-sytems-with-partial-apparmor-support/4707 (I justify why in the topic) | 18:34 |
| zyga | I kind of feel this is something we need to think with a fresh mind | 18:34 |
| jdstrand | that isn't really a snapd thing. the snap can workaround it while the kernel fix makes its way out there | 18:34 |
| pedronis | jdstrand: to be honest it seems we discovered #2 and #3 because there were snapd and lxd release close to each other | 18:34 |
| zyga | jdstrand: since you are here: where's the non-abstract x11 socket normally? | 18:35 |
| pedronis | they are both preexisting bugs | 18:35 |
| zyga | yes, it's an old bug | 18:35 |
| zyga | with very unlucky timing | 18:35 |
| jdstrand | zyga: /tmp/.X11-unix/ | 18:35 |
| zyga | jdstrand: aha | 18:35 |
| jdstrand | /tmp/.X11-unix/X0 | 18:35 |
| zyga | bummer | 18:35 |
| jdstrand | indeed | 18:35 |
| zyga | why not /run? | 18:36 |
| zyga | man that's so sad :/ | 18:36 |
| jdstrand | hysterical raisins | 18:36 |
| zyga | I wanted to refactor snap-update-ns to run pre pivot | 18:36 |
| zyga | then we could save that socket | 18:36 |
| zyga | can we bind mount the socket from hostfs? | 18:36 |
| jdstrand | sure | 18:36 |
| zyga | would that be okay-ish? | 18:36 |
| jdstrand | that is the request | 18:36 |
| zyga | so x11/desktop/whatever interfaces could add a mount profile | 18:36 |
| jdstrand | x11 and unity7, yes | 18:37 |
| jdstrand | note that x11 and unity7 are often declared together | 18:37 |
| zyga | for /run/snapd/hostfs/tmp/.X11-unix/ -> /tmp/.X11-unix/ | 18:37 |
| zyga | aha, good point, we don't handle duplicates | 18:37 |
| zyga | well | 18:37 |
| zyga | we do | 18:37 |
| zyga | but not how we want it here | 18:37 |
| zyga | jdstrand: I understand the issue now | 18:37 |
| zyga | jdstrand: how can I disable the abstract socket for testing? | 18:38 |
| zyga | jdstrand: to know I fixed it | 18:38 |
| zyga | jdstrand: it seems simple (1 day work) | 18:38 |
| jdstrand | zyga: I think you mean /var/lib/snapd/hostfs/tmp/.X11-unix/, no? | 18:38 |
| jdstrand | zyga: otoh, idk. you might ask the desktop team | 18:38 |
| zyga | ah, yes :) | 18:38 |
| zyga | sorry, just tired now | 18:38 |
| zyga | greyback: do you know how to disable the x11 abstract socket on bionic, for testing? | 18:39 |
| jdstrand | pedronis: yes, with 2 and 3 I suspected it had to do with the fact the lxd and core were updated within the same refresh cycle, which is why it has been sporadic | 18:39 |
| jdstrand | that will be a nice bug to have fixed | 18:40 |
| jdstrand | the 3 that is will be nice together :) | 18:40 |
| jdstrand | meh | 18:40 |
| jdstrand | all 3 fixed will be nice :) | 18:40 |
| zyga | mvo: around? | 18:42 |
| mvo | zyga: yes, was just reading backlog | 18:44 |
| mvo | zyga: so the bug is that snapd restarts and we loose state about lxd we only had in memory? | 18:45 |
| zyga | yes | 18:45 |
| zyga | I'm summarizing this in a new thread | 18:45 |
| mvo | zyga: if so, would it make sense to force core refreshes before everything else ? | 18:45 |
| pstolowski | mvo: ok, I got +1 from pedronis on autoconnect fix, but travis failed on device-registration test.. restarted the tests | 18:45 |
| mvo | pstolowski: thanks! | 18:46 |
| zyga | mvo: yes but it's still ugly that we don't have the full state | 18:46 |
| mvo | zyga: on disagreement, just thinking aloud what we can do short term | 18:46 |
| mvo | zyga: s/on/no/ | 18:46 |
| mvo | zyga: sorry | 18:46 |
| zyga | https://forum.snapcraft.io/t/summary-of-core-lxd-refresh-bugs-discovered-today/4869 | 18:46 |
| zyga | mvo: I think it's a viable fix | 18:47 |
| pedronis | mvo: it's a bit unclear what does that mean | 18:47 |
| zyga | pedronis: if core is refreshed: | 18:47 |
| pedronis | force core before anything else | 18:47 |
| zyga | pedronis: abort other refreshes | 18:47 |
| pedronis | that doesn't make sense | 18:47 |
| zyga | pedronis: schedule refresh of all snaps after reboot | 18:47 |
| zyga | (restart) | 18:47 |
| zyga | so "snap refresh core lxd" becomes "snap refresh core && snap refresh lxd" | 18:48 |
| zyga | (internally) | 18:48 |
| pedronis | but you can do snap refresh core | 18:48 |
| pedronis | snap refresh lxd | 18:48 |
| zyga | yes | 18:48 |
| zyga | and lxd would say "changes in proress (core)" | 18:48 |
| zyga | I'm saying for short term | 18:48 |
| zyga | .4 fix | 18:48 |
| zyga | not forever | 18:48 |
| zyga | this would fix more than one bug | 18:49 |
| pedronis | it's not a trivial fix | 18:49 |
| zyga | (it would fix all of them actually) | 18:49 |
| zyga | if it's not trivial then that's not a good solution | 18:49 |
| mvo | so just to double check, the state we are loosing is generated when the tasks for the refresh(core,lxd) is generated? and then we restart and those bits are gone? | 18:49 |
| zyga | but I think it could be easire than a proper fix for each one | 18:49 |
| mvo | it would not help if we make all other tasks wait for core refresh tasks first? | 18:49 |
| zyga | mvo: the state we lose is interface repo | 18:49 |
| zyga | mvo: if snapd hadn't restarted it would be in correct condition | 18:49 |
| zyga | mvo: not fully, (see the autoconnct task issue) but mostly | 18:50 |
| pedronis | my point is that splitting up autorefresh to behave that way | 18:50 |
| mvo | is it because lxd maybe inactive at this point? because if so if we ensure that all refresh tasks wait for core things would be ok as well, no? | 18:50 |
| pedronis | is not that simple | 18:50 |
| zyga | mvo: the low level issue is that when snapd starts and constructs the repo we don't know that we need to add some revision of lxd (inactive) to the repo | 18:50 |
| dpb1 | zyga: have we considred a rollback of the core snap? it's broken all initial installs. | 18:50 |
| mvo | zyga: so that is because the lxd refresh is half-done at this point, right? | 18:51 |
| zyga | dpb1: can you provide more data please? | 18:51 |
| zyga | mvo: yes | 18:51 |
| zyga | dpb1: to answer your question, we considered rollback of core and try not to break it | 18:51 |
| zyga | dpb1: how is it breaking initial installs? | 18:51 |
| pedronis | I think is saying because of bug #1 | 18:52 |
| mup | Bug #1: Microsoft has a majority market share <canonical> <iso-testing> <microsoft> <package-qa-testing> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <LibreOffice:New> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Confirmed for ramvi> | 18:52 |
| mup | <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:Fix Released> <The Linux OS Project:In Progress> <Neobot:New> <Novabot:New> <OpenOffice:In Progress by lh-maviya> <ReactOS Core Operating System:Incomplete> <Tabuntu:Invalid by | 18:52 |
| mup | tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:Invalid> <Ubuntu Malaysia LoCo Team:In Progress by apogee> <Wine:Confirmed> <Ubuntu:Fix Released> <Arch Linux:New> | 18:52 |
| mup | <Baltix:Confirmed> <Debian:In Progress> <Fedora:Confirmed> <Fluxbuntu:Confirmed> <openSUSE:In Progress> <Tilix:New> <https://launchpad.net/bugs/1> | 18:52 |
| pedronis | we should rollback core | 18:52 |
| pedronis | to 31.x | 18:52 |
| pedronis | because all people trying snaps right now will it | 18:52 |
| pedronis | hit it | 18:52 |
| zyga | dpb1: ah | 18:52 |
| zyga | dpb1: I understand your point now | 18:53 |
| zyga | dpb1: yes, I think that's a viable solution | 18:53 |
| zyga | mvo: ^ | 18:53 |
| zyga | jdstrand: ^ | 18:53 |
| mvo | cachio: can you test that a revert of core from 2.32.1 -> 2.31.2 works ok? and if so I think we need to move stable for core back to 2.31.2 | 18:53 |
| zyga | JamieBennett: ^ | 18:53 |
| mvo | zyga: ack, I think we need testing but once that is confirmed +1 | 18:53 |
| zyga | mvo: yes | 18:53 |
| zyga | that's a very good outcome for 20:53 | 18:53 |
| mvo | zyga: also we need to ensure the store is prepared for this (same heads up as with a regular new stable) | 18:53 |
| zyga | and we need .3 with a few more fixes | 18:53 |
| mvo | zyga: indeed | 18:54 |
| zyga | or even 2.31.x with some helper fix | 18:54 |
| zyga | if needed | 18:54 |
| zyga | and we could do a post mortem to learn from this | 18:54 |
| JamieBennett | zyga: mvo: as long as we are sure the rollback is safe and will indeed fix the issue, I'm +1 although I believe we have a fix for #1 very close? | 18:55 |
| cwayne | mvo: zyga: reverting core? Did we miss something? | 18:55 |
| zyga | JamieBennett: FYI: issues numbered: https://forum.snapcraft.io/t/summary-of-core-lxd-refresh-bugs-discovered-today/4869 | 18:55 |
| zyga | cwayne: ^ | 18:55 |
| JamieBennett | zyga: yes, saw that, thanks | 18:55 |
| mvo | JamieBennett: two out of three are almost done | 18:55 |
| jdstrand | zyga: you asked my opinion. I'm not sure if it was meant for me. if it was, what do you want my opinion on? | 18:55 |
| JamieBennett | mvo: so is it safer to rollback or push the fix? | 18:55 |
| zyga | JamieBennett: we don't have a fix yet | 18:56 |
| mvo | JamieBennett: I would (naturally) prefer the fix, but we have one open issue left and we don't know how widespread that is | 18:56 |
| zyga | jdstrand: just FYI | 18:56 |
| pedronis | mvo: zyga: the issue of waiting for core, is also that is can happen in the other direction to, snap refresh lxd, snap refresh core, it's unlikely but we would need to wait to deactivate core until there is no interesting pending task | 18:56 |
| * jdstrand nods | 18:56 | |
| zyga | pedronis: yes, that's a good point | 18:57 |
| zyga | pedronis: I would prefer a fix that tracks state correctly | 18:57 |
| zyga | it feels conceptually easier to explain | 18:57 |
| zyga | than the waiting workaround | 18:57 |
| pedronis | basically it's doable but it escalates quickly a bit into a tangle of cross checks | 18:57 |
| zyga | and also more correct technially | 18:57 |
| zyga | yes | 18:58 |
| zyga | I will look at tracking this in the state, I can iterate locally without pushing anything | 18:58 |
| zyga | but I probably will bail soon, I need to think and rest | 18:58 |
| mvo | zyga, pedronis can we have a quick HO to sync up on possible fixes? or is it too late? | 18:58 |
| zyga | (no need to rebuild core to test) | 18:58 |
| zyga | mvo: sure | 18:59 |
| zyga | in 2 min in standup | 18:59 |
| dpb1 | stgraber: ^ | 18:59 |
| dpb1 | stgraber: would be good to test out side on that too | 18:59 |
| pedronis | mvo: yes | 19:00 |
| dpb1 | stgraber: *our side | 19:03 |
| stgraber | dpb1: not sure how to test that, there's no flag to "snap install lxd" that I can pass to tell it to install a particular version of the core snap | 19:05 |
| zyga | wtf https://www.irccloud.com/pastebin/BIrJMfkH/ | 19:06 |
| stgraber | zyga: ah yeah, we've seen that occasionally before, it's a weird one | 19:08 |
| zyga | stgraber: we understand it now | 19:09 |
| zyga | cachio: ping | 19:10 |
| stgraber | zyga: oh, that explenation makes sense, I always thought it was our weird mntns setup that was causing that, but the socket not being reachable because of the restart makes sense | 19:11 |
| ackk | stgraber, you can snap install core manually though | 19:12 |
| jdstrand | stgraber: you may have noticed me talking about lxd yesterday. that is one of the 3 issues being discussed above | 19:12 |
| stgraber | ackk: not for this bug, no | 19:13 |
| stgraber | ackk: this bug specifically happens when the core snap is auto-installed when you install your first snap | 19:13 |
| stgraber | ackk: installing core seperately avoids the issue :) | 19:13 |
| ackk | oh :) | 19:13 |
| stgraber | jdstrand: ah, glad that the different issues have been tracked down and untangled now :) | 19:14 |
| jdstrand | stgraber: yes. 'lucky' you it is all happening with the lxd snap. the fixes will make things more robust for everyone :) | 19:15 |
| stgraber | yeah, "lucky"... timing is kinda crap though as we're doing the social media round for LXD 3.0 tomorrow :) | 19:16 |
| stgraber | for now I updated our install instructions in that post to "snap install core && snap install lxd" with a link to the snapd issue on the forum, that should save us from most bug reports :) | 19:17 |
| jdstrand | nice | 19:17 |
| cachio | zyga, hey | 19:21 |
| pstolowski | mvo: #4981 merged | 19:22 |
| mup | PR #4981: ifacestate: inject autoconnect <Critical> <Squash-merge> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/4981> | 19:22 |
| mup | PR snapd#4981 closed: ifacestate: inject autoconnect <Critical> <Squash-merge> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/4981> | 19:22 |
| === pstolowski is now known as pstolowski|afk | ||
| mup | PR snapcraft#2046 closed: lxd: specify arch in lxc image list command <Created by kalikiana> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/2046> | 19:23 |
| mvo | pstolowski|afk: thanks! | 19:27 |
| mup | PR snapd#4984 opened: ifacestate: inject autoconnect (#4981) <Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/4984> | 19:28 |
| zyga | JamieBennett: I updated the thread with more details and links to PR that fix the issue | 19:31 |
| zyga | cachio: did you see the question from mvo | 19:31 |
| zyga | cachio: we need to test a rollback of core to 2.31.x | 19:31 |
| zyga | JamieBennett: I don't think we have a rollback decision yet but it is a tempting choice, mvo will discuss with you when he's back | 19:31 |
| cachio | zyga, sure, I'll do it now | 19:32 |
| zyga | JamieBennett: and regardless of the rollback we will do 2.32.4 with fixes for all three issues 2 and 3 | 19:32 |
| zyga | and maybe for issue 1 | 19:32 |
| cachio | zyga, any version in specific? | 19:32 |
| zyga | cachio: latest 2.31.x so I think 2.31.2 | 19:32 |
| JamieBennett | As above, I’m +1 if we are confident that it will not cause issues | 19:33 |
| zyga | JamieBennett: we discussed how to solve issue 2 just now and I'm working on it | 19:33 |
| zyga | JamieBennett: right, mvo said we would only rollback afte cachio gives +1 from test POV | 19:33 |
| mvo | cachio: 4205-4210 is the previous 2.31.2 | 19:33 |
| * JamieBennett nods | 19:33 | |
| zyga | JamieBennett: issue .1 is still a bit more complex but we have some ideas | 19:33 |
| zyga | hey mvo | 19:34 |
| mvo | zyga: hey | 19:34 |
| cachio | mvo, from master, right? | 19:34 |
| mvo | cachio: from current stable 4325..30 to 4205..10 | 19:35 |
| mvo | cachio: i.e. 2.32.1 back to 2.31.2 | 19:35 |
| cachio | mvo, ok | 19:35 |
| mvo | zyga: the fix for "when old snapd starts the process (think: deb) and new one finishes the auto-connect task is missing" has landed in master now, I created a backport and create a new edge core with the fix so that this can be tested for real | 19:36 |
| zyga | mvo: thank you | 19:36 |
| pedronis | mvo: I think we understood the snapctl socket issue | 19:38 |
| pedronis | it's related to the shudown logic on restart | 19:39 |
| pedronis | I don't know if there are other failure modes for running hooks while snapd is inactive though | 19:40 |
| pedronis | zyga: mvo: afaict snap-confine dies if base current is not set? | 19:43 |
| zyga | pedronis: yes but it only does so if it needs to construct a new namespace | 19:43 |
| zyga | pedronis: if the snap is around (like lxd would) it has a namespace to enter | 19:43 |
| pedronis | ok, so there's an issue, but again infrequent (possibly) | 19:44 |
| zyga | looking at the trace I attached it has one | 19:44 |
| zyga | DEBUG: sending information about the state of the mount namespace (keep) | 19:44 |
| zyga | this is critical | 19:44 |
| zyga | it says that the namespace is in sync with what base snap we expect to use | 19:44 |
| zyga | so we reuse it | 19:45 |
| pedronis | well that code does: | 19:46 |
| pedronis | // Read the revision of the base snap by looking at the current symlink. | 19:47 |
| pedronis | sc_must_snprintf(fname, sizeof fname, "%s/%s/current", | 19:47 |
| pedronis | SNAP_MOUNT_DIR, base_snap_name); | 19:47 |
| pedronis | if (readlink(fname, base_snap_rev, sizeof base_snap_rev) < 0) { | 19:47 |
| pedronis | so I suppose it would die if current is not htere | 19:47 |
| zyga | yes, that's true | 19:47 |
| zyga | if current is not there we die earlier | 19:48 |
| zyga | because /dev and what not won't be there | 19:48 |
| zyga | just unmount current and see | 19:48 |
| pedronis | so we have a general problem about upgrading bases and their snaps at the same time | 19:48 |
| pedronis | so this one would need conflicts and wait to be solved | 19:55 |
| cachio | mvo, I ran it manually nad I did not see any issue, should I test it in any specific arch or system | 19:55 |
| cachio | ? | 19:55 |
| pedronis | mvo: zyga: I added a not about base current vs snap ops in that forum topic, it probably merits its own at some point but at least it will not get lost | 20:00 |
| zyga | thanks | 20:00 |
| cachio | mvo, https://paste.ubuntu.com/p/5DWzM6Dytf/ | 20:02 |
| cachio | 2.32.1 -> 2.32.2 -> 2.32.1 worked fine here | 20:03 |
| mvo | cachio: great, do you think you could verify on a core device as well? might be tricky to get 2.31.2 there but you can publish the core snap so "snap refresh --revision=4205 core" should work | 20:04 |
| cachio | mvo, sure, let me try it | 20:05 |
| mvo | cachio: --revision=4206 if you are on amd64 - 4209 for pi2 :) | 20:05 |
| cachio | ok, let me flash the image and I0'll try it | 20:06 |
| zyga | pedronis: still there? | 20:08 |
| zyga | I wonder if it is right still (security revision) | 20:08 |
| zyga | writing the commit message for a WIP patch | 20:09 |
| zyga | when we refresh r1 is setup | 20:09 |
| zyga | but we want r2 | 20:09 |
| zyga | and r2 is not active yet | 20:09 |
| zyga | we don't remove security of r1, just setup for r2 | 20:10 |
| zyga | so we want to store r2 | 20:10 |
| zyga | and reload r2 | 20:10 |
| * zyga actually thinks it is fine again | 20:10 | |
| zyga | just tired | 20:10 |
| zyga | thank you f:) | 20:10 |
| zyga | for listening | 20:10 |
| pedronis | zyga: we need to store the revision of the thing we AddSnap | 20:12 |
| zyga | pedronis: quick WIP patch https://github.com/zyga/snapd/commit/a98a927d1ae472f849ed47f6ad396cdf7d98c636 | 20:13 |
| zyga | untested | 20:13 |
| zyga | I didn't move any code around to minimize the diff as it will have to use some private functions that need to become public first | 20:14 |
| pedronis | zyga: we have an issue | 20:15 |
| pedronis | zyga: when we install I don't think SnapState exists in snaps until we do link | 20:17 |
| zyga | hmm hmm | 20:17 |
| pedronis | and storing it without all the info breaks invariants | 20:18 |
| pedronis | zyga: I fear, we need a separate security-profiles-revision map | 20:19 |
| zyga | hmm hmm | 20:20 |
| zyga | we setup profiles before we link | 20:20 |
| pedronis | yea, that's the issue | 20:20 |
| zyga | we have Candidate | 20:20 |
| zyga | can we use that? | 20:21 |
| pedronis | ? | 20:21 |
| pedronis | that is on the task | 20:21 |
| pedronis | no | 20:21 |
| zyga | ah | 20:21 |
| zyga | hmm | 20:22 |
| zyga | but don't we have a deeper issue the | 20:22 |
| zyga | setupAffectedSnaps has code like | 20:22 |
| zyga | snapstet.Get9st, affectedSnapName, ...) | 20:22 |
| pedronis | we don't connect to things that are not installed | 20:22 |
| zyga | if I install a pair of snaps | 20:22 |
| zyga | would they auto-connect to each other? | 20:23 |
| zyga | or only after installation? | 20:23 |
| zyga | *after linking | 20:23 |
| pedronis | only after installation | 20:23 |
| zyga | ok | 20:23 |
| zyga | yeah, I see your point | 20:23 |
| pedronis | unless they are default-providers | 20:23 |
| zyga | oh | 20:23 |
| pedronis | in which case we wait for some things | 20:23 |
| zyga | we install the provider first? | 20:23 |
| zyga | (fully) | 20:23 |
| pedronis | not fully | 20:23 |
| pedronis | but we don't do autoconnect | 20:24 |
| zyga | but link it? | 20:24 |
| zyga | aha | 20:24 |
| zyga | well | 20:24 |
| zyga | I see the problem | 20:24 |
| mup | Issue snapcraft#1672 closed: Add pre-pull/pull/post-pull <Created by sergiusens> <Closed by kyrofa> <https://github.com/snapcore/snapcraft/issue/1672> | 20:24 |
| mup | PR snapcraft#2045 closed: many: add override-pull scriptlet <Created by kyrofa> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/2045> | 20:24 |
| zyga | thank you for the quick insight | 20:24 |
| pedronis | until they have done setup-prfoiles | 20:24 |
| pedronis | anyway, as I said, I think we need a separate map | 20:24 |
| pedronis | it's really an issue of snapstate has some state, and ifacestate needs some different state | 20:25 |
| zyga | note that the error from that get is non-fatal | 20:25 |
| zyga | as we had "mysterious issues' | 20:25 |
| zyga | so maybe another layer of mud needs digging | 20:25 |
| pedronis | but I see we find autoconnect candidates based on the repo | 20:28 |
| pedronis | so there might be stuff there | 20:28 |
| pedronis | that doesn't exist in snaps yet | 20:28 |
| zyga | but then we go and setup security and that may bite us | 20:30 |
| zyga | since we get the snaps from the state | 20:30 |
| zyga | because we had that bug and gustavo wanted us to always look at the state | 20:30 |
| zyga | maybe that was not right | 20:30 |
| zyga | and maybe refreshing the repo explicitly like we do is enough | 20:30 |
| zyga | but all questions need thinking now | 20:30 |
| pedronis | well there's a disagreement between snapstate and ifacestate/repo | 20:31 |
| pedronis | about wha's active | 20:31 |
| zyga | active is misleading | 20:31 |
| zyga | but yes | 20:31 |
| zyga | there's disagreement | 20:31 |
| pedronis | some of it is just the nature | 20:32 |
| pedronis | of splitting tasks between the two | 20:32 |
| pedronis | some of it could be reconciled | 20:32 |
| pedronis | zyga: anyway that Get skips if there's an error | 20:35 |
| zyga | yes | 20:35 |
| pedronis | so whatever the issue is not explosive | 20:35 |
| zyga | but not sure if it should, I don't know when that may legally happen | 20:35 |
| pedronis | and has been there since long | 20:35 |
| zyga | yes, we made it non-explosive | 20:35 |
| pedronis | zyga: I don't think we can tackle that one now | 20:35 |
| zyga | testing my WIP patch | 20:37 |
| zyga | I security-revision is stored, now running the script | 20:37 |
| zyga | it didn't work | 20:38 |
| zyga | somewhere when we remove profiles | 20:38 |
| zyga | we reset the revision | 20:38 |
| zyga | and then restart must happen after that time | 20:38 |
| zyga | kwi 04 22:38:00 t470 snapd[3451]: 2018/04/04 22:38:00.605460 snapstate.go:1714: DEBUG: skipping inactive snap lxd with snap state &snapstate.SnapState{SnapType:"app", Sequence:[]*snap.SideInfo{(*snap.SideInfo)(0xc4200b5100), (*snap.SideInfo)(0xc4200b5180), (*snap.SideInfo)(0xc4200b5200)}, Active:false, Current:snap.Revision{N:6492}, SecurityProfilesRevision:snap.Revision{N:0}, Channel:"edge", Flags:snapstate.Flags{DevMode:false, | 20:39 |
| zyga | JailMode:false, Classic:false, TryMode:false, Revert:false, RemoveSnapPath:false, IgnoreValidation:false, Required:false, SkipConfigure:false, Unaliased:false, Amend:false}, Aliases:map[string]*snapstate.AliasTarget{"lxc":(*snapstate.AliasTarget)(0xc4204f5120)}, AutoAliasesDisabled:false, AliasesPending:true, UserID:0} | 20:39 |
| zyga | SecurityProfilesRevision is zero | 20:39 |
| pedronis | but then we will do setup-profiles | 20:39 |
| pedronis | is setup-profiles not doing enough? | 20:40 |
| zyga | not sure I understand | 20:41 |
| pedronis | the theory was that we did setup-profiles but not link-snap | 20:41 |
| pedronis | but here we didn't do setup-profiles | 20:41 |
| pedronis | otherwise profile revision would be set | 20:41 |
| zyga | we did, I refreshed lxd manually a few times and used jq to read the profile revision from the state | 20:42 |
| zyga | I probably don't understand the refresh logic well enough | 20:42 |
| zyga | and when we unlink the old snap we remove its profiles | 20:42 |
| zyga | and that resets the revision | 20:42 |
| zyga | and then if we restart we are in a bad state | 20:42 |
| zyga | (maybe) | 20:43 |
| zyga | that's just me guessing now | 20:43 |
| pedronis | but that's always been like that | 20:43 |
| pedronis | also afair we don't do that | 20:43 |
| zyga | this is interesting | 20:43 |
| zyga | https://www.irccloud.com/pastebin/t1zwTx1t/ | 20:43 |
| zyga | we setup profiles for lxd | 20:43 |
| zyga | (probably don't finish) | 20:44 |
| zyga | then core restarts | 20:44 |
| pedronis | only Remove and Disable remove-profiles | 20:44 |
| zyga | so we never set it | 20:44 |
| zyga | ah, maybe my patch is wrong then | 20:44 |
| zyga | I patched it so that doRemoveProfiles resets the revision to zero | 20:44 |
| zyga | maybe it's called in other places | 20:44 |
| * zyga looks | 20:44 | |
| zyga | actually, removeSnapSecurity | 20:45 |
| zyga | which is only called from removeProfilesForSnap | 20:45 |
| pedronis | you should do things close to where | 20:45 |
| pedronis | we do AddSnap | 20:45 |
| pedronis | and RemoveSnap from repo | 20:46 |
| zyga | yeah | 20:46 |
| zyga | tweaking | 20:48 |
| pedronis | not that I see other places calling removeSnapSecurity | 20:49 |
| zyga | quick diff | 20:50 |
| zyga | https://github.com/snapcore/snapd/compare/master...zyga:fix/reload-repo-inactive-snaps?expand=1 | 20:50 |
| zyga | ah | 20:50 |
| zyga | wrong, didn't push | 20:50 |
| zyga | now it's correct | 20:51 |
| zyga | well, up-to-date | 20:51 |
| cachio | mvo, same in pi2 | 20:54 |
| cachio | it works fine | 20:54 |
| Caelum | zyga: fantastic, so now we can say we have a fully working snapd for opensuse :D | 20:56 |
| pedronis | zyga: it looks reasonable, are we missing something obvious? | 20:57 |
| zyga | Caelum: we need to add snapd to factory and package gnome-software extensions | 20:57 |
| zyga | pedronis: testing it now | 20:57 |
| pedronis | (apart that we cannot use SnapState but that's for install) | 20:57 |
| zyga | yes | 20:57 |
| zyga | fingers crossed | 20:57 |
| Caelum | zyga: of course there is always more stuff to do | 20:57 |
| zyga | hmm hmm hmm | 20:58 |
| zyga | pedronis: it still didn't store the rev | 20:58 |
| zyga | on refresh it gets reset to zero | 20:58 |
| Caelum | zyga: the web page didn't get rebuilt yet btw, but I guess that happens in some cron job | 20:58 |
| zyga | maybe the Get's I added just fail | 20:58 |
| zyga | Caelum: if it's not refreshed tomorrow I will poke people for a rebuild | 20:59 |
| Caelum | great | 20:59 |
| Caelum | should I start playing with 42.1 in my branch to get ready for 42.2 ? | 20:59 |
| pedronis | zyga: you should probably log from the places you are setting/resetting it | 20:59 |
| zyga | yes, patching now | 20:59 |
| Caelum | I mean 32.1 | 21:00 |
| Caelum | and you already released 32.2, that's the one you wanted to update to? | 21:00 |
| zyga | Caelum: 32 is a troubled child :) | 21:01 |
| cachio | mvo https://paste.ubuntu.com/p/fqBtRBsX33/ | 21:01 |
| zyga | Caelum: we have some issues to work through | 21:01 |
| zyga | I think 2.32.4 will be good | 21:01 |
| zyga | pedronis: nope | 21:07 |
| zyga | whatever is happening I'm losing the securityt profiles revision | 21:07 |
| zyga | I added logging | 21:08 |
| zyga | is the state overwritten ever without loading? | 21:08 |
| pedronis | we are setting it? | 21:08 |
| zyga | ye | 21:08 |
| zyga | yes, it's set | 21:08 |
| pedronis | and not resetting it? | 21:08 |
| zyga | also seen in jq | 21:08 |
| zyga | we are not unsetting it | 21:09 |
| zyga | unless I made a silly bug in the code | 21:09 |
| mvo | cachio: looks like no issues there | 21:09 |
| zyga | I get the state from the task | 21:09 |
| zyga | then get, update and set the snap state | 21:09 |
| pedronis | seen in jq? | 21:09 |
| mup | PR snapd#4984 closed: ifacestate: inject autoconnect (#4981) <Critical> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4984> | 21:10 |
| pedronis | seen in jq set? | 21:10 |
| zyga | yes, I see both the revision (before running bug.sh) and unset in jq | 21:10 |
| zyga | yes | 21:10 |
| zyga | I snap refresh --edge lxd; then same to --stalbe | 21:10 |
| zyga | stable | 21:10 |
| zyga | and that sets it | 21:10 |
| zyga | (using my hacked edge core) | 21:10 |
| zyga | then refresh core, lxd to stable and back to edge | 21:10 |
| zyga | and it is gone | 21:10 |
| pedronis | well, stable will write is stuff | 21:10 |
| pedronis | and lose that no? | 21:10 |
| zyga | overwrite ! | 21:10 |
| zyga | oh | 21:10 |
| zyga | drat | 21:10 |
| zyga | so yeah | 21:10 |
| zyga | we need a new map | 21:11 |
| pedronis | ? | 21:11 |
| zyga | but this is also scary | 21:11 |
| pedronis | we cannot fix the bug in stable | 21:11 |
| zyga | yes yes, my point is that old snapd overwrites and discards this data | 21:11 |
| zyga | and it's not great | 21:11 |
| pedronis | we need a map | 21:11 |
| pedronis | anyway | 21:11 |
| zyga | it's one to have a field and not use it | 21:11 |
| pedronis | but in general we cannot fix the bug for stable | 21:11 |
| zyga | and another to overwrite and remove a field | 21:11 |
| zyga | sure, I just didn't expect stable to erase this | 21:12 |
| zyga | but yeah | 21:12 |
| zyga | ok | 21:12 |
| zyga | I think I need to EOD now | 21:12 |
| pedronis | stable will still not do the right thing | 21:12 |
| zyga | too tired and too much hectic stuff at home | 21:12 |
| zyga | I will use a new map tomorrow and test this | 21:12 |
| zyga | if this is fine I will write proper tests for everything | 21:12 |
| zyga | how shall we call it? | 21:12 |
| zyga | repostate? | 21:12 |
| pedronis | the question whether, we will need more bits? | 21:13 |
| pedronis | I mean should be start from day 0 | 21:13 |
| zyga | 8ball | 21:13 |
| pedronis | with name -> struct | 21:13 |
| pedronis | or only name -> revision | 21:13 |
| zyga | I'd go struct | 21:13 |
| zyga | but with the problem of destructive updates | 21:13 |
| zyga | it's probably irrelevant | 21:13 |
| zyga | but I'd go with struct anyway | 21:14 |
| zyga | easier to code and nicer | 21:14 |
| zyga | repostate[snapName].Revision | 21:14 |
| pedronis | security-profiles-state ? | 21:14 |
| zyga | ifacestate? to tie this into the interface manager (iface-state) | 21:15 |
| zyga | not sure | 21:15 |
| zyga | we can rename it tomorrow :) | 21:15 |
| zyga | mvo: I'm EODing | 21:16 |
| zyga | let's regroup tomorrow | 21:16 |
| zyga | thank you for the insight tonight pedronis, it was invaluable | 21:17 |
| mvo | zyga: yeah | 21:17 |
| mvo | zyga: I think we need the revert | 21:17 |
| zyga | yes | 21:17 |
| zyga | let's revert, my +1 goes there | 21:17 |
| mvo | cachio: can you please run some more tests, ideally a "snap refresh --revision=4206 core" and some spread testing with such a setup? I think we need to revert tomorrow | 21:18 |
| zyga | not sure if you want to do it tonight | 21:18 |
| zyga | if we need to ack from store | 21:18 |
| mvo | zyga, pstolowski|afk unless I did something wrong in my tests the inject-task is not fully working: https://paste.ubuntu.com/p/BxmxD3VFCD/ | 21:18 |
| mvo | zyga, pstolowski|afk "2018-04-04T23:11:59+02:00 INFO cannot auto-connect plug lxd:lxd-support to core:lxd-support: no state entry for key" | 21:19 |
| mvo | zyga, pstolowski|afk lets talk more tomorrow | 21:20 |
| zyga | no state entry for key :/ | 21:20 |
| zyga | ErrNoState | 21:20 |
| zyga | sucks to be us today | 21:20 |
| zyga | ttyl | 21:20 |
| pedronis | mvo: I think it's probably inserted in the wrong place :/ | 21:21 |
| pedronis | mvo: pstolowski|afk: yes, for a snap that is not core, it needs to come after link-snap | 21:23 |
| pedronis | not after setup-profiles | 21:23 |
| pedronis | see doInstall code | 21:23 |
| pstolowski|afk | oh my | 21:26 |
| pstolowski|afk | mvo thanks for the test... it's a shame a spread test wasnt possible | 21:30 |
| geekgonecrazy | Hey guys.. does anyone know if the nodejs part has been updated recently? | 21:53 |
| geekgonecrazy | I've used the nodejs part for a while, and suddenly its started giving me this error: | 21:54 |
| geekgonecrazy | Failed to run 'npm ls --global --json' for 'node': Exited with code 1. Verify that the part is using the correct parameters and try again. | 21:54 |
| geekgonecrazy | even for builds that previously succeeded on launchpad | 21:54 |
| === Guest20669 is now known as devil_ | ||
| kyrofa | geekgonecrazy, do you specify a node version? | 22:18 |
| geekgonecrazy | kyrofa: yup, otherwise mass chaos in Rocket.Chat :D | 22:20 |
| geekgonecrazy | unfortunately very tied to node.js versions at times | 22:20 |
| geekgonecrazy | trying to locate the `npm ls --global --json` because I can't recall that ever being run when I do `npm install` locally | 22:21 |
| geekgonecrazy | So my assumption was it was in the nodejs part somewhere maybe? | 22:22 |
| kyrofa | geekgonecrazy, when was your most recent successful build? | 22:24 |
| geekgonecrazy | last night. Looking at the wiki page for the parts best I can tell last time it was updated was the 1st | 22:25 |
| vidal72[m] | zyga: to disable abstract socket add "-nolisten local" to X server arguments. Most display managers already add "-nolisten tcp" | 22:25 |
| zyga | thanks | 22:25 |
| kyrofa | Huh, so it doesn't involve the snapcraft update from last week | 22:25 |
| kyrofa | I'm not familiar with the nodejs part | 22:26 |
| kyrofa | That could be it | 22:26 |
| kyrofa | Maybe reach out to the maintainer? | 22:26 |
| diddledan | well this is intriguing. I appear to have managed to get snapcraft into an infinite loop somewhere at "preparing to build desktop-gtk2" - it's sat there using 100% of a cpu core | 22:27 |
| geekgonecrazy | kyrofa: any way via snapcraft command to see maintainer? I might just manually install node.js as a less ideal workaround. | 22:27 |
| kyrofa | geekgonecrazy, wait... I don't see a nodejs part :P | 22:28 |
| kyrofa | diddledan, go away, my day was good | 22:28 |
| diddledan | :-p | 22:28 |
| * diddledan cuddles kyrofa | 22:28 | |
| geekgonecrazy | kyrofa: I didn't see one via that page, but i'm some how using one :D | 22:28 |
| kyrofa | geekgonecrazy, let me take a look at your yaml | 22:29 |
| geekgonecrazy | kyrofa: don't judge too much :D https://github.com/RocketChat/Rocket.Chat/blob/develop/.snapcraft/snapcraft.yaml | 22:30 |
| geekgonecrazy | oh duh.. plugin not part | 22:31 |
| diddledan | oh ello. it finished | 22:33 |
| diddledan | wasn't an infinite loop then, just a veeeery slow install | 22:33 |
| diddledan | maybe it's my ISP - they've been battling a DDoS the past few days | 22:34 |
| popey | ah yeah, desktop helpers can be a bit slow | 22:34 |
| diddledan | this is on the building of the snap, rather than initial launch, popey , so I'm assuming my net connection was taking forever for snapcraft to fetch the needed bits to work out how to build | 22:35 |
| popey | yes | 22:35 |
| popey | building can be slow too | 22:35 |
| diddledan | aah | 22:35 |
| diddledan | ok | 22:35 |
| * zyga will start later tomorrow | 22:35 | |
| zyga | I cannot even sleep now with all the snapd state in my head | 22:36 |
| diddledan | you should share states with someone else ;-p | 22:39 |
| diddledan | shared state is so much more reliable, right? :-D | 22:39 |
| geekgonecrazy | kyrofa: btw its non-working with node-engine: 8.9.4 (which used to work) It might be something on our side.. Maybe I need to force a newer version of npm to be installed with that older version of node | 22:39 |
| Chipaca | zyga: hey | 22:40 |
| === mwhudson_ is now known as mwhudson | ||
| geekgonecrazy | kyrofa: looks like its: https://github.com/snapcore/snapcraft/blob/master/snapcraft/plugins/nodejs.py#L250 that for some reason with node.js 8.9.4 isn't working. Not sure why now. Is there any way to use the nodejs plugin and keep it from doing the npm install? :D | 22:47 |
| diddledan | omg!!!!!!! irccloud 0.6.0 lets you connect to slack channels!!! p.s. irccloud 0.6.0 works in a snap | 22:48 |
| * diddledan pokenprod popey | 22:49 | |
| diddledan | PR incoming! | 22:49 |
| nacc | diddledan: iirc, slack is discontinuing the irc gateway | 22:49 |
| nacc | pretty soon, at that | 22:49 |
| popey | diddledan: srsly? | 22:49 |
| popey | no more chmod nonsense? | 22:49 |
| diddledan | nope, the new core fixes it by making it an EPERM rather than SIGKILL | 22:50 |
| popey | Oh yes, time for a daily hug for jdstrand | 22:50 |
| jdstrand | \o/ | 22:50 |
| * jdstrand hugs popey | 22:50 | |
| diddledan | nacc: it works with a connector installed in slack. i.e. it does not use the IRC gateway | 22:51 |
| popey | wow funky | 22:51 |
| nacc | diddledan: oh interesting! | 22:51 |
| diddledan | you need an admin of each slack workspace to enable the plugin tho | 22:52 |
| nacc | diddledan: ah | 22:52 |
| popey | looking forward to the pr :) | 22:52 |
| kyrofa | geekgonecrazy, sorry, phone call | 22:54 |
| kyrofa | geekgonecrazy, yeah, that hasn't changed for quite some time | 22:54 |
| diddledan | well fooey.. my irccloud desktop repo isn't linked | 23:04 |
| diddledan | popey: https://github.com/snapcrafters/irccloud-desktop/pull/6 | 23:06 |
| mup | PR snapcrafters/irccloud-desktop#6: update to irccloud 0.6.0 <Created by diddledan> <https://github.com/snapcrafters/irccloud-desktop/pull/6> | 23:06 |
| diddledan | aah. the icon doesn't appear | 23:09 |
| mup | PR snapcraft#2049 opened: many: add override-stage scriptlet <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2049> | 23:43 |
| geekgonecrazy | kyrofa: thanks for taking a look. Went ahead and just did the whole manual node.js install. For what ever reason that step where it does npm ls --global --json in combination with one of our npm dependencies... just blows up | 23:56 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
