[06:08] good morning [06:37] battery running low, brb [06:39] re [06:42] mvo: quiet day, eh? [06:45] zyga: good morning [06:45] zyga: yes, pretty quiet [06:46] zyga: did you enjoy your long weekend? [06:46] yes, though lucy had fever yesterday and we were worried [06:47] it's all meant to be quite different, my wife returned to work today and (before I got sick) I was supposed to take two weeks off [06:47] but reality struck and here we are :) [06:48] zyga: yeah, reality is annoying sometimes - but you can't argue with it - it's always right :/ [06:48] :D [06:48] yep [07:01] morning [07:04] good morning pstolowski [07:04] pstolowski: hope you had a nice long weekend [07:10] mvo: hey! yes, was very nice [07:25] * zyga had breakfast [07:25] Lucy still sleeping :) [07:25] I'm working on the udev integration branch [07:26] it's open for so long and should not take long to finish [07:27] zyga: nice! [07:27] hmm, some store EOFs [07:28] - Download snap "core18" (1885) from channel "stable" (unexpected EOF) [07:29] is unexpected EOF something we retry on? [07:29] pstolowski: unless you disagree I will (squash) merge 9152 [07:29] zyga: probably, might be worthwhile to check the logs [07:29] mvo: here it was during "snap install shellcheck" before we ran spread [07:29] so no logs [07:30] mvo: fantastic! yes, go for it, thanks [07:34] PR snapd#9152 closed: cmd/snapd-generator: generate drop-in to use fuse in container

[07:49] store is really unhappy today [07:49] PR snapd#9102 closed: corecfg: add "system.timezone" setting to the system settings

[08:25] jamesh: hi! minor comment to your snapctl is-connected PR [08:25] pstolowski: looking [08:31] pstolowski: would it actually be possible to call snapctl in a context where implicit plugs and slots existed? [08:34] jamesh: we have a 'hack' for implicit slots (AddImplicitSlots helper) that we explicitly call over SnapInfo of system snap in various places. if core had interface hooks for slots than I image it would be needed. but i'd leave it as is for now because it's not going to be needed any time soon [08:35] s/image/imagine/ [08:36] I suppose core has a configure hook where it could make a difference if it actually checked [08:40] jamesh: it has but the hook is "hijacked" and handled internally [08:43] * zyga moves to the office, lucy is now awake and handled by grandparents [08:53] mvo: there is one question from Samuele pending your input on https://github.com/snapcore/snapd/pull/9084, can you take a look? [08:53] PR #9084: o/snapstate: check disk space before creating automatic snapshot on remove (3/N) [08:54] pstolowski: sure, looking [08:58] pstolowski: replied [08:58] ty [09:47] mvo: uh [09:47] we have a very silly bug in master [09:48] mvo: patch coming right up [09:48] zyga: oh, ok [09:50] PR snapd#9171 opened: [RFC] "virtual" configuration for timezone <⛔ Blocked> [09:55] PR snapd#9172 opened: tests: update spread test for unknown plug/slot with snapctl is-connected [09:58] jamesh: i've updated snapctl spread test to test the new error message ^ [10:03] mvo https://github.com/snapcore/snapd/pull/9173 [10:03] PR #9173: cmd: compile snap gdbserver shim correctly [10:05] PR snapd#9173 opened: cmd: compile snap gdbserver shim correctly [10:19] zyga-mbp: thank you [10:19] I really wonder what tests will say [10:19] I never tried this [10:20] 9084 needs a second review if someone has some spare cycles [10:40] PR snapd#9120 closed: interfaces: add kernel-crypto-api interface [10:40] PR snapd#9165 closed: interfaces: add kernel-crypto-api interface - 2.46 [10:44] looking [10:45] PR snapd#9167 closed: many: correctly calculate the desktop file prefix everywhere [10:50] pstolowski https://github.com/snapcore/snapd/pull/9084#pullrequestreview-469207489 [10:50] PR #9084: o/snapstate: check disk space before creating automatic snapshot on remove (3/N) [10:50] brb, I'll fetch some water [10:52] 9081 also needs a second review [11:00] jdstrand: 8301 has conflicts now, could you please have a look? [11:13] pstolowski: just fyi, I uploaded a new snapd to groovy with the lxd generator fix so that should allow groovy lxd preseed images. could you please let the relevant people know (still needs some hours before it enters groovy proper, still building of course) [11:18] mvo: sure, thanks [11:36] * zyga fetched lunched instead [11:36] re [11:40] opensuse tumbleweed seems to be unhappy [12:46] error: cannot query the store for updates: got unexpected HTTP status code 503 via POST to "https://api.snapcraft.io/v2/snaps/refresh" [12:46] more store woes [12:49] pstolowski: ping [12:49] I have this in a debug shell [12:49] + lxd.lxc exec my-nesting-ubuntu -- snap set lxd waitready.timeout=240 [12:49] error: cannot perform the following tasks: [12:49] - Run configure hook of "lxd" snap (snap "lxd" has no "configure" hook) [12:50] nested snaps are all broken, not mounted [12:50] zyga: ok, that's weird [12:51] zyga: is it master? [12:51] yep [12:51] but seems random [12:51] https://paste.ubuntu.com/p/3YspQRfGvT/ [12:51] interesting failed unit [12:51] no snap mounted [12:52] I guess one idea would be to not run hooks on broken snaps [12:52] the snaps are on disk [12:52] looking at the mount units [12:53] there are no mount units?!? [12:53] did i broke something with systemd generator thing..? [12:54] maybe [12:54] looking [12:54] mvo: was #9152 green when it landed, or did you merge manually? [12:54] PR #9152: cmd/snapd-generator: generate drop-in to use fuse in container

[12:55] fstab has only LABEL=cloudimg-rootfs / ext4 defaults 0 0 [12:55] but there's no /dev/disk/by-label [12:55] I guess that's because of a container [12:55] and also explains the failed unit [12:55] seems like a separate container bug [12:56] I don't see any evidence that snapd generator ran [12:56] well [12:56] actually [12:56] I do see snap.mount [12:56] pstolowski: I thought it was, maybe I was mistaken [12:56] and nothing else [12:56] pstolowski: do you see errors now? [12:57] what's the other file we should have created? [12:57] the file /run/systemd/container is present [12:57] and has the word "lxc" [12:57] I guess that's why nothing mounted btw [12:57] but it's funny [12:57] I guess terrible [12:57] that we INSTALLED snaps in broken state after mounting failed [12:57] mounting should have undone everything [12:58] changes and tasks from the container https://paste.ubuntu.com/p/VPH4DwbGvx/ [12:58] zyga: there should drop-ins in /run/systemd/generator for all the snap mount untis for /etc/systemd/system [12:59] zyga: are there mount units in /etc/../system ? [12:59] there are no drop ins [12:59] https://paste.ubuntu.com/p/ft8pr6qXrC/ [12:59] and there are no mount nits [12:59] but we remove them in undo [13:00] although this makes no sense because "snap list" shows all the snaps installed and broken [13:00] this is tests/main/lxd [13:00] perhaps it's using older non-repackaged core18 + snapd [13:00] hmm hmm [13:00] zyga: ok if there are no mount units then root cause is elsewhere, not in the generator [13:00] can you just spread 16.04 + tests/main/lxd [13:00] and see if that fails [13:01] pstolowski: well, mount units do get removed on undo [13:01] but I really don't know what to make of this [13:01] the generator only has the /snap sharing mount [13:01] but doesn't have the drop ins [13:01] ah [13:01] 1337.2.46~pre2 [13:01] well [13:01] 1337 is the repackaged [13:01] so... [13:01] hmmm [13:01] zyga, pstolowski standup [13:01] it seems broken [13:01] oh [13:01] joining [13:17] mvo: hi! certainly, I'd love to. thanks for the reviews/merges! :) [13:17] jdstrand: thank you! [13:20] mvo: fyi, the cups one should good for re-review whenever people have a chance (though I'm going to merge master real quick). I'll quickly respond to any feedback there. moving on to 'misc policy updates' then after, will go through any PR reviews that need me [13:21] jdstrand: could you try to look at https://github.com/snapcore/snapd/pull/7614 again [13:21] PR #7614: cmd/snap-confine: implement snap-device-helper internally [13:21] jdstrand: I pushed it forward [13:23] zyga: sure [13:23] zyga: main/lxd failed for me as well [13:23] good [13:23] so it's a real thing [13:23] (reproducible) [13:24] pstolowski: try reverting the generator changes to see if that does something [13:25] "corecfg: add "system.timezone" setting to the system settings" [13:25] nice! :) [13:25] jdstrand: snap regedit ... [13:25] ;-) [13:25] ah, I wonder if that lxd thing is what was causing my spread tests to fail... [13:25] zyga: heh [13:25] jdstrand: I think we may have broken master [13:26] zyga: ok will do just in case, although it would contradict what ijohnson said [13:26] pstolowski: sure - just a sanity check [13:26] oh it wasnt't squash-merged [13:27] * jdstrand will keep an eye on commits to master for something to unbreak things then [13:27] zyga: yeah I saw it on my PR's last night before the generator stuff had landed [13:27] ah, sorry, I couldn't hear that exactly [13:27] that's good, maybe something environmental changed [13:36] jdstrand: and comments in that PR are a bit in a weird state, I could not reply to some [13:36] jdstrand: so please open new comments on anything that you find wrong [13:37] PR snapcraft#3165 closed: Update cmake plugin to support Ninja generator [13:39] pstolowski, ijohnson: interesting [13:39] in a debug shell look at the other container (my-ubuntu) [13:39] it has core installed as well [13:39] and guess what, core works [13:39] all the seeded snaps are broken [13:39] core is not seeded [13:40] does this make _any_ sense to you? [13:40] maybe what is really broken [13:40] is that /etc/systemd/system just doesn't have any mount units [13:40] maybe the image is broken to begin with [13:40] * zyga looks [13:42] pstolowski: confirmed [13:42] the image is busted [13:43] however it was made is wrong [13:43] pstolowski: should I expect to see mount units inside a squashfs lxd image? (in a container image downloaded by lxd) [13:43] I see the snaps and everything else [13:43] but the mount units are simply missing [13:44] as if it was seeded but then something filtered out the .mount units [13:44] pstolowski: to reproduce look at /var/snap/lxd/common/lxd/images/97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540.rootfs on /tmp/dupa type squashfs (ro,relatime) [13:44] cc stgraber ^ [13:44] mvo: fyi, PR 8301 is deconflicted [13:44] PR #8301: interfaces/many: deny arbitrary desktop files and misc from /usr/share [13:45] stgraber: it seems that 20.04.1 images that lxd pulls in are snap seeded in a broken way [13:45] stgraber: etc/systemd/system does not have any mount units [13:45] zyga: that'd be a bit concerning as the squashfs we use is supposed to be identical filesystem as the cloud image [13:45] stgraber: can you double check that I'm not making some rookie mistake [13:46] I mounted the .rootfs as downloaded by lxd [13:46] and looked at /etc/systemd/system therein [13:46] it's this image in your case: http://cloud-images.ubuntu.com/releases/focal/release-20200804/ [13:47] are they all identical? [13:47] * zyga pulls the squashfs [13:47] they should be [13:48] a bit slow to pull, I'll let it do its thing [13:48] i wonder if they started preseeding them and something went wrong [13:48] the date says 4th of August [13:48] maybe it is before we had some fixes? [13:49] and it's just busted \ [13:50] zyga, my yesterdays focal image that i installed looks fine here ... why would mount units be n the readonly squashfs though ? [13:50] ogra_: IIRC because we pre-seed things [13:50] so they start faster [13:50] isn't the squashfs expanded anyway, it's just a template [13:50] it's not a snap [13:51] root@focal:~# ls /etc/systemd/system/*.mount|wc -l [13:51] 8 [13:51] ogra_: that's not what I'm seeing [13:51] root@focal:~# snap list|wc -l [13:51] 6 [13:51] what are the mount units? [13:51] I had 0 mount units [13:51] (though I'm still getting the image stgraber suggestd) [13:51] root@focal:~# ls /etc/systemd/system/*.mount [13:51] root@focal:~# [13:51] bah [13:51] let me pastebin that [13:51] I saw the failure in a spread test that was just "lxd launch ..." things [13:51] thanks [13:52] https://paste.ubuntu.com/p/HJJ3NN9hxm/ [13:52] right [13:52] this is more like what I would expect [13:52] I think an image is broken then, the only question is why and how to fix it [13:52] stgraber@castiana:~/Downloads$ tar Jtvf ubuntu-20.04-server-cloudimg-amd64-root.tar.xz | grep etc/systemd | grep mount [13:52] thats what i got with "lxc launch ubuntu:20.04 focal" ... [13:52] stgraber@castiana:~/Downloads$ [13:52] zyga: also, it's from 4th Aug but started failing yesterday? [13:53] pstolowski: yeah, I think we are missing something [13:53] stgraber: 97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540.rootfs can we map that hash to something on cdimage? [13:53] pstolowski: assuming it's using ubuntu:, the images are manually promoted by CPC IIRC, though indeed feels a bit old [13:53] zyga: I did it for you [13:53] zyga: that's the URL I gave you [13:54] ah [13:54] * zyga looks at one more idea [13:54] current ubuntu:focal is indeed 20200804 on cloud-images.u.c [13:54] ubuntu-daily:focal would be 20200814 [13:55] there's a pretty good chance that the cloud images would similarly be missing those files, it's really meant to be identical across the board [13:55] so you probably should be talking to CPC :) [13:56] PR snapd#9173 closed: cmd: compile snap gdbserver shim correctly [13:57] thanks mvo [13:57] sorry had to step out for IRL things [13:57] so is the lxd image just busted then ? [13:58] ijohnson: it seems so but unclear why [13:58] I noticed something buggy in the test [13:58] looking now [13:58] lxd.lxc launch --quiet "ubuntu:${VERSION_ID:-}" my-ubuntu [13:58] VERSION_ID is not defined anywhere [13:58] what does that fetch? [13:58] latest? [13:59] ah so you went from bionic to focal last week then [13:59] I think it's a bug and it was meant to fetch the one matching [13:59] so it's likely that this bug triggers another one [13:59] "ubuntu:" fetches the latest recommended LTS [13:59] that was bionic until 20.04.1 [13:59] now is focal [14:00] zyga: I think we get VERSION_ID from /etc/os-release which gets sourced somewhere in the prepare for a spread run iirc [14:00] ijohnson: I doubt that [14:00] thre was some code in that task.yaml [14:00] it's gone now [14:00] must have been killed by accident [14:00] and ${VERSION_ID:-} masked that [14:00] haha fair enough [14:00] it used to load version id from the host [14:00] because it was about alignment of the container to the system [14:00] as we copy stuff inside [14:01] zyga: i'm looking at ubuntu-20.04-server-cloudimg-amd64-root.tar.xz (also dated 4/08) is this as good as any other image there? fwtw it is NOT preseeded, it's still seeding the old way [14:01] I see [14:02] I'll start by fixing that VERSION_ID [14:02] but it seems we have more bugs [14:02] pstolowski: can you focus on trying to understand what's going on in the inner layer [14:03] I'll send a patch fixing the symptom we see [14:05] ijohnson: you broke it ;D [14:06] * ijohnson runs and hides [14:06] zyga: was it my use the same version of lxd everywhere PR ? [14:06] d4a802934b1e62ec8924ea1c165a627695df5044 [14:06] that's the broken patch [14:06] anyway, fixes coming [14:07] zyga: haha to be fair though you approved the PR :-D [14:07] no hard feelings [14:07] at this point I'm beyond shame [14:07] I made so many bugs [14:07] yeah bugs happen [14:07] no worries [14:08] zyga: so what's actually broken? do i need to keep looking? [14:08] pstolowski: that test is broken when invoked with a focal image [14:08] pstolowski: on a xenial host [14:08] pstolowski: it should work [14:08] pstolowski: to reproduce just grab xenial VM [14:08] pstolowski: install lxd the same way [14:08] pstolowski: and boot focal [14:09] pstolowski: that should work and give you a working focal container with working snaps [14:09] zyga: ok so it's a test issue, not a real problem? [14:09] pstolowski: it's a real problem [14:09] pstolowski: the test just now stumbled onto it [14:09] pstolowski: it was not exercising that before [14:09] ah ok [14:09] pstolowski: but effectively started doing that because of another issue and recent promotion of focal [14:10] pstolowski: PR coming up, I've explained it there as well [14:11] just confirming that my patch fixes it [14:17] zyga: so the right thing for this lxd test is for a xenial host to run a xenial container and a focal host to run a focal container ? [14:17] ijohnson: yes [14:17] there's a comment there explaining why [14:18] zyga: I wonder if we should expand the lxd test to have more variants i.e. xenial host launches focal, bionic and xenial [14:18] oh [14:18] * ijohnson goes to read that comment [14:18] yes but then we should not just blindly copy snapd [14:18] but I agree [14:18] ah yes [14:18] I see [14:18] see if we built the snapd snap and used that as part of our tests we could avoid this [14:18] yes [14:18] I'm sure we could [14:18] it's just a limitation of the current setup [14:18] all proper builds will work [14:19] (properly built rpm will work on rpm distros) [14:19] geez that test is sloooooow [14:19] mmmm also this test is disabled for uc20 too [14:19] (network) [14:19] we should re-enable that and make sure lxd works on uc20 [14:19] I agree [14:19] maybe worth splitting lxd test [14:19] and thinking of a cache for lxd images if that helps [14:19] FYI https://github.com/snapcore/snapd/pull/9174 [14:19] PR #9174: tests: fix lxd test wrongly tracking 'latest' <⚠ Critical> [14:20] opened while local test is still going [14:20] so likely it works now [14:20] but I want to break now as my wife just returned from her first day of work after Lucy was born [14:20] I'll work later in the evening, to focus on bug triage [14:20] I think it's still broken though [14:20] just got this [14:21] https://paste.ubuntu.com/p/yJ6HThPdN6/ [14:21] PR snapd#9174 opened: tests: fix lxd test wrongly tracking 'latest'