[05:44] morning [06:08] mvo: hey [06:08] mborzecki: good morning! [06:11] hmm something for zyga https://paste.ubuntu.com/p/fMmRQNryVs/ [06:21] mborzecki, mvo: we must disable parallel instances for classic [06:22] mborzecki: that is already excluded, I suspect someone ran a PR from earlier master [06:22] althouhh [06:22] *although [06:22] 2020-04-17T20:30:19.4282018Z + find /run/snapd -user test '!' -path '*/user@12345.service/*' [06:22] interesting, so there's another form on 20.04 now [06:22] oh well [06:22] I'll exclude that one too [06:22] thanks [06:23] thanks zyga [06:23] https://bugs.launchpad.net/snapd/+bug/1873701 [06:23] Bug #1873701: support for parallel instances on classic breaks all snap remove in lxd [06:23] zyga: what's broken with paralle instances? [06:23] mvo: I believe this is urgent [06:24] mvo: not security related [06:24] mvo: it's just a bug that is particularly annoying, you cannot remove snaps, system is in a broken state [06:24] zyga: right, when does it happen? it's behind a experimental flag so how easy is it to trigger? [06:25] mvo: IIRC the flag is enabled on master now [06:25] mvo: basically using edge breaks when you try to remove snaps [06:25] zyga: iirc it's still experimental on master [06:26] mborzecki: hmmm then something is broken [06:26] mborzecki: we may not respect the flag then [06:26] mborzecki: it definitely takes effect on edge [06:26] zyga: also why did we not caught this in tests :/ [06:26] mborzecki: try the bug instructions please [06:26] zyga: what bugnumber is that again? [06:26] mvo: just above [06:26] mvo: 1873701 [06:28] mvo: in other news snapd is entirely unusable in a 14.04 lxd container [06:28] mvo: due to at least three bugs [06:28] zyga: hm not sure i see it, how's that parallel instances? [06:28] mvo: not sure if we want to be explicit about that [06:28] mborzecki: it's the /snap mount point [06:29] ├─/snap /dev/sda5[/var/snap/lxd/common/lxd/storage-pools/default/containers/sid/rootfs/snap] [06:29] we created this for parallel instances [06:29] mvo: and there's a typical duplication of entries it causes [06:29] er mborzecki ^ [06:30] zyga: ok, see that now, isn't that the shared code in s-c? [06:30] yes [06:30] zyga: oh, interessting. we sometimes hat "cannot umount ..." errors, do you think that could explain it? [06:30] mborzecki: I think it just runs [06:30] mvo: perhaps [06:30] mvo: would have to look at a specific instance [06:31] mvo: related to your work, another bug: https://bugs.launchpad.net/snapd/+bug/1873703 [06:31] Bug #1873703: snapd doesn't start in a Ubuntu 14.04 container [06:31] mvo: this one I believe affects 14.04 in general, I cannot explain why it doesn't fail in our tests [06:31] mvo: it's also easy to reproduce [06:31] mvo: one more https://bugs.launchpad.net/snapd/+bug/1873704 [06:31] mvo: I guess we just never tested this case [06:32] Bug #1873704: snapd cannot be used in a Ubuntu 14.04 container - no squahsfuse [06:32] mvo: there's one more bug that I didn't report, 14.04 containers do not have loopback control device, making it impossible to mount any snap, even with side-loaded snap/squashfuse [06:32] but I want to ask stgraber about it first [06:34] in other news, good morning [06:34] zyga: the lxd one is confusing [06:34] sorry to bring bad news up front [06:34] zyga: dpkg -L snapd |grep fuse [06:34] /usr/bin/snapfuse [06:34] mvo: where? it's not in the archive package [06:34] zyga: apt list --installed snapd [06:34] Listing... Done [06:34] snapd/focal,now 2.44.3+20.04 amd64 [installed,automatic] [06:34] mvo: perhaps I messed something up but it was definitely not there when I tried [06:35] zyga: / wasn't shared under lxd, was it? [06:35] mvo: that's a focal package :) [06:35] mborzecki: correct [06:35] mvo: 14.04 package is older and doesn't have it [06:35] zyga: right, so this is where the bind mount /snap -> /snap comes from [06:35] zyga: maybe it's another instance of 14.04 deb needing an update, I pushed something some weeks ago and then it lost with the people who can actually update things in 14.04 :/ [06:35] mborzecki: I see [06:35] zyga: old pkg> aha, yes, then that's the case [06:36] mborzecki: I wonder what's the new thing then, was removing snaps in containers always broken [06:36] mborzecki: can you reduce my reproduction case to something smaller [06:36] maybe the edge snapd there is not required? [06:36] mborzecki: it was kind of late when I reported it [06:39] zyga: looking at the pr with classic & parallel, the change was to have that mount ns setup when we're running a !classic snap || the experimental flag is set [06:39] zyga: then we make sure that /snap is shared bind mount if / isn't [06:40] (so lxd case?) [06:40] the experimental flag would also trigger /var/snap mount [06:40] indeed! [06:44] mborzecki: so I guess the interesting combination is the edge refresh [06:44] mborzecki: this code used to be absent [06:44] mborzecki: right? [06:48] zyga: hmm i would think a scenario like this, install (mount appears), you run a snap (/snap -> /snap appears), refresh happens (new mount appears), remove only unmounts /snap/foo/{n,n+1}, the original /snap/foo/{n} mount hidden by the /snap -> /snap is still present? [06:49] mborzecki: I don't think so, the app snap has only one revision [06:49] zyga: isnt't that similar to the problem we tried to solve with snap.mount unit that broke things on upgrade? [06:49] mborzecki: it's the core snap that gets refreshed [06:50] zyga: ah core, ok, but the app snap was run, and it triggerd /snap -> /snap bind mount [06:51] yes, that part is essential [06:53] zyga: then the mount /snap/open-watcom that had the /snap->/snap as its parent was cleaned up, but one that was under / was not [06:54] that's how I understand it as well [06:54] hm it'd be intersting to check that on older snapd, but that shared / check was present for a long time [06:54] mborzecki: perhaps we should do something else, detach everything and remount / attach [06:55] wonder if we call it earlier now [07:16] PR snapd#8522 opened: asserts: introduce ModelGrade.Code [07:17] morning [07:17] pstolowski: good morning! [07:36] PR snapd#8523 opened: image,seed/seedwriter: support redirect channel aka default tracks [07:40] PR snapd#8524 opened: bootloader: use binary.Read/Write [07:44] mvo: mborzecki: hi, I proposed some small PRs, two are UC20 related, one is the default track fix for prepare-image [07:57] pedronis: thank you [07:58] mborzecki: is #8487 ready for reviews? [07:58] PR #8487: gadget, cmd/snap-bootstrap: MBR schema support [07:58] pedronis: yes, please do [08:52] re [09:02] mborzecki: https://github.com/snapcore/snapd/pull/8525 [09:03] PR #8525: tests: ignore user-12345 slice and service [09:03] PR snapd#8525 opened: tests: ignore user-12345 slice and service [09:03] mborzecki: I think that should fix it [09:13] I need a review for https://github.com/snapcore/snapd/pull/8525 [09:13] PR #8525: tests: ignore user-12345 slice and service [09:13] er wait [09:13] bad paste [09:13] https://github.com/snapcore/snapd/pull/8515 [09:13] PR #8515: testutil: add NewDBusTestConn [09:29] mvo: hey, found one issue in your initramfs PR; i'm testing it this morning and something is still not quite right, same symptoms as before with cloud-init 🧐 [09:29] (to be clear, your change takes effect and does the thing) [09:36] meh everything is red === pedronis_ is now known as pedronis [10:09] pstolowski: aha, do you have more details? or maye the image so that I can run it myself? [10:16] mvo: cloud-init is not run, extra-users are not created, don't have any other clues yet. i can share the image, give me a moment. nb ubuntu-image will need a change afaict, atm i'm moving system-defaults to the proper location alongside system-data and user-data manually [10:21] pstolowski: yeah, sharing the image would be ideal, if it's not too much effort [10:21] pstolowski: that should allow me to peek at things [10:22] pstolowski: might be something silly, I really want to write this spread test but didn't had time yet :( [10:26] mvo: aaah i think i know [10:26] mvo: i didn't update writable-paths [10:28] pstolowski: no update it in what way? [10:28] mvo: ah nvm, got confused. of course i'm not writing to /etc/systemd anymore, so not an issue. hmmm [10:28] mvo: ok i uploaded the image [10:29] pstolowski: please share the link [10:30] mvo: sent you gdrive link [10:30] pstolowski: checkin [10:30] g [10:32] pstolowski: downloading now [10:34] mvo: actually, it seems it didn't copy the rsyslog symlink from system-data-defaults to system-data (but did run, .done is there). it doesn't explain why cloud-init didn't run though [10:35] pstolowski: interessting [10:35] pstolowski: booting mine now [10:35] maybe i did a mistake somewhere [10:40] pstolowski: inspecting now, looks like something funny/funky with cloud-init [10:41] pstolowski: fwiw, I see the /dev/null symlink in /etc/systemd/system/rsyslog.service [10:42] mvo: yes. but this might have been created by configure hook later [10:42] pstolowski: aha, ok [10:42] pstolowski: thanks, that's good to know [10:42] mvo: we should see rsyslog -> null symlink in system-data/etc/... as well, copied from system-data-defaults afaiu? [10:44] pstolowski: yes, I see it there [10:44] pstolowski: I don't see the var/lib/snapd/features there but that's because snapd cleans this dir on restart [10:44] mvo: ah jeez, you're right it's there. i must have been blind [10:45] pstolowski: no worries, let me look at cloud init some more [10:45] mvo: interesting (about features) [10:45] pstolowski: to remove features future snapd wrote but we don't know abut [10:46] zyga: yes, makes sense [10:48] pstolowski: and I suspect the state has this opiton not set (the experimental robust...) [10:49] mvo: note, robust is now default [10:49] when uset [10:49] unless you explicitly disabled it it should be enabled [10:49] unless this snapd is simply older [10:49] pstolowski: as for cloud-init, it seems like it cannot found a data source, is there a cloud init no-cloud data file somewhere in this image? [10:50] zyga: aha, thanks [10:53] mvo: ahh, i forgot to copy cloud-data onto the image! [10:53] pstolowski: hopefully that is the missing piece of the puzzle! [10:53] mvo: yep. sorry about that! lots of hops and manual steps [10:53] pstolowski: if it looks good after this test I will update my core-build PR and also provide this for core20 [10:53] pstolowski: yeah, no worries [10:53] mvo: note the .done bug [10:54] pstolowski: yeah, thanks for spotting this, silly me [10:54] pstolowski: that's what I meant with "will update my pr" :) [10:56] mvo: afaiu we also need to update ubuntu-image [10:57] pstolowski: please remind me for what bits? [10:57] pstolowski: could config? [10:58] we need to keep it as synced for UC 16/18 because we don't know what people did with hooks [10:58] so u-i should still work [10:58] for them [10:58] we want to clean this up a bit but shouldn't be a blocker, or so I think [10:58] yeah, this is why I'm asking, I wonder if there is more [11:02] mvo, pedronis it doesn't seem to take system-data-defaults directory at the root level (alongside user-data and system-data) [11:02] unless i messed something [11:02] ah [11:03] atm i writing them under system-data and move manually before booting for the first time, jsut for the test [11:03] that's different and might need fixing [11:03] it also works differently in UC16/18 vs UC20 [11:03] i didn't dig into u-i python code yet [11:03] the directory structure is different [11:05] pedronis: should we put defaults under system-data to simplify? [11:05] PR snapd#8524 closed: bootloader: use binary.Read/Write [11:05] pstolowski: maybe, something to discuss with mvo [11:07] mvo: success! [11:07] mvo: cloud-init works [11:12] mvo: ^ so the last bit to consider is what i just discussed above with Samuele [11:17] brb, a small break [11:41] PR snapcraft#3060 closed: V2 npm plugin [11:46] mborzecki: I reviewed #8487 [11:46] PR #8487: gadget, cmd/snap-bootstrap: MBR schema support [11:46] pedronis: thanks [12:01] PR snapd#8523 closed: image,seed/seedwriter: support redirect channel aka default tracks [12:10] PR snapd#8526 opened: image: improve/adjust DownloadSnap doc comment [12:14] PR snapd#8527 opened: seed: clearer errors for missing essential snapd or snap [12:14] pstolowski: aha, indeed, I think this needs fixing in u-i [12:15] pstolowski: let me think about this for a moment - we could use /system-data/.defaults [12:32] mvo: FYI: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools-ubuntu-core/+bug/1873797 [12:32] mvo: I didn't look deeper [12:32] Bug #1873797: add_mountroot_fail_hook is either buggy or not used properly, causes boot problems [12:34] zyga: nice, thanks [12:37] zyga: should https://bugs.launchpad.net/snapd/+bug/1872115 be re-assigned to core? i don't know what to do about it tbh [12:37] Bug #1872115: [UC16] log rotation doesn't properly restart rsyslogd [12:37] pstolowski: sure, feel free to reassign [12:38] pstolowski: I can look after beta [12:38] mvo: ^ if you agree [12:38] pstolowski: +1 [12:39] pstolowski: how do you feel about /writable/system-data/.defaults (cc pedronis) - this avoids the need to modify u-i I think [12:39] mvo: yes, this works for me, it's more less what i was suggesting [12:39] mvo: any particular reason to make it hidden? [12:40] pedronis: no real reason, mostly worried about potential future clashes but probably a bit silly to worry about this [12:40] pedronis: if not hidden, maybe /writable/system-data/config-defaults even ? [12:40] I actually think that making it hidden makes clashes a bit more likely [12:41] I have a /.cache which I don't know what makes it [12:41] pedronis: updated 8487 [12:42] pedronis: ok, not-hidden is fine for me [12:42] mvo: pstolowski: let me think a bit about the name [12:43] ok [12:44] mvo: all other directories there are meant to be bind mounted, right? [12:44] pedronis: correct [12:48] zyga: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools-ubuntu-core/+bug/1873797 is about the new initramfs code, right? [12:48] Bug #1873797: add_mountroot_fail_hook is either buggy or not used properly, causes boot problems [12:49] mvo: I don't know, it's just something that flashed through my inbox [12:49] zyga: it's the old stuff [12:49] zyga: meh, thanks! [13:19] presseding error on ubuntu 20.04 https://www.irccloud.com/pastebin/GMvrEUtj/ [13:19] mvo: ^ the log [13:19] PR snapd#8525 closed: tests: ignore user-12345 slice and service [13:27] zyga: yeah, I saw it, I suspect it's because something changed (lxd from using core->core18?) [13:27] ah, inded [13:27] that did happen just recently [13:41] ENOTEA [13:42] I'll eat lunch and make some tea [13:42] ttyl [13:45] mborzecki: you said you were going to try and pick up the ubuntu-boot grub.cfg writing from snapd? this is also needed for the pi4 boot, since when we run makeBootable20RunMode during install mode, we need the ubuntu-boot uboot.env to be there during the reboot's initrd, otherwise snap-bootstrap can't find a bootloader [13:46] mborzecki: what I was going to do in the short term until we rewrite the ubuntu-boot boot assets like this was to just have the pi gadget install it's empty uboot.env to ubuntu-boot, so that the gadget partition creation effectively installs an empty uboot.env there [13:59] PR snapcraft#3063 opened: cli: update command names to new design [14:20] ijohnson: to be clear, hard coding boot config is only for grub for now [14:20] pedronis: why only grub ? [14:20] ijohnson: because we measure thing only with grub atm [14:21] pedronis: but we basically have the same need for i.e. u-boot too for uc20? [14:21] not for measuring [14:21] but just for booting [14:21] we don't measure [14:21] so there no a deep constraint [14:21] also there's more variability on uboot land [14:21] not sure there will be one uboot.cfg [14:45] * cachio afk -> bank [15:32] mborzecki: did you see my messages above? [15:33] mborzecki: I am pushing forward with just having the gadget put a uboot.env on ubuntu-boot, and pedronis doesn't think it's necessary so probably for now if you pick up my branch just make non-grub bootloaders no-op on that [15:37] PR snapd#8528 opened: tests: naive fix for preseeding failure [15:38] PR pc-amd64-gadget#43 opened: Clean up cmdline, after initrd&core land [15:41] mborzecki: I re-reviewed #8487, couple small comments [15:41] PR #8487: gadget, cmd/snap-bootstrap: MBR schema support [15:41] pedronis: should I re-review that PR? [15:41] my original changes in the PR are probably not relevant [15:41] pedronis: thanks, i'll push it in a minute [15:41] it does need a 2nd review [15:41] ijohnson: sorry missed that, let me check the backlog [15:41] I'll try to review it today then, I'll also re-test it with the pi today [15:42] ijohnson: right, grub.cfg for now [15:42] mborzecki: yep [16:03] cachio: ping [16:05] pstolowski, hey [16:05] cachio: do you have a moment for HO? [16:06] yes [16:06] cachio: ok, in standup HO [16:28] pedronis: having small issue with the gadget + system-files plug + install hook approach as gadget is modified/unasserted, so no autoconected, also connections: in gadget yaml has no effect. not sure if there is any way around this :}. would devmode make sense? [16:30] pstolowski: do you know why connections in gadget don't work? [16:30] pedronis: i suppose snapid is not avilable [16:30] at least this is my understanding of the code atm [16:31] ah, yes [16:31] mborzecki: https://github.com/snapcore/snapd/pull/8487#pullrequestreview-396608940 [16:31] PR #8487: gadget, cmd/snap-bootstrap: MBR schema support [16:35] pedronis: i suppose i could update seed.yaml with devmode for the gadget (so the plug wouldn't be needed).. but that's slightly terrible [16:36] pstolowski: it's terrible but is the only thing that will work short term I fear, so it's ok, the confinment of the hooks is not really what the test is about [16:37] mhm [16:39] ? [16:40] pedronis: no disagreement [16:48] @reviewers bump https://forum.snapcraft.io/t/classic-confinement-request-for-nhc-snap [16:56] pstolowski: also preseed tests are failing on 20.04 [16:56] 2020-04-20T16:49:24.6022877Z + CORE_IMAGE= [16:57] 2020-04-20T16:49:24.6023208Z + unsquashfs '' [16:57] 2020-04-20T16:49:24.6023431Z Could not open , because No such file or directory [17:03] pedronis: seems he quit [17:03] pedronis: I thought I saw zyga send a PR for that [17:12] PR snapcraft#3061 closed: plugins: introduce v2.RustPlugin [17:24] PR snapcraft#3064 opened: cmake v2 plugin: rename configflags to cmake-parameters [17:25] ijohnson: yes, but it says it actually breaks in a different way [17:25] ah I didn't actually read the PR, I just saw the notification [17:28] ijohnson: pushed a fix, thanks for catching this === ijohnson is now known as ijohnson|lunch [17:32] so the issue is that cloud images now carry snapd snap, not the core snap [17:42] I disable those preseed tests on 20.04 for now [17:45] mmh, do we even need the hacks anymore === bdx1 is now known as bdx [18:15] PR snapcraft#3065 opened: tests: fix local plugin spread test to be multi-arch aware [18:48] PR snapcraft#3031 closed: build_providers: pass through SNAPCRAFT_{BUILD,IMAGE}_INFO to container or VM [18:58] PR snapd#8529 opened: cmd/snap-bootstrap/initramfs-mounts: support non-ebl bootloaders <⚠ Critical> === ijohnson|lunch is now known as ijohnson [19:00] pedronis: I opened ^ it would be great if you could review it in your AM and we land it ASAP, I think it's the last bit we need in the initrd specifically to support uc20 on raspi [19:00] PR snapd#8522 closed: asserts: introduce ModelGrade.Code [19:03] ijohnson: thank you so much for this [19:03] yaw mvo [19:04] I will have 1 maybe 2 other PR's in addition to this to get the pi working, but those are just for snapd proper [19:25] ijohnson: mmh [19:26] ijohnson: I think we should really avoid to touch unrelated tests [19:27] pedronis: ? [19:28] the Bootenv16 hack [19:28] pedronis: hmm I guess I can get around that, what do you think the uc20 tests that want to use uc16 style thing should look like? [19:28] there is probably a simple way to do it without having to touch unrelated tests [19:29] pedronis: how about `boottest.MockUC20LegacyKernelBootenv(bootloadertest.Mock("mock", c.MkDir()))` ? [19:29] also we should try to agree how to call the thing, because legacy is kind of misleading [19:29] we are stuck with this [19:29] and also it's using kernel_status [19:30] yes I really don't know what to call it [19:30] what I called it in 8512 is "pure env" [19:30] because it just strictly uses the "bootenv" [19:30] ijohnson: and sorry I'm coming from this angle, instead of great, yay progress. I suppose the touching of unrelated tests tend to put me in a slightly critical state of mind [19:31] pedronis: it's okay [19:31]