[06:03] morning [06:05] hey mborzecki [06:06] mvo: hey [06:07] PR snapd#7782 closed: osutil: handle "rw" mount flag in ParseMountEntry [06:09] PR snapd#7778 closed: seccomp: allow chown 'snap_daemon:root' and 'root:snap_daemon' [06:34] good morning [06:34] wow, everyone is up early [06:39] hey zyga - welcome to the right TZ! [06:41] mvo: ... :D [06:41] mvo: I took Bit for a walk at 1:30AM [06:41] mvo: but I feel okay now [06:42] mvo: I try to take a nap mid-day to compensate [06:42] mvo: I'll jump into the fray soon [06:43] zyga: hah 1:30am ;P sounds like a different tz to time [06:43] zyga: cool [06:45] mvo: regarding https://github.com/snapcore/snapd/pull/7772#pullrequestreview-322085662 maybe something the generators could do, but those run before local fs mounts, so it'd have to be in the core snap if i understand it correctly [06:45] PR #7772: wrappers: write and undo snapd services on core [06:53] mborzecki: aha, interessting idea [07:29] mborzecki: mvo: hi, should we have a chat about the snapd configure issues? [07:30] pedronis: hi, sure [07:31] mborzecki: mvo: standup? [07:32] pedronis: ok === pstolowski|afk is now known as pstolowski [08:02] morning [08:09] pstolowski: hey [08:28] PR snapd#7765 closed: boot,bootloader: make MakeBootable() uc20 aware [08:44] mvo: I did a pass on #7746 [08:44] PR #7746: snap-bootstrap: implement "snap-bootstrap initramfs-mounts" [08:54] PR snapd#7789 opened: bootloader: add new bootloader.InstallBootConfig() [08:59] hi all, store team here. Downloads might see some errors atm, we're working on it [09:00] tomwardill: hi, thanks for heads up! [09:02] pedronis: thank you! [09:10] mvo: hi! how can i try out this snapfuse fix on bionic? https://bugs.launchpad.net/snapd/+bug/1817276 [09:10] Bug #1817276: snapfuse use a lot of CPU inside containers [09:11] mvo: it seems like the snapfuse binary comes from the snapd deb, which is still 2.40 [09:15] BjornT: we have a updated it in bionic-proposed [09:16] BjornT: I think the sru was actually validated so it's mostly a matter of getting this released [09:16] BjornT: I can poke foundatoins [09:21] mvo: cool, i'll install it from -proposed then [09:26] mvo: pedronis well, the spread test seems to be working now with the tweaks in configstate [09:28] mvo: nice. confirmed that the fix is a major improvement. after upgrading snapd to 2.42, my load average isn't constantantly growing anymore and back to normal levels [09:29] BjornT: yay! if you could give me numbers from your workload that would be awesome [09:31] mvo: what kind of numbers? i haven't done any performance tests yet. but before snapfuse was taking 100% constantly, probably while syncing images, which requires a lot of writing. now it still can burst up to 90%, but it quickly goes down again. [09:32] mvo: and the load average just kept growing. it was up to 900 before i upgraded snapd [09:33] 900?! [09:34] BjornT: thanks, that was what I was looking for, some details :) [09:35] mborzecki: great [09:41] mvo: reviewed 7789 [09:52] pstolowski: thanks for the review, I applied the suggestion [09:52] yw [09:53] pedronis: thank you! [09:57] Chipaca: is #7671 ready for review ? [09:57] PR #7671: overlord/patch: normalize tracking channel in state [09:58] pedronis: AFAIK :) [09:58] Chipaca: ok :) [10:05] mvo: now that I remembered how the current bootloader.Options flag is used by lk, I think we do want a 2nd flag, probably just InstallMode bool (would be false for UC 16 / 18) [10:06] mvo: doesn't block the current PR but would be relevant for next step [10:07] pedronis: I was thinking about a "Recovery" flag that could be used by "bootloader.Find()" and "InstallBootConfig()" to tell the system to install bootconfig for recovery or real-boot. That should also address this, but I'm happy with InstallMode as well [10:07] mvo: that works too, Recover = true = InstallMode [10:07] Recovery [10:08] PR snapd#7746 closed: snap-bootstrap: implement "snap-bootstrap initramfs-mounts" [10:08] Chipaca: does #7671 need a master merge? [10:09] PR #7671: overlord/patch: normalize tracking channel in state [10:09] pedronis: ah, maybe; i'll look in a bit [10:11] Chipaca: I see things that I thought I reviewed already, but maybe I'm confused (that's possible) === arnatious_ is now known as arnatious [10:20] sil2100, ogra: do you know who is maintaining the pi gadgets today [10:22] degville: I was looking at https://snapcraft.io/docs/gadget-boot-assets and noticed it's not linked from https://snapcraft.io/docs/gadget-snap - just wanted to check with you if I should edit and make the connection or is there another path somewhere [10:23] zyga: I think you're right - I'd not added the link yet pending the boot assets edits and final changes, but it makes better sense to link it now. [10:24] (as doc and boot assets are in pretty good shape) [10:45] PR snapd#7787 closed: many: share single implementation to list needed default-providers [10:52] PR snapd#7790 opened: boot: add boot.Modeenv that can read/write the UC20 modeenv files [10:53] PR snapd#7789 closed: bootloader: add new bootloader.InstallBootConfig() [10:53] pedronis_: hopefully simple and then I can build on top of this === pedronis_ is now known as pedronis [10:55] mvo: commented [10:59] pedronis: thank you! [11:18] pedronis: ok if i rebase 7671? [11:23] pedronis: i've rebased 7671 [11:24] pedronis: it looks a lot more like a patch patch now [11:32] pedronis: addressed your comments, thanks again [11:35] zyga: hey! That's us (foundations), but it's basically waveform mostly [11:39] sil2100: thanks, I'll reach out to him [11:44] Chipaca: thx, I will look in a bit [11:44] mvo: that's not the rename I proposed, I proposed to match what we use in the file (up to format) [11:44] :) [11:45] mvo: ah, not the code is right, but the commit says another thing [11:51] mvo: +1 with some more comments, mostly about the tests [11:53] pedronis: thank you, on it! [11:53] PR snapd#7791 opened: snap-bootstrap: make cmdline parsing robust [12:05] mvo: pedronis: i've updated #7788 please take a look [12:05] PR #7788: overlord/snapstate: pick up system defaults when seeding the snapd snap <⚠ Critical> <⛔ Blocked> [12:05] need to sort out my flights [12:06] mborzecki: thanks, will look in a bit [12:07] Chipaca: reviewed, maybe pstolowski can give a look too [12:07] mborzecki: about flights, I looked at trains [12:08] mborzecki: but it's a no-go because it's impossible to get tickets for a ride this distant [12:08] mborzecki: assuming that train schedules don't change it's a whole-day trip with one stop in Berlin [12:08] zyga: and it's probably half a day [12:08] mborzecki: and the price is very competitive with flying [12:08] mborzecki: even if you consider first class actually [12:09] mborzecki: but I decided not to use it because << 2 hours is meh-good-enough [12:09] pedronis, Chipaca will do [12:09] mborzecki: there are a number of connections, some take over half a day and involve many stops, some are fast [12:15] mborzecki: done, thank you [12:16] mborzecki: have you looked at the other issue (installing core on a snapd-only system with defaults) or not yet? [12:17] pedronis: not yet, spent silly amount of time fighting the configstate handlers test [12:18] 7790,7791 should be simple reviews and unblock some more of the early boot work [12:18] * mvo considers lunch [12:18] mborzecki: lots of setup needed [12:18] pedronis: might have overdone it a little [12:18] jdstrand: good morning [12:20] PR snapd#7759 closed: devicestate: add reading of modeenv to uc20 firstboot code [12:24] pedronis, Chipaca will do [12:25] ups, wrong window [12:40] mborzecki: 2nd +1 on https://github.com/snapcore/snapd/pull/7788#pullrequestreview-322948196 [12:40] PR #7788: overlord/snapstate: pick up system defaults when seeding the snapd snap <⚠ Critical> [12:41] zyga: thanks [12:50] I'll make coffee and resume patches [12:59] cmatsuoka: hi, quick question about #7743, in the real world scenario, we wouldn't need sfdisk --force, because snap-bootstrap will be running off initramfs right? [12:59] PR #7743: snap-bootstrap: force partition table operations [13:07] mborzecki: it's likely that it will be needed in the real case, I added snap-bootstrap to the spike and it required --force [13:08] mborzecki: --force was completely under my radar until I tested it in a real scenario [13:09] mborzecki: the same happened with the udev node creation, for some reason partx -u triggers node creation but blockdev --rereadpt doesn't [13:10] mborzecki: I started to check the partx source code to see what exactly it's doing there, but it does quite a lot [13:13] cmatsuoka: interesting, i would guess that sfdisk handles that when it triggers the kernel to reread the partition table === ricab is now known as ricab|lunch [13:14] cmatsuoka: afaiu that's what partx does by doing a bunch of ioctls() though it passes an actual partition number there [13:15] cmatsuoka: either way, this goes through the kernel -> devfs, and in parallel to udev which does symlinking under /dev/disk/by-*/, that's probbaly why the extra delay is needed [13:15] I also tried to manually run udevadm trigger but it didn't do anything [13:16] cmatsuoka: was any paritition from taht device mounted at the time when sfdisk ran? [13:16] According to a comment in the partx source code, it needs a few ms to settle after the operation [13:18] mborzecki: hmm, yes, it's possible that we had partitions mounted in initramfs [13:18] mborzecki: ah, I guess core was mounted from seed directly at that point [13:18] cmatsuoka: that would explain why the disk was seen as 'used' [13:19] mborzecki: yes, indeed. but we can't unmount everything, we need seed an boot [13:19] and boot [13:22] cmatsuoka: right, i guess there's no way around that, so --force seems ok [13:22] cmatsuoka: perhaps that's why you needed to call partx [13:23] mborzecki: about the partx -u, I don't like the idea that we're relying on a side effect of a specific utility -- ideally I'd like to redo whatever partx is doing there [13:23] but that would take some time to analyse [13:23] cmatsuoka: was there any mesage from fdisk? specificaly thinking https://github.com/karelzak/util-linux/blob/master/libfdisk/src/context.c#L833-L840 [13:25] mborzecki: no such message, the PT re-read was apparently successful [13:25] heh, fun ;) [13:26] those real life problems are very annoying! [13:27] also I hope that kernel 5.4 doesn't crash in firstboot, 5.3 did but I still didn't check why [13:31] mborzecki: I think I should add an XXX note there [13:32] cmatsuoka: yup, sounds like a good idea [13:32] mborzecki: doing that === ricab|lunch is now known as ricab [14:23] PR snapd#7790 closed: boot: add boot.Modeenv that can read/write the UC20 modeenv files [14:35] anyone know of a tool that lists what desktop files are visible? [14:41] PR snapd#7760 closed: snap-bootstrap: write /run/mnt/ubuntu-data/var/lib/snapd/modeenv [14:47] pedronis, mvo: I'll check if there are any open bugs that are related to nested mimics and see if they have regression tests [14:49] PR snapd#7792 opened: snap-bootstrap: write /run/mnt/ubuntu-data/var/lib/snapd/modeenv [14:58] PR snapd#7671 closed: overlord/patch: normalize tracking channel in state [14:59] *gasp* i was supposed to wait for pstolowski :-( [14:59] pedronis: 7792 should also be easy hopefully [14:59] Chipaca: i'm finishing it atm.. [14:59] pstolowski: please tell me it's not on fire :-| [15:01] mvo: pedronis: #7788 is green [15:01] PR #7788: overlord/snapstate: pick up system defaults when seeding the snapd snap <⚠ Critical> [15:18] * cachio lunch [15:21] hi. cant launch chromium anymore, on debian: https://paste.debian.net/1118076/ [15:21] installed: 78.0.3904.108 (958) 160MB - [15:22] any ideas? thank you. [15:23] thresh: can you pastebin 'snap version' please? [15:24] who was it that was looking at the /sys/kernel/mm/transparent_hugepage/hpage_pmd_size thing [15:24] sure thing. there you go Chipaca: http://paste.debian.net/1118077/ [15:24] mvo: you do remember, offhand? [15:24] yeah, 42 on a 5.3 [15:25] there was somebody looking at this, but i don't remember what the conclusion was [15:26] jdstrand: is it expected that chromium tries to do things that would require hardware-observe? [15:26] Chipaca: I saw some chromium related stuff in some of recent jdstrand PRs [15:27] not sure it relates or not [15:27] maybe he can say [15:27] Chipaca: what things? [15:27] jdstrand: looks like it's trying to read /sys/kernel/mm/transparent_hugepage/hpage_pmd_size [15:27] as per the pastebin above [15:27] jdstrand: https://paste.debian.net/1118076/ [15:28] mvo: can you double check https://github.com/snapcore/snapd/pull/7773 please [15:28] PR #7773: cmd/snap-update-ns: fix overlapping, nested writable mimic handling

<⚠ Critical> [15:28] mvo: it has +2 and as soon as it's green I'd like to merge it and open a backport for 2.42.3 [15:29] Chipaca: seems like we need something added to PR 7779 [15:29] PR #7779: interfaces: misc updates for u2f-devices, browser-support, hardware-observe, et al [15:29] Chipaca: I'll do that in a bit [15:29] jdstrand: is there something thresh can do to get unblocked? [15:30] i'm guessing 'reboot into pre-5.3 kernel' would work but not be a nice way out [15:30] Chipaca: oh, actually, that is snap-update-ns.chromium [15:30] o/ [15:30] It's not a big deal. I use firefox mainly anyway. Just something that is probably best reported & fixed :) [15:30] thresh: <3 [15:31] jdstrand: mvo is looking at doing a 2.42.3, it would be good to know if we need things in it for this [15:32] I'll look into that access. one could modify /var/lib/snapd/apparmor/profiles/snap-update-ns.chromium [15:32] thresh: ^ [15:33] eg, add '/sys/kernel/mm/transparent_hugepage/hpage_pmd_size r,' then run 'sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap-update-ns.chromium' [15:33] that only works until snapd refreshes the policy of course (then you'd have to do it again) [15:33] pedronis: ack [15:35] jdstrand: thank you [15:35] jdstrand, huh. well that worked to remove the apparmor denial line from dmesg indeed. [15:35] but chromium still crashes with the same message [15:35] so maybe those are not related. [15:36] oSoMoN: ^ is that a 'you' thing? [15:37] thresh: at this point i'd say it's probably best to open a topic in forum.snapcraft.io, so it can be looked at async [15:37] right [15:37] either that or a good ol' bug :) [15:38] but forum is probably easier [15:38] I'm going to try a different user / clean profile etc [15:38] thresh, mind filing a bug at https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+filebug ? [15:38] PR snapd#7793 opened: devicestate: read modeenv early and store in devicestate [15:39] Chipaca, the chromium snap is a 'me' thing indeed [15:39] oSoMoN, I will, thanks! [15:39] PR snapd#7791 closed: snap-bootstrap: make cmdline parsing robust [15:39] oSoMoN: :) [15:39] zyga: sure, looking now [15:39] * Chipaca grabs a cuppa tea and tries to stop eating biscuits [15:40] I visited my mum on sunday and cam back with a metric ton of chocolate chip [15:40] send help [15:42] PR snapd#7773 closed: cmd/snap-update-ns: fix overlapping, nested writable mimic handling

<⚠ Critical> [15:42] Chipaca: send chocolate :) [15:43] Chipaca: we should take preorders on treats for Fra [15:43] Fra is three months away [15:43] Chipaca: with everyone taking trains we can bring anything :) [15:44] in fact, fra is two weeks before i run the lisbon half [15:44] so no treats! [15:47] mborzecki: I think we can merge 7788 - do you want to squash merge or do you prefer the commits and then we open a 2.42 version ? either way is fine with me. if squash I think it would be great if you could do it yourself so that you can decide on the commit message [15:49] mvo: https://github.com/snapcore/snapd/pull/7794 [15:49] * zyga goes for lunch [15:50] PR #7794: many: backport pull request #7773 from zyga/fix/lp-1852361 [15:50] PR snapd#7788 closed: overlord/snapstate: pick up system defaults when seeding the snapd snap <⚠ Critical> [15:50] PR snapd#7794 opened: many: backport pull request #7773 from zyga/fix/lp-1852361 [15:55] PR snapd#7795 opened: overlord/snapstate: pick up system defaults when seeding the snapd snap (2.42) [15:57] mvo: ^^ [15:58] zyga, hi, I just ran again into the issue with the maas snap reporting that the command is not there, when the snap is installed. this time it happened right after installing the snap from the store [15:58] zyga, any useful info I can gather ? [16:03] ackk: oh my :D [16:03] ackk: what what the exact error message? [16:04] $ maas [16:04] execv failed: No such file or directory [16:04] ackk: this is worrying, so it was not a "snap try" version [16:04] zyga, it's odd because every time I've seen it happen, it was just after I ran maas init [16:04] ackk: did you have a snap-try version before though? [16:04] and then installed from the store on top of that? [16:05] zyga, that's a good question, I'm not sure [16:05] mborzecki: \o/ [16:05] ackk: that might explain why [16:05] pedronis: you mentioned another potential thing we need for .3 ? [16:05] ackk: we actually landed a feature that might help [16:05] zyga, can I check somehow? [16:05] ackk: it fixes a whole class of issues [16:05] ackk: try edge when it builds please, [16:06] zyga, I would have an x1 dir if I had upgraded from try right? [16:06] yes [16:06] or x$something [16:06] zyga, then no, I only have one revision [16:06] /o\ [16:06] ackk: any ideas on how to reproduce are welcome [16:07] ackk: to finish that thought though: https://github.com/snapcore/snapd/pull/7773 is what I was referring to [16:07] PR #7773: cmd/snap-update-ns: fix overlapping, nested writable mimic handling

<⚠ Critical> [16:07] ackk: it introduces snap set core experimental.robust-mount-namespace-updates=true [16:07] ackk: which changes how mount namespace is updated [16:07] zyga, that's not yet in edge though? [16:07] ondra: ^ btw, it landed, we're going to be good for release soon [16:07] ackk: no, it's just in for a few minute [16:08] zyga, so, snapd edge ? [16:08] zyga cool, I will run edge on one of the devices and confirm it is updating ( and I will also give customer heads up) [16:08] zyga, and that snap set [16:08] zyga, is that valid for core18 as well? [16:09] ackk: snapd edge or core edge once it builds [16:09] ackk: it works across all versions [16:09] cool [16:10] pedronis: mvo: yeah, looks like we have a problem with subsequent install of core on a core18 device [16:10] thanks zyga and mborzecki [16:10] mborzecki: meh, do you have more details? [16:11] oh [16:11] mborzecki: so more fixes for 2.42.x required? [16:11] mvo: do you want to cherry-pick any of jdstrand's recent interface fixes? [16:11] mvo: I can cherry-pick them back to offload you [16:11] mvo: pedronis: https://paste.ubuntu.com/p/mpV9Nr8v9H/ [16:11] zyga: yes, think so [16:12] PR snapd#7761 closed: devicestate: make /var/lib/snapd/seed available in install mode [16:13] ijohnson: hey [16:13] ijohnson: I'd like to land https://github.com/snapcore/snapd/pull/7783 [16:13] PR #7783: osutil/mount: add {Unm,M}outFlagsToOpts helpers [16:13] ijohnson: I explained why the suggestion doesn't work (ordering) [16:14] sure let me take a look [16:14] ijohnson: I'm happy to iterate on ideas but I'd like to merge this and unblock another PR [16:14] ijohnson: so that the useful improved debug output from snap-update-ns doesn't get stuck/lost [16:16] PR snapd#7796 opened: tests/main/ubuntu-core-config-defaults-once: make sure configuration defaults are applied once