[01:16] PR snapcraft#3064 closed: cmake v2 plugin: rename configflags to cmake-parameters [01:16] PR snapcraft#3066 closed: autotools v2 plugin: rename configflags to autotools-configure-parame… [02:16] PR snapcraft#3068 opened: cmake v2 plugin: default to -DCMAKE_BUILD_TYPE=Release [05:48] morning [06:08] o/ [06:09] hey zyga [06:09] hey mvo [06:14] * zyga needs coffee [06:14] PR snapd#8526 closed: image: improve/adjust DownloadSnap doc comment [06:15] PR snapd#8532 opened: tests: install new snapd deb into preseed image [06:17] zyga: mvo: morning guys [06:17] zyga: yeah, coffee sounds like a plan [06:18] * zyga was unable to sleep last night, collapsed after 2AM [06:18] coffee iv then ;) [06:18] s/coffee/caffeine/ [06:19] in the meantime: https://forum.snapcraft.io/t/centos-8-snapd-install-of-authy-doesnt-show-up-in-applications-folder/16684/5 despite all the tricks we pull, looks like XDG_DATA_DIRS is still not in the env sometimes [06:20] funnily enough, it's in the terminal, but that's probably a result of /etc/profile.d bits [06:21] mvo: do you want to take a look at the latest changes in #8487 before i land it? [06:21] PR #8487: gadget, cmd/snap-bootstrap: MBR schema support [06:22] mborzecki: let me quickly look [06:24] PR snapd#8487 closed: gadget, cmd/snap-bootstrap: MBR schema support [06:26] yay! [06:33] PR snapd#8529 closed: cmd/snap-bootstrap/initramfs-mounts: support EnvRefExtractedKernelBootloader's <⚠ Critical> [06:42] re [06:43] 0.5 coffee [06:50] PR core20#46 opened: static: add new handle_writable_defaults() to handle-writable-paths [06:59] morning [07:03] pstolowski: good morning - just fyi, i pushed https://github.com/snapcore/core20/pull/46 [07:03] PR core20#46: static: add new handle_writable_defaults() to handle-writable-paths [07:04] mvo: great, thank you [07:05] * pstolowski is going to look at the 20.04 preseed test hang issue in a few moments [07:09] pstolowski: the issue is that the 20.04 cloud images have snapd snap, not the core snap now [07:10] but I'm not even sure we need the hack or not (do they have 2.44 ? now) [07:16] looking [07:21] pstolowski, pedronis: I fixed that locally [07:21] I'll send an update to my PR in a moment [07:23] zyga: yay! [07:23] fruit of last night sleeplessness [07:26] pstolowski: one interesting thing that changed is that the revisions are no longer unset (in the task log) [07:26] is that expected? [07:26] or is that a consequence of just using asserted blobs [07:27] zyga: if the hack is removed and we no longer taint the snap then yes [07:27] thanks, I was wondering about that [07:28] zyga: if the test was failing because of that then i suppose hanging was (again) caused by restore/qemu-nbd stuff [07:28] pstolowski: actually I never got to the bottom of why it hanged [07:28] pstolowski: I disabled the call to setup_preseeding and adjusted the MATCH rules below to reflect new reality [07:28] weird [07:29] interactively it would not hang [07:30] zyga: yes the hang is misterious and i couldn't figure it out before, it's the 3rd time it bites [07:31] zyga: it is something in the qemu cleanup [07:31] hmmm [07:31] I think I see what it may have hanged [07:31] root 20047 0.7 0.2 466144 8244 ? Ssl 07:31 0:00 qemu-nbd -c /dev/nbd0 cloudimg.img [07:31] this process must die [07:32] zyga: as you say interactively it works [07:32] right but this process must die :) [07:32] zyga: but qemu-nbd -d should do it afaiu? [07:32] yes [07:32] mvo: hi, I reviewed Ian's #8512 [07:32] PR #8512: boot/bootstate20: add EnvRefExtractedKernelBootloader bootstate20 implementation <⚠ Critical> [07:32] oh drat, I passed -shell again [07:34] pstolowski: I figured it out [07:34] pstolowski: such a simple mistake :DDDD [07:34] \o/ [07:34] /o\ [07:34] look at restore [07:34] what is missing? [07:34] the test just failed [07:34] we are running restore [07:37] reset test is equally broken [07:38] yeah i'm not surprised, they reuse same patterns [07:38] i probably spent too much on these tests to see the problem :/ [07:38] what is it? [07:40] https://github.com/snapcore/snapd/pull/8528/files#diff-c6e01e719c875585769d5445f3982ecbR21 [07:40] restore ran on failure [07:40] PR #8528: tests: naive fix for preseeding failure [07:40] zyga: noooooooo [07:40] eh sorry [07:40] https://github.com/snapcore/snapd/pull/8528/files#diff-c6e01e719c875585769d5445f3982ecbR31 [07:41] and ran umount_ubuntu_image without sourcing the definition [07:41] I made prepare/restore handle the mounting [07:41] drat [07:42] so it was that process :) [07:42] it was the only possibility [07:42] zyga: thank you! [07:42] happy insomnia was productive [07:44] manual/preseed may need a tweak too [07:44] pstolowski: thanks, I'll look [07:46] done [07:50] mvo: I did a pass on #8530 as well, some questions there [07:50] PR #8530: boot: enable makeBootable20RunMode for EnvRefExtractedKernel bootloaders <⚠ Critical> [07:51] zyga: let me know when i should review [07:51] pstolowski: sure [07:51] pstolowski: I'll open the draft soon [07:52] pedronis: thanks, looking [07:55] zyga: we need to be careful about 19.10 in these tests, it may not be updated the same way [07:56] pstolowski: yeah, I'm running the more complete set to see what fails [08:18] mvo: mmh, I didn't notice #8531, I made a comment there, we also to think how to keep the info around for run mode [08:18] PR #8531: secboot,cmd/snap-bootstrap: add model to pcr protection profile [08:27] zyga: jamesh proposed a PR for preseed test as well [08:28] pstolowski: mine was mainly just exploring to see if that might fix it [08:29] pstolowski: one of my other PRs changes some packaging (adding some directories), which caused the preseed-reset test to fail since the image had an old snapd in it [08:30] jamesh: i see. yes, that test is very picky about snapd directories, i updated it recently for 19.10 vs 20.04 [08:31] pstolowski: the comments about certain images containing too old versions made me think it would be better to test the image with the snapd version under test installed. [08:32] pstolowski: although that resulted in some complaints about some files under /var/cache/apparmor [08:37] jamesh: this is certainly a fair point, otoh it's good to test that we support existing ubuntu images (once they have recent enough snapd but lag a little bit). it's the transition period that's a little annoying but it should stabilize soon [08:39] pstolowski: it seems like the kind of thing that will destabilise any time snapd changes sufficiently though. [08:44] jamesh: maybe i need to rethink this test; maybe it could look into debian dirs to see what's actually expected atfer a reset [08:45] pstolowski: would that be the debian dir of the snapd under test or the old snapd in the image? [08:51] pstolowski: I have one more tweak that addresses the hang [08:51] jamesh: hmm that's a good question. current reset code is "cheating" and has a simplified view of that and resets for the snapd under test [08:52] pstolowski: that's probably good enough in practice though: a system you reset is eventually going to get the new snapd anyway [09:03] jamesh: true, and the key point is that we don't leave files around (empty directories should be harmless) [09:11] pstolowski: hopefully this will shed some light as to why it was hanging and how to address such problems in general [09:11] pstolowski: (earlier I spoke too soon, the small mistake in the restore was not everything) [09:11] zyga: mhm [09:12] zyga: thank you for looking into it. a fresh look at these tests helps [09:17] pstolowski: it passed now [09:18] pstolowski: I'll run the rest [09:18] nice [09:22] I have a small problem at home [09:22] my internet quota for the month is almost gone [09:23] I have a backup but I need to find the extra modem and connect everything :/ [09:28] pstolowski: you may find this interesting https://github.com/snapcore/snapd/pull/8528/files#diff-2a7ec7462b6ff35150bfb39e2b01fc83R18 [09:28] PR #8528: tests: fix for pre-seeding failures [09:29] zyga: woot. it affects ssh? [09:29] ? [09:29] the other way around [09:29] maybe I should rewrite the comment [09:29] and there's a typo there [09:29] blargh :) [09:30] zyga: do you know why is it a problem? [09:37] yes [09:38] one sec, cannot typpe [09:41] pstolowski: because each stuff spread runs is done so in a new ssh session: https://github.com/snapcore/spread/blob/master/spread/client.go#L337 [09:42] that session will terminate when the remote side descriptors are closed [09:42] but they are not closed because they get inherited into "background" tasks [09:42] systemd-run handles descriptors and closes anything that is not explicitly forwarded [09:50] pstolowski: please look https://github.com/snapcore/snapd/pull/8528 [09:50] PR #8528: tests: fix for pre-seeding failures [09:51] zyga: i see, thank you [10:01] jamesh: I'll push the change you requested along with the shellcheck quote fix in a moment, I just want to see it pass for real [10:10] jamesh: pushed [10:21] mborzecki: there's a failure in selinux lxd test on centos8 [10:22] mborzecki: IIRC it is something new [10:22] I collected some samples last time I saw this [10:22] type=SYSCALL msg=audit(1587118309.956:1881): arch=c000003e syscall=9 success=yes exit=139877378064384 a0=0 a1=4caf68 a2=5 a3=802 items=0 ppid=1371 pid=2743 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="snap-update-ns" exe="/usr/libexec/snapd/snap-update-ns" subj=system_u:system_r:snappy_mount_t:s0 key=(null) [10:22] type=AVC msg=audit(1587118309.956:1881): avc: denied { execute } for pid=2743 comm="snap-update-ns" path="/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" dev="loop2" ino=5726 scontext=system_u:system_r:snappy_mount_t:s0 tcontext=system_u:object_r:snappy_snap_t:s0 tclass=file permissive=1 [10:22] type=AVC msg=audit(1587118309.956:1881): avc: denied { map } for pid=2743 comm="snap-update-ns" path="/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" dev="loop2" ino=5726 scontext=system_u:system_r:snappy_mount_t:s0 tcontext=system_u:object_r:snappy_snap_t:s0 tclass=file permissive=1 [10:23] mmap? [10:27] hmm probably wonder why libcrypto got pulled in [10:28] zyga: iirc snappy_mount_t should have all the permissions to execute/mmap/open from snappy_snap_t [10:28] zyga: we don't expect it to be statically compiled [10:28] mborzecki: no idea, maybe more nsswitch? [10:33] zyga: do you have shell? can you ldd /usr/libexec/snapd/snap-update-ns? [10:33] no I don't -- that was last week [10:34] it's not a priority, centos8 is in the unstable group [10:44] hm so crazy idea abou tthe native endian check [10:45] zyga: how about we have a dbustest2_amd64.go with `func init() { nativeEndian = binary.LittleEndian } dbustest2_other.go that just sets nativeendian = whatever? [10:51] mborzecki: happy to in a follow up [10:51] I'm trying to get the big one merged [10:51] mborzecki: I was thinking that endianness could be a package somewhere [10:51] zyga: nothing urgent, you could just leave it as it is now [10:51] as its not just this place [10:54] mvo: one more annoynace... https://github.com/snapcore/core-build/pull/59/files#r412081969 [10:55] PR core-build#59: initramfs: add new handle_writable_defaults() [10:55] mvo: it's shocking how easy it is to get things wrong in shell [10:56] pstolowski: thnaks! [10:56] * zyga focuses on review for jamesh [10:57] pstolowski: cp "$srcpath/"* [10:57] pstolowski: yeah, plus missing tests :/ I think this needs a test, I will have a look [10:57] zyga: right [10:57] PR core20#46 closed: static: add new handle_writable_defaults() to handle-writable-paths [11:08] pstolowski: the preseed test has different behavior in 19.10 and 20.04 [11:08] mborzecki: not there is really a question whether it should be boot/uboot or just uboot, we don't have boot/uboot in boot or do we? do we want boot/boot/uboot at runtime? [11:08] because of core / snapd [11:09] pstolowski: I kind of switched gears now so if you want to attack that feel free [11:10] zyga: yes, this is expected (if you mean the MATCH parts), that's what I tried to convey before [11:10] I forgot to run the suite on 19.10 again [11:10] oh well :) [11:10] zyga: sure i can take it from you and fix that [11:10] pstolowski: thanks! [11:11] zyga: ty! [11:11] pstolowski: I guess it's just the match and count that has to change [11:11] zyga: have you pushed everything? [11:11] pstolowski: yes [11:11] zyga: yes, i think we need big if/else there and keep the old matches [11:11] indeed [11:11] zyga: okay, i need lunch soon, will get to it after standup i suppose [11:12] where do we assign bugs to core, LP? [11:13] actually github [11:13] because $reasons [11:13] but also on lp [11:13] so dunno [11:19] mhm [11:22] pedronis: so with recovery|noslashboot: true you'd rather use `uboot` as basedir? [11:23] mborzecki: maybe, I wonder what .scr expects and what we do in ubuntu-boot [11:23] pedronis: if i'm reading this right, ubuntu-seed has boot/uboot, but ubuntu-boot will be mounted at /boot, so we'll ahve /boot/boot/uboot? [11:23] pedronis: yes, exactly, something seems off [11:24] but I don't know what the .scr does exactly [11:24] pedronis:the bind mount is /run/mnt/ubuntu-boot -> /boot right? [11:24] that's what I would expect yes [11:24] I'm double checking how this looks on amd64 atm [11:26] pedronis: there's EFI/ubuntu iirc [11:32] jamesh: https://github.com/snapcore/snapd/pull/5822#pullrequestreview-395477722 [11:32] PR #5822: wrappers: allow user mode systemd daemons <:birthday:> [11:34] PR snapd#8527 closed: seed: clearer errors for missing essential snapd or core snap [11:42] mborzecki: FYI https://github.com/snapcore/snapd/pull/7825#discussion_r412110567 [11:42] PR #7825: many: use transient scope for tracking apps and hooks [12:02] mvo: mborzecki: google:ubuntu-core-20-64:tests/core/basic20 from master is getting stuck on me, it's printing Reboot on apr211119-149277 (google:ubuntu-core-20-64:tests/core/) is taking a while... since 20 mins+ [12:03] pedronis: probably failed in initramfs/grub [12:03] did we break something? [12:03] it'd be great to ahve a way to store those artifacts [12:03] mborzecki: did you say artifacts? :) [12:09] mborzecki: we have a new kernel from today in edge and beta fwiw [12:10] ayy [12:10] initramfs changes? :P [12:11] i'm checking out rhel8/centos for popey, i'll build and image and check what's going on after [12:19] PR snapd#8533 opened: [WIP] image, tests: core18 early config test [12:23] PR snapcraft#2911 closed: [experimental] package-management repository configuration [12:29] PR snapd#8534 opened: tests: 16.04 and 18.04 now have mediating pulseaudio (again) [12:49] google:ubuntu-16.04-64:tests/main/interfaces-audio-playback-record keeps failing :/ [12:50] aha [12:50] morning folks [12:50] I think I see why [12:50] https://github.com/snapcore/snapd/pull/8534#pullrequestreview-397296420 [12:50] does #8534 by jdstrand relates? [12:50] hey ian :) [12:50] zyga: hey [12:50] PR #8534: tests: 16.04 and 18.04 now have mediating pulseaudio (again) [12:50] pedronis: indeed! [12:51] anyway everything is terrible [12:51] pedronis: I think people are great, despite the fact we are all stressed and the release is looming [12:52] what a great way to start my day [12:52] :-D [12:52] mvo: https://github.com/snapcore/snapd/pull/8534 is important, please get it into any . releases you may need [12:52] PR #8534: tests: 16.04 and 18.04 now have mediating pulseaudio (again) [12:52] ijohnson: hi, google:ubuntu-core-20-64:tests/core/basic20 doesn't seem to work on master atm [12:52] there was a new kernel [12:53] I'm trying to run it manually and it doesn't get out of reboot [12:53] ah [12:53] pedronis: I'll look into it [12:53] I don't know what the problem is, but I have a hunch it might be related to the initrd things from yesterday [12:58] zyga: oh, ok [13:28] cachio: add a cron timer to run tests on master once a day [13:28] cachio: it's better than nothing and costs very little [13:29] zyga, we can run using spread cron as welll [13:29] cachio: actions are better [13:29] cachio: they update master status [13:30] cachio: they notify everyone automatically [13:30] cachio: and they are managed in the repository [13:30] zyga, ok [13:30] cachio: you can then see it the status alongside each commit that was tested [13:31] mvo: can you please override https://github.com/snapcore/snapd/pull/8534 [13:31] PR #8534: tests: 16.04 and 18.04 now have mediating pulseaudio (again) [13:32] zyga, ok [13:32] I'll add that so we have runs on master [13:41] PR snapd#8534 closed: tests: 16.04 and 18.04 now have mediating pulseaudio (again) [14:26] mborzecki: I tweaked the code, it's pretty interesting how everything can fail in various cases. I didn't push yet but I think this is "the" way [14:26] mborzecki: the thing I learned is that we may have a session bus [14:27] mborzecki: but it may be entirely malfunctioning [14:27] mborzecki: because socket activation of dbus is not always going to work [14:27] mborzecki: because some systems package that separately [14:27] :) [14:27] mborzecki: so I tweaked the logic to cope with "late" failure, after we get the connection, it may turn out to be just "bad" [14:27] mborzecki: I treat this the same as session bus being unavailable now, for uid == 0 go to system bus [14:28] in the end some things got simplified and all error and debug handling is in once spot [14:28] mvo: I need to run an errand soon, going to visit my parents to pick up the LTE modem they are not using, I ran out of my 1TB quota /o\ [14:34] I don't know what's more mind-boggling, that you have a 1TB quota on LTE or that you exhausted it 😉 [14:43] zyga: ok, thanks for letting me know [14:44] PR snapcraft#3069 opened: make v2 plugin: make use of make-parameters [14:59] PR snapcraft#3068 closed: cmake v2 plugin: default to -DCMAKE_BUILD_TYPE=Release [15:00] mvo: up and running again [15:01] cmatsuoka: isn't today supposed to be your day off :-P [15:01] cmatsuoka: great! [15:01] cmatsuoka: welcome back [15:01] ijohnson: I don't have many places to go actually [15:01] * cachio lunch [15:01] cmatsuoka: haha fair enough [15:01] but it's possible that I could be a bit slower today [15:08] * zyga is AFK [15:10] #8512 needs a 2nd review [15:10] PR #8512: boot/bootstate20: add EnvRefExtractedKernelBootloader bootstate20 implementation <⚠ Critical> [15:34] anybody else seeing github 500s? [15:34] launchpad.net is perfectly stable ;) [15:37] ijohnson: is it the roadrunner cliff fall error page? [15:37] cmatsuoka: yes [15:37] ijohnson: saw it a couple of days ago but not today [15:38] ijohnson: ssame error, also pushing is very unhappy [15:39] yes frustrating [15:39] github is down? [15:39] I was trying to push a patch I wrote for libzt while in the car [15:39] and man [15:39] I dunno if it's down but it's real slow and not working well [15:39] interestingly I get new IP addresses for each git push [15:39] maybe they are scrambling to deploy [15:40] "minor service outage" [15:40] oh well [15:40] * zyga looks through the window instead [15:40] maybe with the lockdowns everyone got bored and started to use github [15:41] maybe they did their planned outage testing but nobody was home ;) [15:42] i'm getting weird failures with GC ^on spread prepare [15:42] the day that azure, gce and aws died ;) [15:42] :( [15:47] ijohnson: unicorn error page for me now [15:50] yeah I can't get to any real page now, it's just the cliff or a unicorn [15:51] or a clifficorn [15:52] that would be scary! [16:00] anyone with rhel or centos to try https://bugs.launchpad.net/snapd/+bug/1872368 (but docker may play a role in this)? [16:00] Bug #1872368: Cannot enable snapd.socket in RHEL 7.8 [16:01] pstolowski: if they are running inside docker then this is 100% not our bug to fix [16:06] ijohnson: thanks for commenting on it! [16:10] PR # closed: core-build#11, core-build#22, core-build#26, core-build#37, core-build#51, core-build#58, core-build#59 [16:20] Issue # opened: core18#56, core18#86, core18#89, core18#117, core18#129, core18#137, core18#142, core18#145, core18#151 [16:20] PR # opened: core18#43, core18#72, core18#90, core18#126, core18#134, core18#144, core18#146, core18#148 [16:20] PR # closed: snapd#5822, snapd#6258, snapd#7129, snapd#7168, snapd#7205, snapd#7414, snapd#7417, snapd#7436, snapd#7570, snapd#7586, snapd#7590, snapd#7603, snapd#7614, snapd#7700, snapd#7728, snapd#7825, snapd#7869, snapd#7900, snapd#7927, snapd#7982, snapd#8079, snapd#8123, snapd#8143, [16:20]