[05:58] morning [05:59] good morning mborzecki [05:59] mvo: hey [06:19] mvo: 2.46 branch needs cherry-picks with fixes for the tests [06:20] mborzecki: I think I will just merge master there [06:20] mvo: unless we just merge master [06:20] cool [06:20] mborzecki: I had hoped for some fixes from sergio [06:20] PR snapd#9191 closed: interfaces/{docker,kubernetes}-support: load overlay and support systemd cgroup driver [06:20] mborzecki: before I do that [06:20] mborzecki: thanks for merging this [06:20] mborzecki: also nice to see green PRs again [06:21] mvo: yeah, even if it's just temporary ;) [06:23] mborzecki: ha! you are such a pessimist :) it will be perfect from now on! [06:23] mvo: weird, this branch https://github.com/snapcore/snapd/pull/9196 is in the upstream repo instead of zyga-mbp's fork [06:23] PR #9196: osutil: add OpenExistingLockForReading [06:24] mborzecki: he probably made this by mistake :/ [06:25] mvo: wonder if there's some fine grained permission switches to prevent from pushing new branches unless they meet certain pattern (eg. release/..) [06:25] mborzecki: aha, I think so [06:35] PR snapd#9186 closed: interfaces: add vcio interface [06:50] good morning [06:51] wha'ts up? [06:51] oh did I push to origin? [06:51] * zyga-mbp reads backlog while eating breakfast cereals on the side [06:52] interesting how this failed, wonder why [06:55] PR snapd#9196 closed: osutil: add OpenExistingLockForReading === amurray is now known as amurray` [07:00] reopened from my fork === amurray` is now known as amurray [07:00] PR snapd#9197 opened: osutil: add OpenExistingLockForReading [07:01] morning [07:01] good morning pstolowski [07:03] hey Pawel [07:05] PR snapd#9198 opened: features: add HiddenSnapFolder feature flag [07:21] pstolowski: zyga-mbp: hey [07:35] * zyga-mbp reboots for upgrades [07:42] stgraber: hey, do you think you could have a quick look at lp 1892568 ? it's about a nested lxd test we run that is failing on 16.04 and we need advise if it's worth keeping this test. I can trivial reproduce this if you need any debug from me [07:42] zyga: lp 1892568 might also be interessting for you, it has my findings about the nested lxd issue on 16.04 [07:47] * mvo takes a short break [07:48] sourcing nested.sh in the debug shell sets -e, so a first non-successful command ends your shell :/ [07:50] PR snapd#9161 closed: kernel: add kernel.Validate() === zyga is now known as zyga-mbp === 895AAPCTC is now known as zyga === zyga is now known as zyga-x240 [09:11] * zyga-x240 goes afk for 30 min [10:04] mborzecki: I think/hope that I have kernel-refs in gadget updates working now but writing the spread test for it is such a chore [10:05] mvo: hahah, i feel your pain [10:05] mvo: have you looked at the gadget update spread tests? [10:06] PR snapd#9199 opened: snapstate: installSizeInfo helper that calculates total size of snaps and their prerequisites [10:14] re [10:14] man, lucy misses mom badly today [10:15] * zyga-mbp hugs mvo and mborzecki for writing excellent tests for hard cases [10:21] PR snapd#9197 closed: osutil: add OpenExistingLockForReading [10:23] mvo: back? [10:23] meeting [10:23] mvo: so about https://launchpad.net/bugs/1892568 - I get 404 [10:23] what am I missing? [10:24] https://github.com/snapcore/snapd/pull/9200 <- super simple [10:24] PR #9200: runinhibit: open the lock file in read-only mode in IsLocked [10:26] PR snapd#9200 opened: runinhibit: open the lock file in read-only mode in IsLocked [11:36] * zyga-x240 small break [11:45] mvo, hi, the PR 116 on core has been merged today right? [11:45] PR #116: Feature/mount snaps [11:45] it is not included in core on edge yet, right? [11:55] cachio: yes, this morning [11:55] cachio: quite possible, I can trigger a rebiuld for core [11:58] zyga-mbp: https://bugs.launchpad.net/snapd/+bug/1892468 [11:58] Bug #1892468: Nested dbus test useful? [11:58] mvo, yes please [11:59] so I can test the fix for the failover test [11:59] mvo: thanks, weird that the other link did not work [11:59] cachio: I triggered a build now [11:59] zyga-x240: maybe I did a typo? [11:59] mvo, thanks [12:03] i see selinux-lxd failures on all selinux systems, not sure if this is related to my PR; has anyone seen it on other branches? [12:07] pstolowski: got logs? [12:09] mborzecki: https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/11690/signedlogcontent/67?urlExpires=2020-08-21T12%3A10%3A26.5605826Z&urlSigningMethod=HMACV1&urlSignature=vfCuP7XdBACxjXIQPWWvoGYGTxdhjW9ptv9tIA%2BbDPY%3D [12:09] (PR #9084) [12:09] PR #9084: o/snapstate: check disk space before creating automatic snapshot on remove (3/N) [12:23] 2020-08-21T11:51:03.3746915Z type=AVC msg=audit(1598010641.669:5223): avc: denied { getattr } for pid=131226 comm="snapd" path="/var/snap/lxd/common/ns/shmounts" dev="nsfs" ino=4026532238 scontext=system_u:system_r:snappy_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=file permissive=1 <- [12:23] oh, pawel is gone [12:41] PR snapd#9201 opened: [RFC] boot: observe update & rollback of trusted assets [12:48] OMG [12:48] * zyga-x240 learned something fantastically terrible [12:48] O M G [12:48] zyga-x240: hm? [12:49] you will never believe how insanely surprising shell is [12:49] set -e is ignored in some cases [12:49] and man, there's a super super super silly case [12:49] you can have pages of shell code [12:49] you can set -e [12:49] you can set -e 10 times in a row [12:49] run false [12:49] then true [12:49] and that doesn't stop [12:49] set -e [12:50] echo "LOL shell" [12:50] false [12:50] echo "I'm still here" [12:50] true [12:50] how you ask? [12:50] simple [12:50] just _at any level above_ [12:50] put that in a context where set -e is ignored [12:50] such as [12:50] if foo; then ... fi [12:50] if that set -e and echo and false and true was in a "foo" function [12:50] set -e is irrelevant [12:51] I'm still here prints [12:51] and $? is 0 [12:51] because that's a shell list [12:51] the return value is the return value of the last element [12:51] and the fact that this is a function which sets -e is irrelevant [12:51] PR snapd#9202 opened: tests/nested/core20/tpm: verify trusted boot assets tracking [12:52] because the context of the function call (at any level above!) is one where set -e is ignored [12:52] like ! foo [12:52] if foo [12:52] foo | bar [12:52] or other similar [12:52] mvo: 404 [12:52] how f**** crazy is that [12:52] stgraber: try https://bugs.launchpad.net/snapd/+bug/1892468 [12:52] Bug #1892468: Nested lxd test useful? [12:52] mborzecki: surprise [12:53] zyga-x240: it's friday and i think i'm a bit slow already, do you have an example? [12:53] mborzecki: yeah, I have a very complex one [12:53] let me try to make a simple script now that I get it [12:53] and the manual documents this [12:53] but doesn't connect the dot [12:54] between set -e ignored [12:54] and _any parent call_ set -e ignored [12:59] https://paste.ubuntu.com/p/QDh6JwqCW9/ [12:59] mvo: ^ tell me how many bugs we have left? [12:59] mvo: in shell [13:07] PR snapd#9172 closed: tests: update spread test for unknown plug/slot with snapctl is-connected [13:32] mvo: onto that test now [13:32] ogra_: https://paste.ubuntu.com/p/7njRQNhRZz/ [13:32] ogra_: check this out [13:32] zyga-x240: \o/ *thank you* [13:32] zyga-x240: also for sharing that snippet [13:33] zyga-x240: I wonder if shellcheck should warn about "if func()", I would argue it should [13:33] * ijohnson afk little bit [13:34] mvo: I don't know, it's such a misfeature [13:34] maybe it should if set -e is in effect [13:34] I feat it may be very noisy [13:34] zyga-x240: I think this is worth a bugreport [13:34] zyga-x240: not sure, do you think if func() is common? [13:34] mvo: yeah, I'll file one over weekend [13:34] I want to write a blog post about this as well [13:34] zyga-x240: amen [13:34] I'll refer to bugs in shellcheck and the example [13:34] zyga-x240: please do! [13:35] after being at canonical for a decade I'll write a mail to tech@ ... [13:42] mborzecki: can you re-paste what you found about that selinux denial before i dropped from irc? [13:42] mvo: test written, just waiting to see if it passes [13:45] zyga-x240: thanks so much [13:46] zyga-x240: I'm sure for some people it's not a surprise [13:46] mvo: maybe we should hire them? [13:47] zyga-x240: maybe we should write less shell [14:07] pstolowski: zyga-x240 belived that snapd traverses too far and trespassed to locations managed by lxd [14:07] mborzecki: not quite, it's not like it's not our space [14:07] it's just that there are weird and wonderful objects there [14:07] that are not just files [14:07] and selinux doesn't like that [14:07] right [14:08] pstolowski: but i'd ahve to look at the PR again, maybe there's more to it [14:08] zyga-x240, is this enough for the failover test on a core image https://paste.ubuntu.com/p/dTmddSv3Cw/ ? [14:09] cachio: not quite, remember that you cannot restart logind [14:09] you must reboot [14:09] yes [14:09] well I try both things [14:09] well, we tried already [14:10] you cannot restart logind safely until systemd 246 [14:10] but still see [14:10] + systemctl --user daemon-reload [14:10] Failed to connect to bus: No such file or directory [14:10] cachio: I'd have to review your changes, [14:10] didnt push it yet [14:10] trying manually first [14:11] but that change that manually did is not enough to make the tests work [14:12] change what manually? [14:12] https://paste.ubuntu.com/p/dTmddSv3Cw/ [14:12] that [14:12] cachio: sorry that's not correct [14:12] but with reboot [14:12] I see [14:13] can you tell me loginctl list-sessions after rebooting? [14:13] ok [14:14] cachio: one thing that may be at play, is that root has no session [14:14] since external target uses external user [14:14] so disable-linger root may still, really, just shut stuff down [14:14] but prepare enables linger [14:14] which should make that irrelevant [14:15] PR snapcraft#3256 opened: repo: consolidate BaseRepo and Ubuntu & reorganize package [14:15] zyga-x240, https://paste.ubuntu.com/p/yp37m39TsD/ [14:16] is the external user logged in? [14:17] no [14:18] zyga-x240, https://paste.ubuntu.com/p/d2Wcn7JJdh/ [14:18] this is with the external user logged in [14:18] yes [14:18] ok [14:18] well, this much looks sensible [14:19] perhaps the problem is that here we use external instead of root to run the tests [14:20] zyga-x240, could be that affect? [14:20] cachio: well, as I said above [14:20] it should not matter [14:20] so what fails now? [14:20] which of the tests beaks [14:20] fails the same [14:20] spread -debug external:ubuntu-core-16-64:tests/core/core-to-snapd-failover16 [14:20] this fails [14:20] mvo: updated https://github.com/snapcore/snapd/pull/9200 [14:20] PR #9200: runinhibit: open the lock file in read-only mode in IsLocked [14:20] during restore [14:21] cachio: how does it fail? [14:21] zyga-x240, https://paste.ubuntu.com/p/HGjKDsXpKk/ [14:22] yeah, I think that test needs tests.session -u root prepare / restore [14:22] try adding that (prepare in 1st line, restore in last line) [14:22] and run that test alone [14:22] sure [14:23] it should pass then, the problem is that systemctl --user really assumes you have root [14:23] that's implicit for our tests [14:23] actually [14:23] cachio: wait [14:23] cachio: so in external mode tests run as non-root entirely? [14:23] or run as root somehow via sudo or something like that? [14:23] they use external user [14:23] they are not root [14:24] mvo: https://github.com/snapcore/snapd/pull/9200/files [14:24] PR #9200: runinhibit: open the lock file in read-only mode in IsLocked [14:24] cachio: hmmm [14:24] cachio: then that will not suffice [14:24] cachio: we need a bigger hammer [14:24] cachio: and in addition [14:24] cachio: that test should *not* fail anymore [14:24] + systemctl --user daemon-reload [14:24] if this runs as external [14:24] then when it fails, please list sessions and do systemctl status [14:24] (just without logging in again) [14:25] let's examine what is going on [14:25] zyga-x240, ok [14:30] PR snapcraft#3257 opened: plugins v2: use repo.Repo not repo.Ubuntu in colcon [14:31] zyga-x240, https://paste.ubuntu.com/p/RMb9qPVBFp/ [14:31] so it connects using external [14:31] but then uses root [14:31] as for other backends [14:32] │ │ ├─3480 sudo -i /bin/bash /tmp/tmp.zwRt33mi04 [14:32] but it uses sudo [14:32] cachio: I'm about to EOD [14:32] this needs more oomph [14:32] sudo is incorrect [14:32] ok [14:32] lets continue on Mondya [14:32] ok [14:33] enjoy the weekend [14:33] * zyga-x240 takes a break and heads for PT [14:33] cachio: thank you, you too [14:33] cachio: we need to either not use sudo [14:33] cachio: or give root a real session [14:33] cachio: not a mismatch of the two [14:33] zyga-x240: github meeting? or will you miss that? [14:34] ijohnson: I'll pass [14:34] k [14:34] I need to prep for PT [14:34] I marked myself as optional [14:34] and I'm tired [14:35] cachio is that sudo from spread or from us? [14:35] I think spread does it [14:35] ok [14:36] let me check [14:36] zyga-x240, I see this in spread [14:36] func (c *Client) sudo() string { [14:36] if c.config.User == "root" { [14:36] return "" [14:36] } [14:36] return "sudo -i " [14:36] } [14:36] in the client [14:44] we will need some changes then [14:44] it's not broken per se [14:44] but it's not the same as logging in as root [14:44] so bummer [14:46] cachio: a few comments to nested PR [14:47] pstolowski, thanks! [14:47] zyga-x240, nice, tanks for the help, so lets continue on Monday [15:03] * cachio lunch [16:09] jdstrand: hey in #9191 I noticed that you didn't add overlay as a kernel module for the kubernetes-support interface like you did with docker, is there a reason you made it docker-support only? [16:09] PR #9191: interfaces/{docker,kubernetes}-support: load overlay and support systemd cgroup driver [16:10] jdstrand: also not sure if you saw my ping about a possible typo in #9188 [16:10] PR #9188: interfaces: misc policy updates xlvi [16:55] PR snapcraft#3257 closed: plugins v2: use repo.Repo not repo.Ubuntu in colcon [17:06] man it feels good to have green PR's again that we can just merge [17:07] PR snapd#9190 closed: cmd/s-b/initramfs-mounts: make recover -> run mode transition automatic [17:37] * zyga is back from PT [17:37] ijohnson: it does feel good :) [17:38] PR snapd#9200 closed: runinhibit: open the lock file in read-only mode in IsLocked [18:01] PR snapcraft#3258 opened: schema/snapcraft.json: allow type: os with build-base [18:32] ijohnson, hey [18:32] for some raeson nested tests are fialing to boot on uc16 and 18 after merging with master [18:32] * ijohnson is meeting, will look in a bit [18:33] ijohnson, sure, thanks [19:07] cachio: so what's failing? [19:07] cachio: and is it 100% reproducible ? [19:07] I can't login [19:07] it is like hte user is not created [19:07] cachio: for all nested spread tests ? [19:08] not sure if all but many [19:08] I ran [19:08] google-nested:ubuntu-16.04-64:tests/nested/core/core-snap-refresh-on-core [19:08] and could reproduce it [19:08] still have a shell [19:09] cachio: ok I will take a look [19:09] the cloud init info is copied correctly [19:09] cachio: usually what I have taken to doing to reproduce these things is to shut down the nested-vm systemd service, then start the VM manually with qemu, but using -serial mon:stdio then create a user via console-conf to login to the VM and poke around to see what ran/didn't run, etc. [19:10] cachio: it's running now [19:10] let's see what we get [19:10] yes [19:10] ip 34.74.111.154 [19:28] ijohnson: I'm not really here but it is because overlay is needed for docker and containerd. k8s can be configured for either and therefore must also plugs docker-support [19:29] jdstrand: ack, makes sense [20:02] cachio: I opened #9203 to fix that [20:02] PR #9203: tests/lib/nested.sh: fix partition typo, unmount the image on uc20 too <⚠ Critical> [20:03] PR snapd#9203 opened: tests/lib/nested.sh: fix partition typo, unmount the image on uc20 too <⚠ Critical> [20:08]