| mup | PR snapd#7855 opened: snap-confine: revert suppress noisy classic snap file_inherit denials <Simple 😃> <⚠Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7855> | 01:04 |
|---|---|---|
| mup | PR snapcraft#2833 opened: Arnatious/remove rospack workaround <Created by Arnatious> <https://github.com/snapcore/snapcraft/pull/2833> | 01:45 |
| mup | PR snapd#7856 opened: snap-confine: revert, with comment, explicit unix deny for nested lxd <Simple 😃> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7856> | 02:01 |
| mup | PR snapd#7857 opened: tests: update google ubuntu 16.04-64 expected host mount ns <Simple 😃> <⚠Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7857> | 04:41 |
| mup | PR snapd#7858 opened: tests: add nested-lxd test to confirm lxd inside lxd works <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7858> | 04:46 |
| mup | PR snapd#7855 closed: snap-confine: revert suppress noisy classic snap file_inherit denials <Simple 😃> <⚠Critical> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7855> | 04:49 |
| mborzecki | morning | 06:30 |
| mup | PR snapd#7846 closed: devicestate: add missing test for failing task setup-run-system <Simple 😃> <UC20> <Created by mvo5> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7846> | 06:59 |
| mborzecki | mount-ns test failing on google? | 07:04 |
| zyga | Gooos morning | 07:27 |
| zyga | I saw that. Suspicious. Could be a test merged recently. Could be a package upgrade (though I doubt that) | 07:28 |
| zyga | Oho | 07:29 |
| zyga | Yesterday evening was eventful | 07:29 |
| zyga | I need to scan the backlog | 07:29 |
| zyga | But first... dog | 07:29 |
| mborzecki | zyga: my best bet is the lxd test, but who knows, tryig main/tests/lxd right now | 07:32 |
| sdhd-sascha | Good morning | 07:37 |
| zyga | Do you have backlog to read? | 07:40 |
| mborzecki | zyga: so on vanilla system it's 31 22 0:26 / /sys/fs/cgroup rw shared:9 - tmpfs tmpfs rw,mode=755 | 07:46 |
| mborzecki | zyga: in -shell-before of tests/main/lxd it's already 1 22 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755 | 07:47 |
| zyga | mborzecki: on which kernel | 07:51 |
| zyga | mborzecki: systemd / kernel change if /sys/fs/cgroup is ro or rw | 07:51 |
| zyga | it used to be rw | 07:51 |
| mborzecki | zyga: heh, the gcp one | 07:52 |
| zyga | nowadays it is always ro | 07:52 |
| zyga | perhaps that's why | 07:52 |
| zyga | are we consistently getting "new" behavior? | 07:52 |
| zyga | if so let's just change the test and carry on | 07:52 |
| mborzecki | zyga: but it's not consistent, a clean node has rw | 07:52 |
| zyga | I'd only worry if half of the machines got one kernel and other half another | 07:52 |
| mborzecki | zyga: also, the reboot variant assumes rw and observes rw on the host | 07:53 |
| zyga | mborzecki: note, we do prepare.sh changes | 07:53 |
| zyga | oh | 07:53 |
| zyga | I'll dig in a moment | 07:53 |
| zyga | need to get that dog out first | 07:53 |
| zyga | there's some drama about apparmor bug last night | 07:53 |
| mborzecki | zyga: yeah, saw the PRs | 07:54 |
| mborzecki | zyga: also, don't know if you saw this comment from jdstrand https://github.com/snapcore/snapd/pull/7555#discussion_r354452340 | 07:55 |
| mup | PR #7555: tests: add a test demonstrating that snaps can't access the session agent socket <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7555> | 07:55 |
| sdhd-sascha | zyga: is there a summary (bug report) of the conversion. A year ago, i also had trouble to install kubernetes-suite with conjure-up. I also need to reparse apparmor at some installation step to make it work. But i didn't know if it was my fault, because i deployed everthing on zfs and kubernetes at this time the most kubernetes containers has missing zfs-utils | 08:03 |
| mborzecki | zyga: it's systemd version | 08:05 |
| pstolowski | mornings | 08:05 |
| mborzecki | zyga: on a clean host it's 229-4ubuntu21.22, then it gets update dto 229-4ubuntu21.23 | 08:05 |
| mborzecki | pstolowski: hey | 08:05 |
| pstolowski | mounts ns fun again | 08:06 |
| mborzecki | zyga: still, it's a bit of a myster why the mount-ns:reboot works | 08:06 |
| mborzecki | pstolowski: yeah | 08:06 |
| mborzecki | pstolowski: it's friday, some critical fix in a PR, and mount-ns failing, i.e. business as usual | 08:07 |
| mborzecki | zyga: hahah, soo, after a reboot it's back to rw | 08:12 |
| mborzecki | zyga: and only gets switched to ro after systemctl daemon-reexec, wtf | 08:13 |
| pstolowski | can we disable this test until a fix is ready to unblock landings? | 08:13 |
| zyga | re | 08:14 |
| zyga | mborzecki: maybe compund bug | 08:14 |
| zyga | compound* | 08:15 |
| zyga | perhaps the fixes we do ought to be each-boot and are first-boot | 08:15 |
| mborzecki | zyga: what fixes? | 08:15 |
| zyga | mborzecki: we do, or did some project wide changes, let me find that part | 08:15 |
| mborzecki | mvo: hey | 08:16 |
| mvo | hey mborzecki | 08:17 |
| zyga | hey mvo | 08:17 |
| mvo | hey zyga | 08:18 |
| mvo | how are you guys? | 08:18 |
| zyga | winter is here | 08:18 |
| mvo | I'm tired, tried to explore some ideas | 08:18 |
| pstolowski | hey mvo o/ | 08:18 |
| mvo | last night | 08:18 |
| mvo | but only medium successful :/ | 08:18 |
| mvo | hey pstolowski ! | 08:18 |
| zyga | other than that I wanted to swap off today | 08:18 |
| zyga | what did you try mvo? | 08:18 |
| mvo | zyga: ok | 08:18 |
| zyga | (but no swap because omg master red) | 08:19 |
| mvo | zyga: oh? | 08:19 |
| zyga | mborzecki: tests/lib/prepare-restore.sh | 08:19 |
| mvo | pstolowski: is 7771 ready? | 08:19 |
| mvo | I still want to branch 2.43 :) | 08:19 |
| zyga | mborzecki: look at line 215 | 08:19 |
| zyga | we undo lxd changes | 08:19 |
| zyga | that's only once on 1st boot | 08:19 |
| zyga | maybe that has consequences? | 08:19 |
| pstolowski | mvo: thanks for asking.. it is, but master is broken | 08:19 |
| zyga | mborzecki: we do -o remount | 08:20 |
| zyga | mborzecki: we don't do -o remount,ro -- maybe that makes it rw? | 08:20 |
| mborzecki | zyga: i don't think so, i have a clean host, after reboot i have rw, issue systemctl daemon-reexec and i have ro now | 08:20 |
| mvo | pstolowski: ok | 08:20 |
| zyga | ooooooooh | 08:20 |
| zyga | mborzecki: so | 08:20 |
| zyga | mborzecki: maybe that's just new systemd | 08:21 |
| zyga | it does do ro on /sys/fs/cgroup on modern systemd | 08:21 |
| mvo | zyga: anything we can do quickly to unbreak master? any test to skip for now until we have a solution? | 08:21 |
| mborzecki | zyga: it's triggered in the test, because we install/upgrade systemd as a build ependency, and postinst runs daemon-reexec | 08:21 |
| zyga | mvo: yes, we can disable mount-ns test | 08:21 |
| mborzecki | zyga: also why the :reboot variant works | 08:21 |
| zyga | it picked up something weird | 08:21 |
| zyga | mborzecki: that's curious, no explanation there | 08:21 |
| pstolowski | can we disable the mount-ns test until a fix is ready to unblock landings? or is there more? | 08:21 |
| pstolowski | zyga: ^ | 08:21 |
| mborzecki | pstolowski: yeah, i think so | 08:21 |
| zyga | pstolowski: that's one way, yeah | 08:21 |
| zyga | mvo: did you see last night chat between jdstrand and stgraber? | 08:22 |
| mvo | zyga: I did not | 08:22 |
| zyga | there's some kernel bug drama and broken lxd | 08:22 |
| zyga | mvo: please scan that - I bet it affects the releaase | 08:22 |
| pstolowski | zyga, mvo, mborzecki ok, i'll prep a PR | 08:22 |
| zyga | pstolowski: thanksI | 08:22 |
| mvo | pstolowski: thank you | 08:23 |
| zyga | I will read the backlog now | 08:23 |
| mborzecki | zyga: this is the changelog https://paste.ubuntu.com/p/zMvnwTHrDH/ | 08:23 |
| zyga | nothing scary there | 08:24 |
| mvo | hm, looks harmless | 08:24 |
| mborzecki | meh, there's no log or anythin that systemd remounts /sys/fs/cgroup | 08:26 |
| zyga | ijohnson: we don't have nested lxd tests AFAIK | 08:27 |
| mup | PR snapd#7859 opened: tests: disable mount-ns test on 16.04 for now <âš Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7859> | 08:29 |
| mborzecki | so we should probably be seeing ro after the reboot, but it's rw | 08:29 |
| mborzecki | btw. did you gus notice that the lxd test is not run on ubuntu-19.* ? | 08:30 |
| pstolowski | mborzecki: oh | 08:31 |
| zyga | mvo: you have mail | 08:31 |
| zyga | mvo: look for a message from stgraber please | 08:31 |
| mborzecki | pstolowski: it lists only this [ubuntu-16*, ubuntu-18.04*, ubuntu-2*, ubuntu-core-*] | 08:32 |
| pstolowski | yeah | 08:32 |
| zyga | mvo: we need to revert one line for the next release to unbreak lxd | 08:32 |
| zyga | mvo: and to check it we need to install lxd inside lxd | 08:32 |
| zyga | mvo: and then run snaps in the 2nd nested lxd to confirm | 08:32 |
| zyga | mborzecki: can you expand that to more systems please, it's likely related to the bug | 08:33 |
| mvo | zyga: ok | 08:33 |
| * mvo looks | 08:34 | |
| zyga | mvo: e7afbc34b1d630aeae4a7d20c34da75f4cb67546 | 08:34 |
| zyga | mvo: there we need to remove the "deny unix," rule | 08:34 |
| zyga | that's all | 08:34 |
| zyga | the regression test is nested lxd | 08:35 |
| zyga | I'll be back in 20 minutes, need to handle something at home | 08:35 |
| pedronis | hi | 08:35 |
| pstolowski | hi pedronis | 08:35 |
| mvo | zyga: uh, yeah, just reading | 08:35 |
| mvo | zyga: sucks :( | 08:35 |
| mvo | zyga: shows once more that dot releases can't be conservative enough :( | 08:36 |
| mborzecki | pedronis: hey | 08:36 |
| pedronis | mvo: seems we need a .5, also master is broken, see #7857 #7859 | 08:36 |
| mup | PR #7857: tests: update google ubuntu 16.04-64 expected host mount ns <Simple 😃> <⚠Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7857> | 08:36 |
| mup | PR #7859: tests: disable mount-ns test on 16.04 for now <âš Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7859> | 08:36 |
| zyga | mvo: kernel bugs are just bugs but yeah | 08:36 |
| mvo | pedronis: yes on both counts | 08:37 |
| mvo | pedronis: I will take care of this this morning | 08:38 |
| pedronis | mborzecki: if I understand correctly we are not sure about 7857 and your are looking into that? | 08:40 |
| pedronis | so we should go for 7859 for now? | 08:40 |
| mvo | I am in favour of 7859 for now to unblock things while it's investigated | 08:41 |
| pedronis | mvo: it's fine, trying to understand where we are, this mount-ns failures are starting to get a bit annoying | 08:42 |
| mborzecki | pedronis: yes | 08:42 |
| pedronis | mvo: not the most high prio right now, but I did leave comments late last night on your late PR(s) | 08:43 |
| mvo | pedronis: thank you so much | 08:43 |
| mvo | pedronis: will try to get to it as soon as I can. sorry that it's not easy :/ | 08:44 |
| pedronis | np, let's try to get master green and .5 ready | 08:44 |
| * pedronis is also not feeling 100% today, not sure why/what | 08:45 | |
| mvo | pedronis: just read your comment, makes a lot of sense | 08:45 |
| pstolowski | pedronis: yep (about mount-ns test). it's fine to have a test like this but it seems to be picking platform changes which are more frequent than expected, rather than our bugs. this area is unit-tested, so perhaps this test should be run nightly | 08:48 |
| zyga | re | 08:50 |
| zyga | pstolowski: we can also tweak the test to skip certain areas | 08:50 |
| pstolowski | zyga: that's an option too. it should probably be less picky about flags on the filesystems that are not controlled by us | 08:52 |
| zyga | pstolowski: it's all a learning exercise, we figure out as we go what is broken | 08:52 |
| zyga | mborzecki: do you need a hand on that flag mystery or can I focus on something else? | 08:54 |
| mup | PR snapd#7261 closed: interfaces/serial-port: support pci bus serial-port with HotplugKey() <â›” Blocked> <Created by anonymouse64> <Closed by pedronis> <https://github.com/snapcore/snapd/pull/7261> | 09:04 |
| sdhd-sascha | Chipaca: would it be ok, to build spread snap on core18 ? | 09:11 |
| Chipaca | sdhd-sascha: why do i get the impression you're breadth-first digging all the rabbit holes? | 09:12 |
| sdhd-sascha | Chipaca: just for learning. You mention yesterday, that the snap needs a update. | 09:13 |
| zyga | brb, rebooting | 09:13 |
| Chipaca | sdhd-sascha: the updated snapcraft.yaml I have here is base:core18 | 09:14 |
| pedronis | mborzecki: we are getting failures on selinux-clean | 09:15 |
| pedronis | https://api.travis-ci.org/v3/job/621494044/log.txt | 09:16 |
| pedronis | that are failing on the disable mount-ns PR | 09:16 |
| pedronis | pstolowski: ^ | 09:16 |
| sdhd-sascha | Chipaca: does your snapcraft.yaml has kvm inside ? then i could took this and experiment. | 09:17 |
| Chipaca | sdhd-sascha: it does not, no :) | 09:17 |
| Chipaca | sdhd-sascha: i can push what i have here so you start from it | 09:18 |
| sdhd-sascha | Chipaca: ok. Yesterday, i only added git and gcc for core18 on multipass/qemu. | 09:19 |
| pstolowski | pedronis, mborzecki mhm.. looking | 09:19 |
| mborzecki | pstolowski: i think the policy needs to beupdated | 09:20 |
| pstolowski | mborzecki: yes, i'm just wondering what changed that triggered it\ | 09:20 |
| mborzecki | pstolowski: i think it's the interfaces-kvm test that ran before, the kvm inteface writes out a modprobe conf file that loads the relevant kvm driver | 09:24 |
| zyga | mborzecki: I'm doing other things, assuming you are good | 09:24 |
| mborzecki | pstolowski: i guess you can restart the travis job, and open a fix in a separate PR (or i'll do it after the meeting) | 09:24 |
| zyga | mborzecki: please drag me back if you need assistance on anything | 09:25 |
| zyga | trying to cut the number of my open branches | 09:25 |
| pstolowski | mborzecki: ok, let me try | 09:27 |
| pstolowski | mborzecki: ok, i've the policy update, will give it a try and wait for #7859 (in case it needs to go together) | 09:42 |
| mup | PR #7859: tests: disable mount-ns test on 16.04 for now <âš Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7859> | 09:42 |
| zyga | pstolowski: thank you for a targeted patch! | 09:42 |
| pedronis | zyga: are you going to force push again against --explain? I might get to do an actual review pass today | 09:50 |
| zyga | pedronis: not anymore, I just changed where the spread test runs to limit it to ubuntu classic 64 | 09:51 |
| pedronis | ok | 09:51 |
| mborzecki | pstolowski: ok, thanks! | 10:00 |
| mborzecki | zyga: btw. another data point, clean cloud image from cloud-images.u.c has ro mode after booting | 10:00 |
| mborzecki | quick errand, back in 30 | 10:01 |
| zyga | that's good, that's what I would expect | 10:01 |
| mborzecki | zyga: makes me wonder, why our spread images have rw, obviously the kernel is different, but i would expect systemd to apply very specific mount options when setting up /sys/fs/cgroup | 10:02 |
| zyga | mborzecki: yeah, it does | 10:02 |
| zyga | I read that part of systemd source code | 10:02 |
| zyga | mborzecki: two ideas: cloud agent | 10:02 |
| zyga | mborzecki: or custom kernel playing a factor | 10:02 |
| mborzecki | zyga: you mean some gcp cloud agent? bc i'm using cloud-init under qemu too | 10:03 |
| mborzecki | brb | 10:03 |
| zyga | yes | 10:03 |
| mup | PR snapd#7859 closed: tests: disable mount-ns test on 16.04 for now <âš Critical> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/7859> | 10:11 |
| mup | PR snapd#7860 opened: selinux: update policy to allow modifications related to kmod backend <Created by stolowski> <https://github.com/snapcore/snapd/pull/7860> | 10:15 |
| pedronis | mborzecki: you can try to run the tests with an error in the early prepares and see how the world looks there | 10:15 |
| mborzecki | re | 10:37 |
| mborzecki | pedronis: i'm using https://github.com/bboozzoo/spread-mini to spin up the node without any of our test setup | 10:38 |
| pedronis | ah | 10:45 |
| mvo | mborzecki: which test was it again that caused the selinux denial that was just fixed? | 10:46 |
| mborzecki | mvo: interfaces-kvm | 10:46 |
| mvo | mborzecki: ta | 10:48 |
| pstolowski | mvo: i think it's a good idea to have selinux check in kvm-interfaces | 10:50 |
| mvo | pstolowski: thank you | 10:54 |
| sdhd-sascha | Chipaca: if you have pushed, maybe in a feature-branch. you could inform me. I just cleanup git-history and build a current sway version | 11:01 |
| Chipaca | sdhd-sascha: https://github.com/chipaca/spread/tree/update-snapcraft-yaml | 11:05 |
| zyga | I'll be right back, I'll make something warm to drink | 11:23 |
| mup | PR snapd#7771 closed: o/hookstate/ctlcmd: snapctl is-connected command <Needs Samuele review> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7771> | 11:42 |
| mup | PR snapd#7860 closed: selinux: update policy to allow modifications related to kmod backend <Test Robustness> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7860> | 11:42 |
| mup | PR snapd#7861 opened: tests: check for SELinux denials in interfaces-kvm spread test <Simple 😃> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7861> | 11:47 |
| mup | PR snapcraft#2824 closed: Support for go.mod <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2824> | 12:07 |
| mup | PR snapcraft#2832 closed: appstream extractor: take xml comments into account <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2832> | 12:07 |
| mup | PR snapcraft#2822 closed: xattrs: ignore errors if SNAPCRAFT_BUILD_INFO is unset <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2822> | 12:10 |
| mup | PR snapcraft#2830 closed: elf: properly handle corrupted ELF files <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2830> | 12:10 |
| mup | PR snapd#7856 closed: snap-confine: revert, with comment, explicit unix deny for nested lxd <Simple 😃> <⚠Critical> <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7856> | 12:28 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/7856#issuecomment-562517492 | 12:29 |
| mup | PR #7856: snap-confine: revert, with comment, explicit unix deny for nested lxd <Simple 😃> <⚠Critical> <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7856> | 12:29 |
| zyga | mvo: do we plan to add this? | 12:29 |
| mvo | zyga: add what exactly? | 12:30 |
| zyga | mvo: a spread test verifying that this is fixed, | 12:30 |
| zyga | mvo: a lxd inside lxd | 12:31 |
| Tuor | Hi, I just install vlc with snap on a kubuntu 19.10. My mouseppointer changes when it hovers over the vlc window. Can I keep my normal mousepointer somehow? | 12:31 |
| mvo | zyga: it's running in master | 12:31 |
| zyga | mvo: I don't follow, sorry | 12:31 |
| zyga | mvo: are you saying we have that test already? | 12:31 |
| zyga | mvo: a test running nested lxd | 12:31 |
| zyga | Tuor: hey, this is a known bug | 12:31 |
| zyga | Tuor: I can refer you to kenvandine | 12:31 |
| Tuor | Shall I upvote a bugrequest or something? | 12:32 |
| Tuor | *bugreport | 12:32 |
| mvo | zyga: sorry in a meeting | 12:33 |
| zyga | mvo: I think we don't have that test, please double check that we add one before doing a .5 | 12:33 |
| zyga | mvo: or we may realize more is broken and .6 is required | 12:33 |
| zyga | Tuor: I don't know where it is tracked, please ask kenvandine for details | 12:35 |
| Tuor | kenvandine: I encountered a known bug, that my mouse pointer changes when I use the vlc snap. Where can I report a bug or where should I upvote? | 12:36 |
| mborzecki | anyone up for a 2nd review of https://github.com/snapcore/snapd/pull/7821 ? | 12:37 |
| mup | PR #7821: interfaces/seccomp: parallelize seccomp backend setup <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7821> | 12:37 |
| pstolowski | mborzecki: yeah, i'll, was meaning to | 12:46 |
| === pedronis_ is now known as pedronis | ||
| pedronis | zyga: Ian wrote one I think | 12:46 |
| pedronis | zyga: https://github.com/snapcore/snapd/pull/7858 | 12:46 |
| mup | PR #7858: tests: add nested-lxd test to confirm lxd inside lxd works <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7858> | 12:47 |
| pedronis | it needs reviews | 12:47 |
| zyga | pedronis: that's good, we should ensure it passes with updated master | 12:47 |
| zyga | yep | 12:47 |
| pedronis | zyga: does #7830 need jdstrand review? I would say no but open otherwise... or just your re-review | 12:50 |
| mup | PR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830> | 12:50 |
| mborzecki | pstolowski: cool, thnanks! | 12:55 |
| === ricab is now known as ricab|bbl | ||
| mvo | zyga: sorry, was in a meeting - 7858 is the one I had in mind | 13:07 |
| zyga | pedronis: checking | 13:08 |
| mvo | zyga: i.e. we have a test for this | 13:08 |
| mvo | zyga: in a PR | 13:08 |
| mvo | zyga: but it's incomplete, see PR | 13:08 |
| mup | PR snapd#7862 opened: release: 2.42.5 <Created by mvo5> <https://github.com/snapcore/snapd/pull/7862> | 13:20 |
| Chipaca | woo! got an appointment with the orthopedician! woo | 13:22 |
| Chipaca | with xmas coming up i was sure i was going to get punted into the new year | 13:23 |
| mborzecki | heh, our test cleanup, or lack thereof, keeps on giving | 13:29 |
| mvo | mborzecki: hm? | 13:39 |
| mvo | mborzecki: more catastrophies? | 13:39 |
| mborzecki | mvo: yeah, though suprised this one didn't happen earlier, the failure in selinux-context in 7570 is install-socket-activation test leaking socket units apparently | 13:40 |
| mborzecki | and we're probably missing cleanup for sockt files too, meh | 13:41 |
| mup | PR snapd#7767 closed: tests: run snap-set-core-config on all core devices <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7767> | 13:51 |
| mup | PR snapd#7863 opened: interfaces/builtin: add uio interface <Created by zyga> <https://github.com/snapcore/snapd/pull/7863> | 13:57 |
| Chipaca | huh, firefox crashed trying to get to the meet | 14:01 |
| mup | PR snapd#7864 opened: cmd/snap-mgmt, packaging/postrm: stop and remove socket units when purging <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7864> | 14:24 |
| pstolowski | https://github.com/snapcore/snapd/pull/7861 needs 2nd review (trivial) | 14:35 |
| mup | PR #7861: tests: check for SELinux denials in interfaces-kvm spread test <Simple 😃> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7861> | 14:35 |
| * zyga breaks for food | 14:46 | |
| pstolowski | thanks ijohnson ! | 14:47 |
| mup | PR snapd#7861 closed: tests: check for SELinux denials in interfaces-kvm spread test <Simple 😃> <Test Robustness> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/7861> | 14:48 |
| mborzecki | off to pick up the kids | 15:04 |
| * cachio lunch | 15:06 | |
| pedronis | pstolowski: thx for the card | 15:13 |
| pedronis | ijohnson: will you get to 7768 today? | 15:14 |
| ijohnson | pedronis: yes I will look at it now; sorry I didn't have time yesterday with the k8s stuff and then this lxd issue | 15:15 |
| pedronis | np, thx | 15:15 |
| mup | PR snapd#7858 closed: tests: add nested-lxd test to confirm lxd inside lxd works <Test Robustness> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7858> | 15:17 |
| === ricab|bbl is now known as ricab | ||
| mup | PR snapd#7857 closed: tests: update google ubuntu 16.04-64 expected host mount ns <â›” Blocked> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7857> | 15:19 |
| * Chipaca goes for tea | 15:22 | |
| ogra | is there an api call that would trigger a restart of snapd ? i.e. like https://github.com/snapcore/snapd/wiki/REST-API#request-2 but with snapd as the snap ? or is that blocked | 15:24 |
| zyga | re | 15:25 |
| zyga | what should I review? | 15:25 |
| ogra | (this is obviously a core18 and beyond question where snapd is its own snap) | 15:28 |
| ogra | hmm, k ... i think this answers it ... | 15:28 |
| ogra | $ snap restart snapd | 15:29 |
| ogra | error: snap "snapd" has no services | 15:29 |
| zyga | jdstrand: fyi https://github.com/snapcore/snapd/pull/7850/files#r354888954 | 15:31 |
| mup | PR #7850: apparmor: allow 'r' /sys/kernel/mm/transparent_hugepage/hpage_pmd_size <Simple 😃> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7850> | 15:31 |
| pedronis | zyga: #7830 needs your re-review | 15:32 |
| mup | PR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830> | 15:32 |
| zyga | I'm doing that now :) | 15:33 |
| zyga | pstolowski, pedronis: https://github.com/snapcore/snapd/pull/7830/files#r354891136 | 15:35 |
| mup | PR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830> | 15:35 |
| pstolowski | zyga: ty | 15:35 |
| pstolowski | btw i'm looking at your uio iface | 15:35 |
| zyga | cool :) | 15:35 |
| zyga | pstolowski: one thing to keep in mind is that it must grant mmap access to /dev/uioN | 15:36 |
| zyga | I fixed that but missed it in the initial testing because the test tool didn't require that | 15:36 |
| zyga | pstolowski: https://github.com/snapcore/snapd/pull/7830#pullrequestreview-328280697 +1 | 15:37 |
| mup | PR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830> | 15:37 |
| zyga | pstolowski: let me know if you want to work on the test device remotely | 15:38 |
| jdstrand | zyga: responded, thanks | 15:39 |
| zyga | jdstrand: cool :-) | 15:41 |
| jdstrand | zyga: it was even in the commit missage for snaap-update-ns... not surre why I put it there :) | 15:41 |
| jdstrand | message* | 15:41 |
| zyga | haha, no worries | 15:42 |
| zyga | I'm happy I asked a meaningful question :) | 15:42 |
| * ijohnson needs to go downtown for an hour or so, probably will miss tgif | 15:44 | |
| zyga | take care ijohnson | 15:44 |
| ijohnson | zyga: I'll be back! And I realize in retrospect that phrase is a bit loaded, it's just some paperwork things :-) | 15:51 |
| zyga | pstolowski: updated https://github.com/snapcore/snapd/pull/7863 -- thank you for the quick review! | 16:31 |
| mup | PR #7863: interfaces/builtin: add uio interface <Created by zyga> <https://github.com/snapcore/snapd/pull/7863> | 16:31 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/7825#issuecomment-562645242 | 16:37 |
| mup | PR #7825: many: use transient scope for tracking apps and hooks <â›” Blocked> <Created by zyga> <https://github.com/snapcore/snapd/pull/7825> | 16:37 |
| mvo | zyga: oh, it is? sorry, then let's remove it | 16:38 |
| zyga | yeah | 16:38 |
| mvo | zyga: sorry, too much going on at the same time | 16:38 |
| zyga | it's a nop without it | 16:38 |
| zyga | no worries | 16:38 |
| zyga | I +1 the idea initially because it feels safe | 16:38 |
| pedronis | it's still not merge ready | 16:38 |
| pedronis | though | 16:38 |
| zyga | but I think it's not unsafe in principle | 16:38 |
| zyga | that's true :) | 16:38 |
| pedronis | it's unlikely to get merge ready by Mon or Tue | 16:39 |
| pedronis | tbh | 16:39 |
| zyga | pedronis: what is missing there? | 16:40 |
| * zyga EODs and goes for a walk | 16:55 | |
| mup | PR snapd#7858 opened: tests: also check nested lxd container <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7858> | 17:25 |
| ijohnson | mvo / pedronis: should I open a PR for the lxd snap regression test against release/2.42 ? | 17:26 |
| mup | PR snapd#7847 closed: snap-bootstrap: parse seed if either kernel or base are not mounted <UC20> <Created by xnox> <Merged by cmatsuoka> <https://github.com/snapcore/snapd/pull/7847> | 17:40 |
| cmatsuoka | cachio: did you notice random test failures on debian recently? | 17:56 |
| cachio | cmatsuoka, do you have a link? | 17:56 |
| cachio | cmatsuoka, didn't see that | 17:56 |
| cmatsuoka | cachio: I just restarted it but if it fails again I'll let you know | 17:57 |
| cachio | cmatsuoka, nice, thanks | 17:57 |
| ijohnson | hey jdstrand you around for a couple minutes to discuss the kubernetes-worker snap ? | 19:18 |
| sdhd-sascha | zyga: yesterday, i tried to build a *.deb package for snapd on launchpad.net with a receipe. Is the release-deb also build on launchpad, or ... ? | 19:22 |
| sdhd-sascha | zyga: oh, wait... maybe it was build with snapcraft... i looking for it | 19:25 |
| sdhd-sascha | niemeyer: hello, sorry for mention you. Just try to build a *.deb package of snapd for testing. https://code.launchpad.net/~sdhd/+snap/snapd-daily | 19:40 |
| sdhd-sascha | But there was no place to configure the target ppa ... | 19:41 |
| sdhd-sascha | Inside of the snap-build process, there i found: dpkg-buildpackage: source package snapd | 20:00 |
| sdhd-sascha | niemeyer: | 20:14 |
| sdhd-sascha | zyga: My fault. I think i found it | 20:14 |
| jdstrand | ijohnson: I am here. what's up? | 20:29 |
| jdstrand | ijohnson: (sorry I missed the initial question) | 20:29 |
| ijohnson | no it's okay I don't think I asked the actual question here | 20:29 |
| ijohnson | so I have this kubernetes-worker snap here and I was trying to figure out why the containers launched by containerd can't create their own top level directories because AppArmor denies it | 20:30 |
| ijohnson | it was odd because docker containers are allowed to do this, and after a while it occured to me that docker will perform a profile transition to the docker-default profile which doesn't constrain the container by AppArmor for top level directories, etc. | 20:31 |
| ijohnson | after looking some more, it appears that containerd doesn't by default transition the containers to an apparmor profile, but I was wondering what your thoughts are if we somehow got the kubernetes-worker snap to do that? | 20:31 |
| ijohnson | do you think it's okay to have containers launched by the kubernetes-worker snap be confined by the docker-default profile | 20:32 |
| ijohnson | ? | 20:32 |
| jdstrand | ijohnson: the profile was designed with this in mind. it is supposed to do this: https://github.com/snapcore/snapd/blob/master/interfaces/builtin/docker_support.go#L160 | 20:34 |
| jdstrand | ijohnson: when I did the interface initially with microk8s, I made it microk8s load the profile in one of the wrappers so it was able to transition | 20:35 |
| ijohnson | nice | 20:35 |
| jdstrand | ijohnson: I thought we did that with kubernetes-worker as well. I distinctly recall pointing out this should be done. and, iirc, containerd would use the profile if it was loaded | 20:35 |
| jdstrand | ijohnson: but wouldn't load it itself. I might be misremembering, but I thought that was the case | 20:36 |
| ijohnson | jdstrand: that might be the case, I'm not sure | 20:36 |
| jdstrand | ijohnson: regardless, yes, it is expected that the containerd app can load the profile and transition containers to it | 20:36 |
| jdstrand | how to make it do that is another thing... :) | 20:37 |
| ijohnson | I'm not running the full setup, I'm just trying to drive containerd manually and noticed it doesn't do this by default | 20:37 |
| ijohnson | ok, well my question for you was if this is okay, I assumed it was because it's in the policy but wanted to be sure :-) | 20:37 |
| ijohnson | thanks | 20:37 |
| jdstrand | ijohnson: let me see if that code is in microk8s... | 20:37 |
| jdstrand | ijohnson: it is not only ok, it is recommended and best practice :) | 20:38 |
| ijohnson | :-) | 20:38 |
| jdstrand | ijohnson: it does mean that containerd is more privilged, but the containers are less, and the container attack surface so what's most important | 20:38 |
| ijohnson | right | 20:38 |
| jdstrand | s/so/is/ | 20:39 |
| ijohnson | jdstrand: I don't see anything in the kubernetes-worker snap to load that profile | 20:39 |
| ijohnson | joedborg: any idea on where that bit of code might have went in the kubernetes-worker snap? | 20:39 |
| jdstrand | which is why I clal the docker (and containerd) policy 'advisory', since they can load prolicy | 20:39 |
| jdstrand | policy | 20:39 |
| jdstrand | and transition to it | 20:39 |
| ijohnson | brb | 20:40 |
| joedborg | ijohnson: do you mean this bit? https://github.com/charmed-kubernetes/snap-kubernetes-worker/blob/master/wrappers/containerd.wrapper#L13 | 20:43 |
| jdstrand | ijohnson: https://github.com/ubuntu/microk8s/blob/feature/strict-v2/microk8s-resources/containerd-profile and https://github.com/ubuntu/microk8s/blob/feature/strict-v2/microk8s-resources/wrappers/run-containerd-with-args#L11 | 20:43 |
| jdstrand | joedborg: if you run 'sudo aa-status', you see cri-containerd.apparmor.d listed? | 20:44 |
| joedborg | no, i don't seem to | 20:48 |
| joedborg | https://www.irccloud.com/pastebin/IVv1DKT0/ | 20:48 |
| joedborg | jdstrand: ^ | 20:48 |
| ijohnson | joedborg: ah yes I didn't see that for some reason | 20:48 |
| jdstrand | joedborg: I think there is something wrong then with the startup. that profile needs to be loaded every time containerd starts | 20:49 |
| joedborg | ijohnson: it is missing from the eks branch since yesterday because i was getting errors | 20:49 |
| jdstrand | also, fyi I found https://kubernetes.io/docs/tutorials/clusters/apparmor/ | 20:49 |
| ijohnson | jdstrand: with the eks branch that joedborg showed me yesterday I see that profile being loaded into the kernel | 20:50 |
| jdstrand | that hints at defaults and things and might provide some insight if the pods still aren't running in cri-containerd.apparmor.d once it is confirmed that it is loaded in the kernel | 20:50 |
| ijohnson | so I think that parts fine, it seems now that the issue is just in configuring $CONTAINER_TECHNOLOGY to actually run under that profile | 20:51 |
| ijohnson | https://www.irccloud.com/pastebin/fTe1mJiV/ | 20:51 |
| jdstrand | ijohnson: ok, then if it is loaded, you might be able to use the above link to make k8s put it in that profile. there is probably something somewhere for defining the default profile to use | 20:51 |
| jdstrand | ijohnson: ok, cool, and yes | 20:52 |
| jdstrand | https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/pods/security/hello-apparmor.yaml | 20:52 |
| ijohnson | joedborg: have you looked at https://aws.amazon.com/blogs/opensource/using-pod-security-policies-amazon-eks-clusters/ ? | 20:52 |
| jdstrand | (from the above link) | 20:53 |
| ijohnson | that seems to imply that the pods metadata stuff from the generic kubernetes stuff can be configured in EKS | 20:53 |
| ijohnson | thanks jdstrand I saw that link earlier today too | 20:53 |
| jdstrand | ijohnson: nice find | 20:53 |
| ijohnson | BTW word of caution: don't go looking for containerd's CLI tool "ctr" docs or examples because they don't exist haha | 20:53 |
| ijohnson | (I was trying to be sneaky and just skip the k8s part and just reproduce the issue with ctr, but that doesn't seem to be the way to go) | 20:54 |
| jdstrand | oh yeah, I had some trouble with that in the past | 20:54 |
| joedborg | ijohnson: i did briefly, but saw that `Read Only Root Filesystem: false` so moved on | 20:55 |
| joedborg | ijohnson: jdstrand: ctr is also a massive trap because it behaves completely differently to anything else that drives containerd (all over CRI) | 20:56 |
| ijohnson | oh hmm so does the psp stuff work by some aws agent running on the pod to enable these things? if so that's really unfortunate | 20:56 |
| jdstrand | ijohnson: I have this in my notes when I did a seccomp security update: https://paste.ubuntu.com/p/P6M4hhkphw/ | 20:56 |
| jdstrand | ijohnson: but based on what joedborg just said, you may want to ignore that :) | 20:56 |
| ijohnson | jdstrand: yes `runc spec` is a good trick until you realize how much other config settings $OTHER_CONTAINER_TECHNOLOGIES set when they drive CRI like joedborg just said | 20:57 |
| * jdstrand nods | 20:57 | |
| jdstrand | this was just for the deb | 20:58 |
| jdstrand | ijohnson: I like when you say $CONTAINER_TECHNOLOGY and $OTHER_CONTAINER_TECHNOLOGIES | 20:58 |
| jdstrand | it is just right ;) | 20:58 |
| ijohnson | :-) there's just so so so many of them | 20:59 |
| jdstrand | indeed :) | 20:59 |
| joedborg | ijohnson: | 21:01 |
| joedborg | jdstrand: | 21:01 |
| joedborg | `apparmor_parser -r $SNAP/containerd-profile` i took this out because i started getting `permission denied` when it runs the wrapper | 21:02 |
| joedborg | even if i put sudo in front of it | 21:02 |
| joedborg | so i think an apparmor rule (to allow to run this) has gone missing | 21:02 |
| joedborg | that's using the custom snapd | 21:02 |
| jdstrand | joedborg: you can't put sudo in front of it | 21:03 |
| jdstrand | joedborg: can you: sudo snap run --shell snap.kubernetes-worker.conttainerd | 21:03 |
| jdstrand | joedborg: then apparmor_parser -r $SNAP/containerd-profile | 21:03 |
| jdstrand | joedborg: and ive me the denials? | 21:04 |
| jdstrand | joedborg: (from journalctl) | 21:04 |
| jdstrand | ijohnson, joedborg: not, I have a very hard stop in 15 minutes, but could circle back | 21:04 |
| jdstrand | note* | 21:04 |
| joedborg | jdstrand: okay, i've got that now | 21:11 |
| joedborg | https://www.irccloud.com/pastebin/2q7ch6D4/ | 21:12 |
| jdstrand | jo | 21:12 |
| jdstrand | joedborg: I'm confused. I thought you said from within the snap it didn't work? | 21:12 |
| joedborg | jdstrand: it wasn't, then i ran it manually like you suggested and it did | 21:13 |
| jdstrand | joedborg: I was wanting you to run apparmor_parser -r $SNAP/containerd-profile from under sudo snap run --shell | 21:13 |
| jdstrand | jo | 21:13 |
| jdstrand | dang it | 21:13 |
| joedborg | but `Dec 06 20:54:31 ip-192-168-35-165 kubernetes-worker.containerd[18362]: /snap/kubernetes-worker/x1/bin/containerd.wrapper: line 13: /sbin/apparmor_parser: Permission denied` | 21:13 |
| jdstrand | joedborg: you ran it from under snap run --shell? | 21:13 |
| joedborg | yeah | 21:13 |
| jdstrand | joedborg: ok, between that and that ijohnson said it loaded for him, I'm going to chalk that up to the docker-support interface wasn't connected at the time of that denial | 21:15 |
| jdstrand | /sbin/apparmor_parser ixr, | 21:15 |
| jdstrand | that is in the docker-support policy ^ | 21:15 |
| joedborg | jdstrand: ah yes, quite possibly | 21:15 |
| joedborg | either way, i'm still getting the RO filesystem errors | 21:16 |
| jdstrand | joedborg: I didn't look at the code for the kubernetes-worker, but be sure that this apparmor_parser invocation is not in a configure hook or under some conditional where it only sometimes loads the policy | 21:16 |
| jdstrand | joedborg: for the rofs issues, this is where ijohnson can come in | 21:17 |
| joedborg | jdstrand: it's with the containerd wrapper, so run everytiime containerd is (re)started | 21:17 |
| jdstrand | joedborg: perfect | 21:17 |
| ijohnson | (in meeting) | 21:17 |
| joedborg | https://www.irccloud.com/pastebin/50YgamaD/ | 21:18 |
| joedborg | jdstrand: is that still looking sane? ^ | 21:18 |
| jdstrand | joedborg, ijohnson: ok, I need to head out for my appt, but I'll circle back and see if there is anything for me to do | 21:18 |
| joedborg | jdstrand: +1 thanks! | 21:18 |
| jdstrand | joedborg: it looks like a containerd default profile, yes | 21:19 |
| jdstrand | sane is in the eye of the beholder :) | 21:19 |
| jdstrand | but jokes aside, yes | 21:19 |
| joedborg | jdstrand: perfect :) | 21:19 |
| jdstrand | joedborg: so, if there are apparmor denials, ijohnson can help you add them to the right places for working around stuff and moving forward. if that happens, I can collect them and give you a new demo deb. I can do that tonight/over the weekend/etc | 21:20 |
| jdstrand | joedborg: I'll keep an eye on irc | 21:21 |
| joedborg | jdstrand: sadly, there aren't any. i think that's the main issue now | 21:21 |
| jdstrand | joedborg: yes, I just wanted to reiterate that I'll update the demo deb as needed | 21:21 |
| jdstrand | I'm also on tg | 21:22 |
| joedborg | ahhh :) | 21:22 |
| ijohnson | joedborg: ok I'm back | 21:49 |
| ijohnson | joedborg: so we're still at the issue with the filesystem being ro? | 21:51 |
| joedborg | ijohnson: yeah | 21:53 |
| joedborg | ijohnson: sadly | 21:54 |
| ijohnson | joedborg: I'm wondering if you could do something like `watch -n 0.05 bash -c 'sudo aa-status | grep $(pgrep install-aws.sh)'` on the node to see if it shows up under an apparmor profile | 21:57 |
| ijohnson | err maybe make that snippet bit smarter so it actually filters properly | 21:57 |
| ijohnson | one second | 21:57 |
| joedborg | ijohnson: +1 | 21:58 |
| ijohnson | joedborg: try `until pgrep install-aws.sh; do true; done; pid=$(pgrep install-aws.sh); sudo aa-status | grep "$pid"` | 22:03 |
| ijohnson | watch is a bit difficult to get to work there | 22:03 |
| joedborg | ijohnson: i don't appear to be getting any results as the pod starts and fails | 22:06 |
| ijohnson | joedborg: what about if you did: `until pgrep install-aws.sh; do true; done; pgrep install-aws.sh` | 22:08 |
| ijohnson | that should eventually print off some pid | 22:08 |
| joedborg | ijohnson: i still have the same node running if you wanted to watch the byobu | 22:08 |
| joedborg | ijohnson: ill try it | 22:08 |
| ijohnson | unless the name of this script is not install-aws.h | 22:08 |
| ijohnson | err install-aws.sh | 22:08 |
| joedborg | ijohnson: that latter one works | 22:09 |
| ijohnson | cool, one sec let me give you something else to run | 22:09 |
| ijohnson | joedborg: can you run `until pgrep install-aws.sh; do true; done; sudo nsenter -t $(pgrep install-aws.sh) --all /bin/bash -c "cat /proc/self/mountinfo"` | 22:10 |
| ijohnson | that should show us the mount namespace of the process when its running | 22:10 |
| ijohnson | joedborg: actually I think I will join that byobu if that's alright | 22:12 |
| ijohnson | might make this quicker | 22:12 |
| joedborg | ijohnson: of course :) | 22:12 |
| joedborg | ijohnson: fyi i have 2 tabs inside it open | 22:12 |
| * ijohnson googles how to switch tabs in byobu | 22:13 | |
| joedborg | F3 and F4 | 22:13 |
| mup | PR snapcraft#2834 opened: dirs: support --user install on Linux <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2834> | 22:53 |
| diddledan | --user? | 23:15 |
| mup | PR snapcraft#2835 opened: colcon plugin: support ROS 2 Eloquent <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2835> | 23:50 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!