mupPR snapd#7855 opened: snap-confine: revert suppress noisy classic snap file_inherit denials <Simple 😃> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7855>01:04
mupPR snapcraft#2833 opened: Arnatious/remove rospack workaround <Created by Arnatious> <https://github.com/snapcore/snapcraft/pull/2833>01:45
mupPR snapd#7856 opened: snap-confine: revert, with comment, explicit unix deny for nested lxd <Simple 😃> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7856>02:01
mupPR snapd#7857 opened: tests: update google ubuntu 16.04-64 expected host mount ns <Simple 😃> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7857>04:41
mupPR snapd#7858 opened: tests: add nested-lxd test to confirm lxd inside lxd works <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7858>04:46
mupPR snapd#7855 closed: snap-confine: revert suppress noisy classic snap file_inherit denials <Simple 😃> <⚠ Critical> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7855>04:49
mupPR snapd#7846 closed: devicestate: add missing test for failing task setup-run-system <Simple 😃> <UC20> <Created by mvo5> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7846>06:59
mborzeckimount-ns test failing on google?07:04
zygaGooos morning07:27
zygaI saw that. Suspicious. Could be a test merged recently. Could be a package upgrade (though I doubt that)07:28
zygaYesterday evening was eventful07:29
zygaI need to scan the backlog07:29
zygaBut first... dog07:29
mborzeckizyga: my best bet is the lxd test, but who knows, tryig main/tests/lxd right now07:32
sdhd-saschaGood morning07:37
zygaDo you have backlog to read?07:40
mborzeckizyga: so on vanilla system it's 31 22 0:26 / /sys/fs/cgroup rw shared:9 - tmpfs tmpfs rw,mode=75507:46
mborzeckizyga: in -shell-before of tests/main/lxd it's already 1 22 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=75507:47
zygamborzecki: on which kernel07:51
zygamborzecki: systemd / kernel change  if /sys/fs/cgroup is ro or rw07:51
zygait used to be rw07:51
mborzeckizyga: heh, the gcp one07:52
zyganowadays it is always ro07:52
zygaperhaps that's  why07:52
zygaare we consistently getting "new" behavior?07:52
zygaif so let's just change the test and carry on07:52
mborzeckizyga: but it's not consistent, a clean node has rw07:52
zygaI'd only worry if half of the machines got one kernel and other half another07:52
mborzeckizyga: also, the reboot variant assumes rw and observes rw on the host07:53
zygamborzecki: note, we do prepare.sh changes07:53
zygaI'll dig in a moment07:53
zyganeed to get that dog out first07:53
zygathere's some drama about apparmor bug last night07:53
mborzeckizyga: yeah, saw the PRs07:54
mborzeckizyga: also, don't know if you saw this comment from jdstrand https://github.com/snapcore/snapd/pull/7555#discussion_r35445234007:55
mupPR #7555: tests: add a test demonstrating that snaps can't access the session agent socket <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7555>07:55
sdhd-saschazyga: is there a summary (bug report) of the conversion. A year ago, i also had trouble to install kubernetes-suite with conjure-up. I also need to reparse apparmor at some installation step to make it work. But i didn't know if it was my fault, because i deployed everthing on zfs and kubernetes at this time the most kubernetes containers has missing zfs-utils08:03
mborzeckizyga: it's systemd version08:05
mborzeckizyga: on a clean host it's 229-4ubuntu21.22, then it gets update dto 229-4ubuntu21.2308:05
mborzeckipstolowski: hey08:05
pstolowskimounts ns fun again08:06
mborzeckizyga: still, it's a bit of a myster why the mount-ns:reboot works08:06
mborzeckipstolowski: yeah08:06
mborzeckipstolowski: it's friday, some critical fix in a PR, and mount-ns failing, i.e. business as usual08:07
mborzeckizyga: hahah, soo, after a reboot it's back to rw08:12
mborzeckizyga: and only gets switched to ro after systemctl daemon-reexec, wtf08:13
pstolowskican we disable this test until a fix is ready to unblock landings?08:13
zygamborzecki: maybe compund bug08:14
zygaperhaps the fixes we do ought to be each-boot and are first-boot08:15
mborzeckizyga: what fixes?08:15
zygamborzecki: we do, or did some project wide changes, let me find that part08:15
mborzeckimvo: hey08:16
mvohey mborzecki08:17
zygahey mvo08:17
mvohey zyga08:18
mvohow are you guys?08:18
zygawinter is here08:18
mvoI'm tired, tried to explore some ideas08:18
pstolowskihey mvo o/08:18
mvolast night08:18
mvobut only medium successful :/08:18
mvohey pstolowski !08:18
zygaother than that I wanted to swap off today08:18
zygawhat did you try mvo?08:18
mvozyga: ok08:18
zyga(but no swap because omg master red)08:19
mvozyga: oh?08:19
zygamborzecki: tests/lib/prepare-restore.sh08:19
mvopstolowski: is 7771 ready?08:19
mvoI still want to branch 2.43 :)08:19
zygamborzecki: look at line 21508:19
zygawe undo lxd changes08:19
zygathat's only once on 1st boot08:19
zygamaybe that has consequences?08:19
pstolowskimvo: thanks for asking.. it is, but master is broken08:19
zygamborzecki: we do -o remount08:20
zygamborzecki: we don't do -o remount,ro -- maybe that makes it rw?08:20
mborzeckizyga: i don't think so, i have a clean host, after reboot i have rw, issue systemctl daemon-reexec and i have ro now08:20
mvopstolowski: ok08:20
zygamborzecki: so08:20
zygamborzecki: maybe that's just new systemd08:21
zygait does do ro on /sys/fs/cgroup on modern systemd08:21
mvozyga: anything we can do quickly to unbreak master? any test to skip for now until we have a solution?08:21
mborzeckizyga: it's triggered in the test, because we install/upgrade systemd as a build ependency, and postinst runs daemon-reexec08:21
zygamvo: yes, we can disable mount-ns test08:21
mborzeckizyga: also why the :reboot variant works08:21
zygait picked up something weird08:21
zygamborzecki: that's curious, no explanation there08:21
pstolowskican we disable the mount-ns test until a fix is ready to unblock landings? or is there more?08:21
pstolowskizyga: ^08:21
mborzeckipstolowski: yeah, i think so08:21
zygapstolowski: that's one way, yeah08:21
zygamvo: did you see last night chat between jdstrand and stgraber?08:22
mvozyga: I did not08:22
zygathere's some kernel bug drama and broken lxd08:22
zygamvo: please scan that - I bet it affects the releaase08:22
pstolowskizyga, mvo, mborzecki  ok, i'll prep a PR08:22
zygapstolowski: thanksI08:22
mvopstolowski: thank you08:23
zygaI will read the backlog now08:23
mborzeckizyga:  this is the changelog https://paste.ubuntu.com/p/zMvnwTHrDH/08:23
zyganothing scary there08:24
mvohm, looks harmless08:24
mborzeckimeh, there's no log or anythin that systemd remounts /sys/fs/cgroup08:26
zygaijohnson: we don't have nested lxd tests AFAIK08:27
mupPR snapd#7859 opened: tests: disable mount-ns test on 16.04 for now <⚠ Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7859>08:29
mborzeckiso we should probably be seeing ro after the reboot, but it's rw08:29
mborzeckibtw. did you gus notice that the lxd test is not run on ubuntu-19.* ?08:30
pstolowskimborzecki: oh08:31
zygamvo: you have mail08:31
zygamvo: look for a message from stgraber please08:31
mborzeckipstolowski: it lists only this [ubuntu-16*, ubuntu-18.04*, ubuntu-2*, ubuntu-core-*]08:32
zygamvo: we need to revert one line for the next release to unbreak lxd08:32
zygamvo: and to check it we need to install lxd inside lxd08:32
zygamvo: and then run snaps in the 2nd nested lxd to confirm08:32
zygamborzecki: can you expand that to more systems please, it's likely related to the bug08:33
mvozyga: ok08:33
* mvo looks08:34
zygamvo: e7afbc34b1d630aeae4a7d20c34da75f4cb6754608:34
zygamvo: there we need to  remove the "deny unix," rule08:34
zygathat's all08:34
zygathe regression test is nested lxd08:35
zygaI'll be back in 20 minutes, need to handle something at home08:35
pstolowskihi pedronis08:35
mvozyga: uh, yeah, just reading08:35
mvozyga: sucks :(08:35
mvozyga: shows once more that dot releases can't be conservative enough :(08:36
mborzeckipedronis: hey08:36
pedronismvo: seems we need a .5, also master is broken, see #7857 #785908:36
mupPR #7857: tests: update google ubuntu 16.04-64 expected host mount ns <Simple 😃> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7857>08:36
mupPR #7859: tests: disable mount-ns test on 16.04 for now <⚠ Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7859>08:36
zygamvo: kernel bugs are just bugs but yeah08:36
mvopedronis: yes on both counts08:37
mvopedronis: I will take care of this this morning08:38
pedronismborzecki: if I understand correctly we are not sure about 7857 and your are looking into that?08:40
pedronisso we should go for 7859 for now?08:40
mvoI am in favour of 7859 for now to unblock things while it's investigated08:41
pedronismvo: it's fine, trying to understand where we are, this mount-ns failures are starting to get a bit annoying08:42
mborzeckipedronis: yes08:42
pedronismvo: not the most high prio right now, but I did leave comments late last night on your late PR(s)08:43
mvopedronis: thank you so much08:43
mvopedronis: will try to get to it as soon as I can. sorry that it's not easy :/08:44
pedronisnp, let's try to get master green and .5 ready08:44
* pedronis is also not feeling 100% today, not sure why/what08:45
mvopedronis: just read your comment, makes a lot of sense08:45
pstolowskipedronis: yep (about mount-ns test). it's fine to have a test like this but it seems to be picking platform changes which are more frequent than expected, rather than our bugs. this area is unit-tested, so perhaps this test should be run nightly08:48
zygapstolowski: we can also tweak the test to skip certain areas08:50
pstolowskizyga: that's an option too. it should probably be less picky about flags on the filesystems that are not controlled by us08:52
zygapstolowski: it's all a learning exercise, we figure out as we go what is broken08:52
zygamborzecki: do you need a hand on that flag mystery or can I focus on something else?08:54
mupPR snapd#7261 closed: interfaces/serial-port: support pci bus serial-port with HotplugKey() <⛔ Blocked> <Created by anonymouse64> <Closed by pedronis> <https://github.com/snapcore/snapd/pull/7261>09:04
sdhd-saschaChipaca: would it be ok, to build spread snap on core18 ?09:11
Chipacasdhd-sascha: why do i get the impression you're breadth-first digging all the rabbit holes?09:12
sdhd-saschaChipaca: just for learning. You mention yesterday, that the snap needs a update.09:13
zygabrb, rebooting09:13
Chipacasdhd-sascha: the updated snapcraft.yaml I have here is base:core1809:14
pedronismborzecki: we are getting failures on selinux-clean09:15
pedronisthat are failing on the disable mount-ns PR09:16
pedronispstolowski: ^09:16
sdhd-saschaChipaca: does your snapcraft.yaml has kvm inside ? then i could took this and experiment.09:17
Chipacasdhd-sascha: it does not, no :)09:17
Chipacasdhd-sascha: i can push what i have here so you start from it09:18
sdhd-saschaChipaca: ok. Yesterday, i only added git and gcc for core18 on multipass/qemu.09:19
pstolowskipedronis, mborzecki mhm.. looking09:19
mborzeckipstolowski: i think the policy needs to beupdated09:20
pstolowskimborzecki: yes, i'm just wondering what changed that triggered it\09:20
mborzeckipstolowski: i think it's the interfaces-kvm test that ran before, the kvm inteface writes out a modprobe conf file that loads the relevant kvm driver09:24
zygamborzecki: I'm doing other things, assuming you are good09:24
mborzeckipstolowski: i guess you can restart the travis job, and open a fix in a separate PR (or i'll do it after the meeting)09:24
zygamborzecki: please drag me back if you need assistance on anything09:25
zygatrying to cut the number of my open branches09:25
pstolowskimborzecki: ok, let me try09:27
pstolowskimborzecki: ok, i've the policy update, will give it a try and wait for #7859 (in case it needs to go together)09:42
mupPR #7859: tests: disable mount-ns test on 16.04 for now <⚠ Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7859>09:42
zygapstolowski: thank you for a targeted patch!09:42
pedroniszyga: are you going to force push again against --explain? I might get to do an actual review pass today09:50
zygapedronis: not anymore, I just changed where the spread test runs to limit it to ubuntu classic 6409:51
mborzeckipstolowski: ok, thanks!10:00
mborzeckizyga: btw. another data point, clean cloud image from cloud-images.u.c has ro mode after booting10:00
mborzeckiquick errand, back in 3010:01
zygathat's good, that's what I would expect10:01
mborzeckizyga: makes me wonder, why our spread images have rw, obviously the kernel is different, but i would expect systemd to apply very specific mount options when setting up /sys/fs/cgroup10:02
zygamborzecki: yeah, it does10:02
zygaI read that part of systemd source code10:02
zygamborzecki: two ideas: cloud agent10:02
zygamborzecki: or custom kernel playing a factor10:02
mborzeckizyga: you mean some gcp cloud agent? bc i'm using cloud-init under qemu too10:03
mupPR snapd#7859 closed: tests: disable mount-ns test on 16.04 for now <⚠ Critical> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/7859>10:11
mupPR snapd#7860 opened: selinux: update policy to allow modifications related to kmod backend <Created by stolowski> <https://github.com/snapcore/snapd/pull/7860>10:15
pedronismborzecki: you can try to run the tests with an error in the early prepares and see how the world looks there10:15
mborzeckipedronis: i'm using https://github.com/bboozzoo/spread-mini to spin up the node without any of our test setup10:38
mvomborzecki: which test was it again that caused the selinux denial that was just fixed?10:46
mborzeckimvo: interfaces-kvm10:46
mvomborzecki: ta10:48
pstolowskimvo: i think it's a good idea to have selinux check in kvm-interfaces10:50
mvopstolowski: thank you10:54
sdhd-saschaChipaca: if you have pushed, maybe in a feature-branch. you could inform me. I just cleanup git-history and build a current sway version11:01
Chipacasdhd-sascha: https://github.com/chipaca/spread/tree/update-snapcraft-yaml11:05
zygaI'll be right back, I'll make something warm to drink11:23
mupPR snapd#7771 closed: o/hookstate/ctlcmd: snapctl is-connected command <Needs Samuele review> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7771>11:42
mupPR snapd#7860 closed: selinux: update policy to allow modifications related to kmod backend <Test Robustness> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7860>11:42
mupPR snapd#7861 opened: tests: check for SELinux denials in interfaces-kvm spread test <Simple 😃> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7861>11:47
mupPR snapcraft#2824 closed: Support for go.mod <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2824>12:07
mupPR snapcraft#2832 closed: appstream extractor: take xml comments into account <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2832>12:07
mupPR snapcraft#2822 closed: xattrs: ignore errors if SNAPCRAFT_BUILD_INFO is unset <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2822>12:10
mupPR snapcraft#2830 closed: elf: properly handle corrupted ELF files <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2830>12:10
mupPR snapd#7856 closed: snap-confine: revert, with comment, explicit unix deny for nested lxd <Simple 😃> <⚠ Critical> <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7856>12:28
zygamvo: https://github.com/snapcore/snapd/pull/7856#issuecomment-56251749212:29
mupPR #7856: snap-confine: revert, with comment, explicit unix deny for nested lxd <Simple 😃> <⚠ Critical> <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7856>12:29
zygamvo: do we plan to add this?12:29
mvozyga: add what exactly?12:30
zygamvo: a spread test verifying that this is fixed,12:30
zygamvo: a lxd inside lxd12:31
TuorHi, I just install vlc with snap on a kubuntu 19.10. My mouseppointer changes when it hovers over the vlc window. Can I keep my normal mousepointer somehow?12:31
mvozyga: it's running in master12:31
zygamvo: I don't follow, sorry12:31
zygamvo: are you saying we have that test already?12:31
zygamvo: a test running nested lxd12:31
zygaTuor: hey, this is a known bug12:31
zygaTuor: I can refer you to kenvandine12:31
TuorShall I upvote a bugrequest or something?12:32
mvozyga: sorry in a meeting12:33
zygamvo: I think we don't have that test, please double check that we add one before doing a .512:33
zygamvo: or we may realize more is broken and .6 is required12:33
zygaTuor: I don't know where it is tracked, please ask kenvandine for details12:35
Tuorkenvandine: I encountered a known bug, that my mouse pointer changes when I use the vlc snap. Where can I report a bug or where should I upvote?12:36
mborzeckianyone up for a 2nd review of https://github.com/snapcore/snapd/pull/7821 ?12:37
mupPR #7821: interfaces/seccomp: parallelize seccomp backend setup <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7821>12:37
pstolowskimborzecki: yeah, i'll, was meaning to12:46
=== pedronis_ is now known as pedronis
pedroniszyga: Ian wrote one I think12:46
pedroniszyga: https://github.com/snapcore/snapd/pull/785812:46
mupPR #7858: tests: add nested-lxd test to confirm lxd inside lxd works <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7858>12:47
pedronisit needs reviews12:47
zygapedronis: that's good, we should ensure it passes with updated master12:47
pedroniszyga: does #7830 need jdstrand review? I would say no but open otherwise... or just your re-review12:50
mupPR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830>12:50
mborzeckipstolowski: cool, thnanks!12:55
=== ricab is now known as ricab|bbl
mvozyga: sorry, was in a meeting - 7858 is the one I had in mind13:07
zygapedronis: checking13:08
mvozyga: i.e. we have a test for this13:08
mvozyga: in a PR13:08
mvozyga: but it's incomplete, see PR13:08
mupPR snapd#7862 opened: release: 2.42.5 <Created by mvo5> <https://github.com/snapcore/snapd/pull/7862>13:20
Chipacawoo! got an appointment with the orthopedician! woo13:22
Chipacawith xmas coming up i was sure i was going to get punted into the new year13:23
mborzeckiheh, our test cleanup, or lack thereof, keeps on giving13:29
mvomborzecki: hm?13:39
mvomborzecki: more catastrophies?13:39
mborzeckimvo: yeah, though suprised this one didn't happen earlier, the failure in selinux-context in 7570 is install-socket-activation test leaking socket units apparently13:40
mborzeckiand we're probably missing cleanup for sockt files too, meh13:41
mupPR snapd#7767 closed: tests: run snap-set-core-config on all core devices <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7767>13:51
mupPR snapd#7863 opened: interfaces/builtin: add uio interface <Created by zyga> <https://github.com/snapcore/snapd/pull/7863>13:57
Chipacahuh, firefox crashed trying to get to the meet14:01
mupPR snapd#7864 opened: cmd/snap-mgmt, packaging/postrm: stop and remove socket units when purging <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7864>14:24
pstolowskihttps://github.com/snapcore/snapd/pull/7861 needs 2nd review (trivial)14:35
mupPR #7861: tests: check for SELinux denials in interfaces-kvm spread test <Simple 😃> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7861>14:35
* zyga breaks for food14:46
pstolowskithanks ijohnson !14:47
mupPR snapd#7861 closed: tests: check for SELinux denials in interfaces-kvm spread test <Simple 😃> <Test Robustness> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/7861>14:48
mborzeckioff to pick up the kids15:04
* cachio lunch15:06
pedronispstolowski: thx for the card15:13
pedronisijohnson: will you get to 7768 today?15:14
ijohnsonpedronis: yes I will look at it now; sorry I didn't have time yesterday with the k8s stuff and then this lxd issue15:15
pedronisnp, thx15:15
mupPR snapd#7858 closed: tests: add nested-lxd test to confirm lxd inside lxd works <Test Robustness> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7858>15:17
=== ricab|bbl is now known as ricab
mupPR snapd#7857 closed: tests: update google ubuntu 16.04-64 expected host mount ns <⛔ Blocked> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7857>15:19
* Chipaca goes for tea15:22
ograis there an api call that would trigger a restart of snapd ? i.e. like https://github.com/snapcore/snapd/wiki/REST-API#request-2 but with snapd as the snap ? or is that blocked15:24
zygawhat should I review?15:25
ogra(this is obviously a core18 and beyond question where snapd is its own snap)15:28
ograhmm, k ... i think this answers it ...15:28
ogra$ snap restart snapd15:29
ograerror: snap "snapd" has no services15:29
zygajdstrand: fyi https://github.com/snapcore/snapd/pull/7850/files#r35488895415:31
mupPR #7850: apparmor: allow 'r' /sys/kernel/mm/transparent_hugepage/hpage_pmd_size <Simple 😃> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7850>15:31
pedroniszyga: #7830 needs your re-review15:32
mupPR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830>15:32
zygaI'm  doing that now :)15:33
zygapstolowski, pedronis: https://github.com/snapcore/snapd/pull/7830/files#r35489113615:35
mupPR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830>15:35
pstolowskizyga: ty15:35
pstolowskibtw i'm looking at your uio iface15:35
zygacool :)15:35
zygapstolowski: one thing to keep in mind is that it must grant mmap access to /dev/uioN15:36
zygaI fixed that but missed it in the initial testing because the test tool didn't require that15:36
zygapstolowski: https://github.com/snapcore/snapd/pull/7830#pullrequestreview-328280697 +115:37
mupPR #7830: interfaces: include hooks in plug/slot apparmor label <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7830>15:37
zygapstolowski: let me know if you want to work on the test device remotely15:38
jdstrandzyga: responded, thanks15:39
zygajdstrand: cool :-)15:41
jdstrandzyga: it was even in the commit missage for snaap-update-ns... not surre why I put it there :)15:41
zygahaha, no worries15:42
zygaI'm happy I asked a meaningful question :)15:42
* ijohnson needs to go downtown for an hour or so, probably will miss tgif15:44
zygatake care ijohnson15:44
ijohnsonzyga: I'll be back! And I realize in retrospect that phrase is a bit loaded, it's just some paperwork things :-)15:51
zygapstolowski: updated https://github.com/snapcore/snapd/pull/7863 -- thank you for the quick review!16:31
mupPR #7863: interfaces/builtin: add uio interface <Created by zyga> <https://github.com/snapcore/snapd/pull/7863>16:31
zygamvo: https://github.com/snapcore/snapd/pull/7825#issuecomment-56264524216:37
mupPR #7825: many: use transient scope for tracking apps and hooks <⛔ Blocked> <Created by zyga> <https://github.com/snapcore/snapd/pull/7825>16:37
mvozyga: oh, it is? sorry, then let's remove it16:38
mvozyga: sorry, too much going on at the same time16:38
zygait's a nop without it16:38
zygano worries16:38
zygaI +1 the idea initially because it feels safe16:38
pedronisit's still not merge ready16:38
zygabut I think it's not unsafe in principle16:38
zygathat's true :)16:38
pedronisit's unlikely to get merge ready by Mon or Tue16:39
zygapedronis: what is missing there?16:40
* zyga EODs and goes for a walk16:55
mupPR snapd#7858 opened: tests: also check nested lxd container <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7858>17:25
ijohnsonmvo / pedronis: should I open a PR for the lxd snap regression test against release/2.42 ?17:26
mupPR snapd#7847 closed: snap-bootstrap: parse seed if either kernel or base are not mounted <UC20> <Created by xnox> <Merged by cmatsuoka> <https://github.com/snapcore/snapd/pull/7847>17:40
cmatsuokacachio: did you notice random test failures on debian recently?17:56
cachiocmatsuoka, do you have a link?17:56
cachiocmatsuoka, didn't see that17:56
cmatsuokacachio: I just restarted it but if it fails again I'll let you know17:57
cachiocmatsuoka, nice, thanks17:57
ijohnsonhey jdstrand you around for a couple minutes to discuss the kubernetes-worker snap ?19:18
sdhd-saschazyga: yesterday, i tried to build a *.deb package for snapd on launchpad.net with a receipe. Is the release-deb also build on launchpad, or ... ?19:22
sdhd-saschazyga: oh, wait... maybe it was build with snapcraft... i looking for it19:25
sdhd-saschaniemeyer: hello, sorry for mention you. Just try to build a *.deb package of snapd for testing. https://code.launchpad.net/~sdhd/+snap/snapd-daily19:40
sdhd-saschaBut there was no place to configure the target ppa ...19:41
sdhd-saschaInside of the snap-build process, there i found: dpkg-buildpackage: source package snapd20:00
sdhd-saschazyga: My fault. I think i found it20:14
jdstrandijohnson: I am here. what's up?20:29
jdstrandijohnson: (sorry I missed the initial question)20:29
ijohnsonno it's okay I don't think I asked the actual question here20:29
ijohnsonso I have this kubernetes-worker snap here and I was trying to figure out why the containers launched by containerd can't create their own top level directories because AppArmor denies it20:30
ijohnsonit was odd because docker containers are allowed to do this, and after a while it occured to me that docker will perform a profile transition to the docker-default profile which doesn't constrain the container by AppArmor for top level directories, etc.20:31
ijohnsonafter looking some more, it appears that containerd doesn't by default transition the containers to an apparmor profile, but I was wondering what your thoughts are if we somehow got the kubernetes-worker snap to do that?20:31
ijohnsondo you think it's okay to have containers launched by the kubernetes-worker snap be confined by the docker-default profile20:32
jdstrandijohnson: the profile was designed with this in mind. it is supposed to do this: https://github.com/snapcore/snapd/blob/master/interfaces/builtin/docker_support.go#L16020:34
jdstrandijohnson: when I did the interface initially with microk8s, I made it microk8s load the profile in one of the wrappers so it was able to transition20:35
jdstrandijohnson: I thought we did that with kubernetes-worker as well. I distinctly recall pointing out this should be done. and, iirc, containerd would use the profile if it was loaded20:35
jdstrandijohnson: but wouldn't load it itself. I might be misremembering, but I thought that was the case20:36
ijohnsonjdstrand: that might be the case, I'm not sure20:36
jdstrandijohnson: regardless, yes, it is expected that the containerd app can load the profile and transition containers to it20:36
jdstrandhow to make it do that is another thing... :)20:37
ijohnsonI'm not running the full setup, I'm just trying to drive containerd manually and noticed it doesn't do this by default20:37
ijohnsonok, well my question for you was if this is okay, I assumed it was because it's in the policy but wanted to be sure :-)20:37
jdstrandijohnson: let me see if that code is in microk8s...20:37
jdstrandijohnson: it is not only ok, it is recommended and best practice :)20:38
jdstrandijohnson: it does mean that containerd is more privilged, but the containers are less, and the container attack surface so what's most important20:38
ijohnsonjdstrand: I don't see anything in the kubernetes-worker snap to load that profile20:39
ijohnsonjoedborg: any idea on where that bit of code might have went in the kubernetes-worker snap?20:39
jdstrandwhich is why I clal the docker (and containerd) policy 'advisory', since they can load prolicy20:39
jdstrandand transition to it20:39
joedborgijohnson: do you mean this bit? https://github.com/charmed-kubernetes/snap-kubernetes-worker/blob/master/wrappers/containerd.wrapper#L1320:43
jdstrandijohnson: https://github.com/ubuntu/microk8s/blob/feature/strict-v2/microk8s-resources/containerd-profile and https://github.com/ubuntu/microk8s/blob/feature/strict-v2/microk8s-resources/wrappers/run-containerd-with-args#L1120:43
jdstrandjoedborg: if you run 'sudo aa-status', you see cri-containerd.apparmor.d listed?20:44
joedborgno, i don't seem to20:48
joedborgjdstrand: ^20:48
ijohnsonjoedborg: ah yes I didn't see that for some reason20:48
jdstrandjoedborg: I think there is something wrong then with the startup. that profile needs to be loaded every time containerd starts20:49
joedborgijohnson: it is missing from the eks branch since yesterday because i was getting errors20:49
jdstrandalso, fyi I found https://kubernetes.io/docs/tutorials/clusters/apparmor/20:49
ijohnsonjdstrand: with the eks branch that joedborg showed me yesterday I see that profile being loaded into the kernel20:50
jdstrandthat hints at defaults and things and might provide some insight if the pods still aren't running in cri-containerd.apparmor.d once it is confirmed that it is loaded in the kernel20:50
ijohnsonso I think that parts fine, it seems now that the issue is just in configuring $CONTAINER_TECHNOLOGY to actually run under that profile20:51
jdstrandijohnson: ok, then if it is loaded, you might be able to use the above link to make k8s put it in that profile. there is probably something somewhere for defining  the default profile to use20:51
jdstrandijohnson: ok, cool, and yes20:52
ijohnsonjoedborg: have you looked at https://aws.amazon.com/blogs/opensource/using-pod-security-policies-amazon-eks-clusters/ ?20:52
jdstrand(from the above link)20:53
ijohnsonthat seems to imply that the pods metadata stuff from the generic kubernetes stuff can be configured in EKS20:53
ijohnsonthanks jdstrand I saw that link earlier today too20:53
jdstrandijohnson: nice find20:53
ijohnsonBTW word of caution: don't go looking for containerd's CLI tool "ctr" docs or examples because they don't exist haha20:53
ijohnson(I was trying to be sneaky and just skip the k8s part and just reproduce the issue with ctr, but that doesn't seem to be the way to go)20:54
jdstrandoh yeah, I had some trouble with that in the past20:54
joedborgijohnson: i did briefly, but saw that `Read Only Root Filesystem:              false` so moved on20:55
joedborgijohnson: jdstrand: ctr is also a massive trap because it behaves completely differently to anything else that drives containerd (all over CRI)20:56
ijohnsonoh hmm so does the psp stuff work by some aws agent running on the pod to enable these things? if so that's really unfortunate20:56
jdstrandijohnson: I have this in my notes when I did a seccomp security update: https://paste.ubuntu.com/p/P6M4hhkphw/20:56
jdstrandijohnson: but based on what joedborg just said, you may want to ignore that :)20:56
ijohnsonjdstrand: yes `runc spec` is a good trick until you realize how much other config settings $OTHER_CONTAINER_TECHNOLOGIES set when they drive CRI like joedborg just said20:57
* jdstrand nods20:57
jdstrandthis was just for the deb20:58
jdstrandijohnson: I like when you say $CONTAINER_TECHNOLOGY and $OTHER_CONTAINER_TECHNOLOGIES20:58
jdstrandit is just right ;)20:58
ijohnson:-) there's just so so so many of them20:59
jdstrandindeed :)20:59
joedborg`apparmor_parser -r $SNAP/containerd-profile` i took this out because i started getting `permission denied` when it runs the wrapper21:02
joedborgeven if i put sudo in front of it21:02
joedborgso i think an apparmor rule (to allow to run this) has gone missing21:02
joedborgthat's using the custom snapd21:02
jdstrandjoedborg: you can't put sudo in front of it21:03
jdstrandjoedborg: can you: sudo snap run --shell snap.kubernetes-worker.conttainerd21:03
jdstrandjoedborg: then apparmor_parser -r $SNAP/containerd-profile21:03
jdstrandjoedborg: and ive me the denials?21:04
jdstrandjoedborg: (from journalctl)21:04
jdstrandijohnson, joedborg: not, I have a very hard stop in 15 minutes, but could circle back21:04
joedborgjdstrand: okay, i've got that now21:11
jdstrandjoedborg: I'm confused. I thought you said from within the snap it didn't work?21:12
joedborgjdstrand: it wasn't, then i ran it manually like you suggested and it did21:13
jdstrandjoedborg: I was wanting you to run apparmor_parser -r $SNAP/containerd-profile from under sudo snap run --shell21:13
jdstranddang it21:13
joedborgbut `Dec 06 20:54:31 ip-192-168-35-165 kubernetes-worker.containerd[18362]: /snap/kubernetes-worker/x1/bin/containerd.wrapper: line 13: /sbin/apparmor_parser: Permission denied`21:13
jdstrandjoedborg: you ran it from under snap run --shell?21:13
jdstrandjoedborg: ok, between that and that ijohnson said it loaded for him, I'm going to chalk that up to the docker-support interface wasn't connected at the time of that denial21:15
jdstrand/sbin/apparmor_parser ixr,21:15
jdstrandthat is in the docker-support policy ^21:15
joedborgjdstrand: ah yes, quite possibly21:15
joedborgeither way, i'm still getting the RO filesystem errors21:16
jdstrandjoedborg: I didn't look at the code for the kubernetes-worker, but be sure that this apparmor_parser invocation is not in a configure hook or under some conditional where it only sometimes loads the policy21:16
jdstrandjoedborg: for the rofs issues, this is where ijohnson can come in21:17
joedborgjdstrand: it's with the containerd wrapper, so run everytiime containerd is (re)started21:17
jdstrandjoedborg: perfect21:17
ijohnson(in meeting)21:17
joedborgjdstrand: is that still looking sane? ^21:18
jdstrandjoedborg, ijohnson: ok, I need to head out for my appt, but I'll circle back and see if there is anything for me to do21:18
joedborgjdstrand: +1 thanks!21:18
jdstrandjoedborg: it looks like a containerd default profile, yes21:19
jdstrandsane is in the eye of the beholder :)21:19
jdstrandbut jokes aside, yes21:19
joedborgjdstrand: perfect :)21:19
jdstrandjoedborg: so, if there are apparmor denials, ijohnson can help you add them to the right places for working around stuff and moving forward. if that happens, I can collect them  and give you a new demo deb. I can do that tonight/over the weekend/etc21:20
jdstrandjoedborg: I'll keep an eye on irc21:21
joedborgjdstrand: sadly, there aren't any.  i think that's the main issue now21:21
jdstrandjoedborg: yes, I just wanted to reiterate that I'll update the demo deb as needed21:21
jdstrandI'm also on tg21:22
joedborgahhh :)21:22
ijohnsonjoedborg: ok I'm back21:49
ijohnsonjoedborg: so we're still at the issue with the filesystem being ro?21:51
joedborgijohnson: yeah21:53
joedborgijohnson: sadly21:54
ijohnsonjoedborg: I'm wondering if you could do something like `watch -n 0.05 bash -c 'sudo aa-status | grep $(pgrep install-aws.sh)'` on the node to see if it shows up under an apparmor profile21:57
ijohnsonerr maybe make that snippet bit smarter so it actually filters properly21:57
ijohnsonone second21:57
joedborgijohnson: +121:58
ijohnsonjoedborg: try `until pgrep install-aws.sh; do true; done; pid=$(pgrep install-aws.sh); sudo aa-status | grep "$pid"`22:03
ijohnsonwatch is a bit difficult to get to work there22:03
joedborgijohnson: i don't appear to be getting any results as the pod starts and fails22:06
ijohnsonjoedborg: what about if you did: `until pgrep install-aws.sh; do true; done; pgrep install-aws.sh`22:08
ijohnsonthat should eventually print off some pid22:08
joedborgijohnson: i still have the same node running if you wanted to watch the byobu22:08
joedborgijohnson: ill try it22:08
ijohnsonunless the name of this script is not install-aws.h22:08
ijohnsonerr install-aws.sh22:08
joedborgijohnson: that latter one works22:09
ijohnsoncool, one sec let me give you something else to run22:09
ijohnsonjoedborg: can you run `until pgrep install-aws.sh; do true; done; sudo nsenter -t $(pgrep install-aws.sh) --all /bin/bash -c "cat /proc/self/mountinfo"`22:10
ijohnsonthat should show us the mount namespace of the process when its running22:10
ijohnsonjoedborg: actually I think I will join that byobu if that's alright22:12
ijohnsonmight make this quicker22:12
joedborgijohnson: of course :)22:12
joedborgijohnson: fyi i have 2 tabs inside it open22:12
* ijohnson googles how to switch tabs in byobu22:13
joedborgF3 and F422:13
mupPR snapcraft#2834 opened: dirs: support --user install on Linux <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2834>22:53
mupPR snapcraft#2835 opened: colcon plugin: support ROS 2 Eloquent <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2835>23:50

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!