[05:15] morning [05:16] mborzecki: good morning [05:16] zyga: hey [05:17] mborzecki: election results, uuuuh [05:17] zyga: not a surprise really [05:18] hey mborzecki and zyga [05:18] zyga: i'm happy sejm is more diversified, but wan's really expecting a significant power shift [05:18] hey mvo [05:18] mborzecki: did you see the new results this morning [05:18] mborzecki: pis has almost 50 now [05:20] mvo: hey [05:21] zyga: those aren't final yet [05:21] mborzecki: exactly :( [05:43] PR core-build#57 opened: many: drop static files for "core" snap from the package [05:44] mborzecki, zyga: can I get a review for this please? and for https://github.com/snapcore/core/pull/83 [05:44] PR core#83: move most of the ubuntu-core config deb into the snap snap build [05:45] mvo: can you take a quick look at https://github.com/snapcore/snapd/pull/7587 ? [05:45] without one review from a team member I cannot merge [05:45] PR #7587: spread: generate delta when using google backend [05:45] mborzecki: sure [05:47] I'll be around in 15 [05:50] PR core-build#48 closed: config: remove static files in etc,lib,usr,var [05:51] mborzecki: is school cancelled for your kids as well? [05:56] PR snapd#7587 closed: spread: generate delta when using google backend [05:58] mvo: left some comments under https://github.com/snapcore/core/pull/83 [05:58] PR core#83: move most of the ubuntu-core config deb into the snap snap build [05:58] mvo: i suppose it was like this before too [06:00] mborzecki: yeah, these files just move in this PR - happy to do a followup with fixes [06:07] mvo: +1 [06:07] mborzecki: \o/ [06:11] mvo: hm the travis job is red anyway [06:13] mborzecki: yeah, I need to tweak the ppa:snappy-dev/image archive as right now the builds are fighting over which content is correct [06:14] mborzecki: the coresponding one in core-build (57) is what is needed in the ppa to properly build [06:50] mborzecki: if you could also please look at https://github.com/snapcore/core-build/pull/57 [06:50] PR core-build#57: many: drop static files for "core" snap from the package [06:50] mborzecki: its the coresponding one for the one you reveiwed before [06:50] PR core#83 closed: move most of the ubuntu-core config deb into the snap snap build [06:53] eoan woes :/ [06:53] moving back to vm === pstolowski|afk is now known as pstolowski [07:02] mornings [07:03] hey pstolowski [07:09] hey pawel [07:11] PR core-build#57 closed: many: drop static files for "core" snap from the package [07:11] PR core#107 opened: extra-file: drop restorecon from sshd-host-keygen [07:13] pstolowski: hey [07:16] PR core#108 opened: extra-files: add /var/home to make snaps work on some distros [07:16] thanks for that [07:17] got to run an errand, back in a bit [07:17] zyga: my pleasure [07:17] PR core18#141 opened: static: add /var/home to make snaps work on some distros [07:42] PR snapd#7594 opened: tests: replace "test-snapd-base-bare" with real "bare" base snap [07:47] zyga: also https://github.com/snapcore/bare-base/pull/1 (which I think mup is not monitoring yet) [07:47] PR bare-base#1: Makefile: add /var/home to make snaps work on some distros #141 [07:55] re [08:01] one sec [08:02] hmm [08:17] : [08:17] mvo: can you take another look at https://github.com/snapcore/snapd/pull/7536 ? it's unclear to me whether we're still missing something there or not [08:17] PR #7536: gadget: accept system-seed role and ubuntu-data label [08:18] mborzecki: in a meeting, but happy to do so later [08:18] mvo: cool, thanks! [08:21] zyga: http://whatthecommit.com/index.txt [08:51] Chipaca: looking [08:51] hello btw :) [08:51] Chipaca: will you tell if I uses that :D ? [08:51] brb [08:51] let me get coffee [08:55] coffee sounds like a good idea [09:01] heh [09:01] my woes with eoan on suspend [09:01] translate to same woes in vmware on suspend [09:01] (vmware suspends VMs to save power if asked to by asking the os to suspend) [09:10] PR snapd#7595 opened: seed/seedwriter: support writing Core 20 seeds (aka recovery systems) [09:34] * zyga runs spread _and_ gets that coffee he wanted [09:55] do you guys think it's legit to poke snapd api when a snap service is being stoped? [09:56] Chipaca I think I found another 20 seconds in first boot :0 [09:56] Chipaca or more :) [10:16] ondra: hey, i'm working on pre-baking of firstboot stuff, with the plan to shave off those most expensive ops [10:17] pedronis: btw i saw your comment to Friday's standup notes, do you have a moment today to discuss? [10:23] ondra: tell me more :-) [10:24] pstolowski sure [10:24] pstolowski: ondra is finding things we can do that'll help current core16/18 though so as long as it's not disruptive it's good imho [10:25] Chipaca reading logs, we are trying to refresh catalogue from store, at very early stage. I'd think this is operation do not need to do pre-seeding [10:25] Chipaca it's so early in the boot, we have no network, so it keeps timing out. [10:26] ondra: you're saying skipping that saves us 20+ seconds? [10:26] yeah, snapd really shouldnt assume network ... we have many wlan equipped boards that first need wifi setup [10:26] Chipaca in our case there will be never network a this stage ( wireless only net interface, image has no baked in AP credentials) [10:26] Chipaca yep [10:26] ondra: funny you mention this as it's currently erroring/making noise in my pre-baking code (precisely because of no network in the chroot env) and i was about to look at what to do about it [10:27] ondra: you can trick it to skipping that step fwiw [10:27] Chipaca https://paste.ubuntu.com/p/73vXpjW3jf/ [10:28] pstolowski Chipaca I'd argue that we have not even parsed seed.yaml, what do we actually need to refresh, or what do we expect to gain from it? [10:29] Chipaca pstolowski https://paste.ubuntu.com/p/3f2ZPHrtmt/ [10:29] this shaved me 20 seconds and no noise [10:29] ondra: if you drop in a snapd.service.d file with [Service] ExecStartPre=touch /var/cache/snapd/names, that should have it skip that catalog refresh [10:29] pstolowski Chipaca does it sounds like something we can assume? [10:30] ondra: or that :-) [10:30] ondra: but, if you do that, can you make seeded trigger a catalog refresh? [10:30] Chipaca I was thinking to simply check if we are seeded, and if not, assume this is first boot [10:31] Chipaca you mean to trigger refresh after it's seeded? [10:31] ondra: that would mean no catalogues for the first day of the device [10:31] ondra: yes, please [10:31] ondra: it's called "refresh" but they don't exist before the first refresh succeeds [10:31] Chipaca this is first shot, I'm sure this can be done better way [10:32] ondra: fair enough [10:32] Chipaca but are you sure it works this way? because this refresh cannot succeed for many reasons [10:32] ondra: right, it failing means it tries again sooner [10:33] Chipaca there are no snaps installed, so what is it actually refreshing? if this is brand store it should have no way to access store anyway [10:33] Chipaca we only get serial assertion once seeded [10:33] ondra: right but you're changing code that affects everybody :) [10:33] pfft [10:33] who cares about the others [10:33] Chipaca then let's make it way it works way we want :) [10:34] Chipaca this was my first shot to see if I can use seeded as detection of first boot [10:34] Chipaca now we need to decide what to do next, ideally I was thinking to schedule refresh in say 10 minutes or something [10:35] Chipaca but I think you guys know first boot sequence way better than me, so you might have better idea [10:36] does refresh uses deltas ? could we pre-seed it with data at build time so it only needs to do a delta update ? [10:36] I do not know if we even know at that point number of snaps we are about to seed, so we can guess delay [10:36] ogra what refresh? this pings server before we have any snap installed [10:36] at that point, there zero snaps to check for refresh [10:37] i thought it queies the store and fills a local db [10:37] *queries [10:38] ogra: catalog refresh is not refresh [10:38] oh dear looks like i'm lagged [10:38] ah, ok [10:38] (just thinking about cutting down the amount of data to request and transfer to cut down time) [10:39] ogra you want to refresh before you have even first version, ever heard "trying to run before you can walk" [10:39] :P [10:39] ondra: so, what we want at some point is to move catalog-refresh to a task, and then trigger that task from the right places/times [10:40] mvo: some packaging changes may be needed https://forum.snapcraft.io/t/stop-commands-and-snapd-package-cleanup/13688 [10:40] Chipaca that sounds good [10:40] ondra: easy peasy :-p [10:40] ondra, no, i want to pre-fill a db and only apply a delta so you dont need the full db data [10:41] Chipaca I'd do test boot to see refresh time, because we might trigger refresh when we get serial assertion [10:41] ogra I think we do not get db delta, as we are only pulling assertions we care about and relevant to the system [10:42] ondra: meanwhile your approach is a good step in the right direction. I'd suggest moving the check earlier, even, to where the CanAutoRefresh check is [10:42] ogra but I might be wrong [10:42] i'm probably tricked by the name "catalog-refresh" then :) [10:42] Chipaca I was not brave to put it that early, you say it's safe, then indeed better place [10:43] ondra: putting it super early means it gets to try again every ~5 minutes or so [10:43] ondra: that's what we want :-) [10:43] Chipaca ah yeah indeed that's good one [10:44] then once seeded it'll do the catalog refresh, and set the next catalog refresh time and back off [10:46] Chipaca yep, it seems it triggered refresh once it got serial assertion, even with current change [10:47] zyga: packaging change where exactly? in lxd? or snapd? [10:47] Chipaca I just did fresh boot with change I had, so we have good trigger post seeding as well [10:47] Chipaca OK I will update change based on your suggestion and prepare PR [10:48] Chipaca we can then iterate there [10:48] Chipaca thanks for help :) [10:48] ondra: thank you! [10:49] mvo: in snapd [10:55] zyga: mvo: yeah, the situation is puzzling [10:56] maybe let's talk after/during standup? [10:56] i guess there's a reason why it's done in postrm and not prerm like other distros [10:59] mborzecki: after sounds ok [10:59] PR core18#141 closed: static: add /var/home to make snaps work on some distros [11:00] wow, there are distros using /var/home as default ? crazy ! [11:01] ogra: silverblue i think [11:01] (i used to do that on servers with /var being the data partition ... but would never have imagined a distro doing this by default) [11:31] * pstolowski lunch [11:35] PR snapd#7596 opened: Skip catalog refresh on unseeded system [11:36] Chipaca pstolowski https://github.com/snapcore/snapd/pull/7596 [11:36] PR #7596: Skip catalog refresh on unseeded system [11:37] pstolowski I also created this PR, to run assertion check in parallel https://github.com/snapcore/snapd/pull/7590 [11:37] PR #7590: seed: seed16: run adding snaps in parallel [11:45] brb [11:53] 1 [11:56] * zyga needs to leave the house and think [11:56] or work anywhere but here [11:56] I cannot stand the smell anymore (kitchen nearby) [12:11] Bug #1650738 changed: Scan network failure error after first reboot [12:14] Bug #1649840 changed: unknown keys in model assertion are silently ignored [12:14] Bug #1651722 changed: Latest candidate snap breaks running snapcraft in classic snap [12:14] sil2100, hey, did you see my update to https://github.com/CanonicalLtd/ubuntu-image/pull/175 ? [12:14] PR CanonicalLtd/ubuntu-image#175: Little kernel bootloader support [12:20] * Chipaca goes for a short walk === ricab is now known as ricab|lunch [12:54] abeato: hey! Yes, let me look at that and action in a bit, I'm on the release sprint right now [12:56] mvo: I'm on a slow network, I cannot join the standup with video today [12:56] not sure if audio will work, I'll do my best [12:57] zyga: no worries [13:00] sil2100, cool, take you time, thanks [13:01] I cannot connect [13:01] Eoan and modem manager :-/ [13:26] Bug #1749538 changed: refresh time docs lacks the correct command [13:32] pedronis: wrt #1659153, I don't think there's a bug beyond the impedance mismatch (which could be alleviated by blocking private+name) [13:32] Bug #1659153: /v2/find with select=private has different behaviour for queries and name searches [13:33] pedronis: OTOH I'm not sure what store bug you mean :-) [13:33] pedronis: snap find --name hits the store's 'info' endpoint [13:33] not the search one [13:39] mvo: pedronis: ok, added a note in the forum about the prerm thung we agreed on [13:41] pedronis: can you take another look at claudio's PR https://github.com/snapcore/snapd/pull/7536 ? i'm not sure there's more to do there, and if we land it then i could start looking into https://github.com/snapcore/snapd/pull/7593 [13:41] PR #7536: gadget: accept system-seed role and ubuntu-data label [13:41] PR #7593: recovery-tool: add sfdisk wrapper [13:47] mborzecki: I'll try to look today or tomorrow morning at that PR [13:47] pedronis: great, thanks! === ricab|lunch is now known as ricab [13:58] I managed to somehow connect [13:59] at this rate I will run windows with WSL to get a machine that works :| [14:02] * Chipaca sends zyga a raspberry pi [14:16] zyga: what's wrong with eoan? [14:16] mborzecki: suspend resume kills my input [14:16] mborzecki: then does something really odd to my modem [14:16] ended up powering off [14:16] and then powering on [14:17] then I can connect [14:17] reboot is not good enough [14:17] gnome-shell connection menu gets very confused when this happens, so I really reboot out of a habit now [14:18] zyga: btw. i think i asked abut it already, is the input and modem connected via usb? [14:19] mborzecki: I don't know [14:19] mvo: packaging question: should /usr/bin/snap be a symbolic link to /usr/lib/snapd/snap? [14:19] mvo: in our debian/ubuntu packages [14:20] ANy help with hacking on snapcraft please? [14:21] mborzecki: I don't think the touchpad / trackpoint is using usb [14:21] I'm trying to look at https://bugs.launchpad.net/snapcraft/+bug/1841861 but a day later I still haven't managed to figure out how to run snapcraft from the source tree :-/ [14:21] Bug #1841861: Python plugins fails to build a snap when some parts depends on unpublished modules [14:22] mborzecki: my knowledge ends at xinput that says: [14:22] SynPS/2 Synaptics TouchPad [14:22] snapcraft seems to have its own plugin discovery and it's a plugin I'm trying to modify [14:22] TPPS/2 IBM TrackPoint [14:22] But it's not using PYTHONPATH, and looking in the pip-installed area first. [14:22] (having followed HACKING.md) [14:22] kenvandine: ^ actually, do you know how to debug xinput things going away on suspend [14:23] kenvandine: on my laptop suspend kills the trackpoint/touchpad until reboot [14:23] And I'm about five levels deep in indirection trying to figure out how to get snapcraft to look in the source tree :-/ [14:24] zyga: maybe, whats the advantage of doing it? [14:25] mvo: /usr/lib/snapd is enough to get all the new binaries [14:25] mvo: IIRC we do something like that on core18 [14:26] zyga: hmm same on x250, not listed in lsusb [14:26] anyway [14:26] back to work [15:01] rbasak: I think the actual snapcraft devs such as cjp256 and sergiusens are off today, but also note that there is an additional channel for snapcraft specifically #snapcraft [15:02] Joined, thanks! [15:16] abeato: can we get https://snapcraft.io/docs/gadget-snap updated with the new names too? [15:16] sil2100, sure, let me edit [15:16] sil2100, ah, I thought that was the forum [15:17] sil2100, I don't know who needs to change that one [15:18] Ah, the forum one too! I never know which one is the official one in the end [15:19] pedronis: do you want to review 7593 or do you think a review from maciej and me will be enough? [15:20] mvo: I probably will not do a deep review, but I should look at it [15:20] sil2100, anyway, I've edited the forum one: https://forum.snapcraft.io/t/the-gadget-snap/696 [15:20] mvo: also is recovery-tool called recovery-tool? shouldn't it be snap-recovery ? [15:21] where will it get installed? [15:22] I asked in the PR itself [15:25] ijohnson, hi! do you know where is the latest version of the dpdk/hugepages interfaces? [15:25] abeato: you mean the one that luke worked/is working on? [15:25] yes [15:26] one minute let me look [15:26] Chipaca: I made a meta-comment in #7589, would like your input [15:26] pedronis, fyi I have edited https://forum.snapcraft.io/t/the-gadget-snap/696 , updating the role names for lk partitions as was decided [15:26] PR #7589: cmd/snap: add ability to register "snap internal" commands [15:26] ijohnson, k [15:27] pedronis: I think snap-recovery is more approprate, yes [15:28] abeato: thank you [15:28] abeato: last I heard from Luke, https://github.com/snapcore/snapd/pull/5885 was up-to-date, I helped him a bit with the dpdk snap here: https://github.com/anonymouse64/dpdk-snap/tree/wip [15:28] PR #5885: Adding DPDK interface for DPDK Snap [15:29] err up-to-date insofar as that's all the work Luke had on it, not necessarily that it was complete [15:29] ijohnson, ok, thanks for the pointers [15:48] Wrapping up for now. Time to eat something [15:51] abeato: merged your PR, thanks again! We should have it in edge soonish [15:51] There's still one thing we want to add before we promote this further [15:51] Hopefully having it in edge will be enough for now? [15:54] sil2100, awesome, thank you! [15:54] sil2100, sure, no hurry for me === pstolowski is now known as pstolowski|afk [20:27] zyga: sorry, I was out today for a holiday. I'll catch up with you in the morning === jdstrand_ is now known as jdstrand [20:34] joedborg: hey, I just saw this: 16:03 < joedborg> `[285928.025967] audit: type=1400 audit(1570741250.597:1384598): apparmor="DENIED" operation="signal" profile="docker-default" pid=703 comm="kubelet" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.kubernetes-worker.kubelet"` [20:35] joedborg: kubelet shouldn't be talking to a 'docker-default' profile, it should be talking to the one that containerd sets up [20:36] joedborg: oh, I read that backward. why is a docker container sending signals to kubelet? [20:49] jdstrand: joedborg: I don't remember the specifics, but IIRC the child profile (i.e. the container) needs to signal to the parent that it is done, I think that's something runC does [20:49] also welcome back jdstrand :-) [20:50] ijohnson: thanks, right, but the kubernetes-worker snap has a containerd with a different profile name than 'docker-default', so I'm confused why a profile named 'docker-default' is in play at all [20:52] joedborg: eff, I am a little slow today. that denial says that a process running under a 'docker-default' profile name was sent 'kill' by snap.kubernetes-worker.kubelet, and that was blocked because the docker-default profile doesn't allow receiving signals from snap.kubernetes-worker.kubelet [20:52] erf* [20:53] joedborg: so, why is kubelet talking to a container running under the 'docker-default' profile? the snap should be configured for it to talk to a containerd profile [20:55] jdstrand, well regardless of why it's named docker-default, the container is only allowed to transition to a docker-default profile from `docker-support` interface or to the `systemd-run` one from `k8s-support` interface if I'm reading those correctly [20:55] jdstrand: so if it started running under containerd-default we would have to change the apparmor transition rules wouldn't we? [20:57] ijohnson: the interfaces were adjusted for this before I left and there were no denials and no docker-default profiles on the system. I'm wondering what has changed. like, did something get dropped from the snap packaging? is the packaging moving to docker instead of containerd? something else? [20:58] ijohnson: ie: [20:58] # defaults for containerd [20:58] change_profile unsafe /** -> cri-containerd.apparmor.d, [20:58] signal (send) peer=cri-containerd.apparmor.d, [20:58] ptrace (read, trace) peer=cri-containerd.apparmor.d, [20:58] ijohnson: it all worked fine. this is for kubelet to *send* signals. the denial is about the container to *receive* them though [20:59] ijohnson: and there were patches to cri-containerd.apparmor.d in the kubernetes-worker package to allow this [20:59] (and for said profile to be named 'cri-containerd.apparmor.d', not 'docker-default' [20:59] ) [21:00] so I'm confused why 'docker-default' is the profile name [21:01] the snap wasn't changed in 17 days according to github. I need more context from joedborg [21:02] * jdstrand wonders if the control plane has a mix of docker and containerd. that would be weird... [21:04] Hey jdstrand can we pick it up tomorrow please? I’ve got today off. The eks-support branch in GitHub is up to date but I don’t think it’s very relevant [21:04] joedborg: yes, of course :) [21:04] joedborg: enjoy the rest of the day :) [21:05] jdstrand: thanks :) I think it’s an issue of stuff in containerd still being constructed by apparmor [21:05] That docker signal may be a red herring [21:05] Constricted * [21:08] joedborg: well, kubelet shouldn't be sending a signal to anything with a 'docker-default' profile name when containerd is spinning up containers under the cri-containerd.apparmor.d profile [21:09] joedborg: unless there is an external docker that is spinning up stuff and the CP is telling kubelet to work with that container. [21:09] there are other things that could be wrong. we can look tomorrow [21:10] jdstrand: yeah you might well be right as it’s a brown field deployment. I’ll take a look in the AM and circle back. Hope you had a good vacation btw [21:20] jdstrand: hmm I guess I never noticed that new containerd transition in docker-support, I only looked at kubernetes-support interface, it's still a bit confusing to me why certain things are in docker-support and not kubernetes-support when the docker snap doesn't use those things [21:20] jdstrand: anyways, for the docker snap specifically, the docker-default profile isn't persistent, it's just created in /tmp somewhere, loaded into the kernel and then the file is deleted, so it wouldn't show up anywhere in the normal apparmor packaging dirs [21:21] jdstrand: but anyways anyways I'll let you take care of this, as I haven't actually built this kubernetes snap and am just theorizing about everything [21:23] joedborg: thanks, it was great [21:24] ijohnson: yeah, I'm familiar with the way docker does things these days, but 'aa-status' would show it since the profile need to be loaded into the kernel for a process to be running under that profile. thanks [21:25] yes aa-status would still show it that's correct, I assumed you meant no docker-default profiles on the filesystem, sorry [21:26] no worries at all :) [22:58] PR snapd#7597 opened: overlord/snapstate: add LastActiveDisabledServices, ComputeMissingDisabledServices [23:17] PR snapd#7598 opened: test/lib/names.sh: make backslash escaping explicit