[05:31] morning [06:09] google:ubuntu-18.04-64:tests/main/uc20-snap-recovery seems to fail randomly [06:38] PR snapd#7650 opened: o/ifacestate: unify code into autoConnectChecker.addAutoConnections [06:45] PR snapd#7651 opened: asserts: support and parsing for slots-per-plug/plugs-per-slot [06:51] mborzecki: hi, should we disable temporarely google:ubuntu-18.04-64:tests/main/uc20-snap-recovery it seems indeed flaky [06:52] pedronis: i'll try to find out what's going on there [06:52] ok [06:53] pedronis: can you take a look at https://github.com/snapcore/snapd/pull/7643 ? should be easy to land, it's just moving the code around into separate files [06:53] PR #7643: overlord/devicestate: refactor and split into per-functionality files, drop dead code [06:53] mborzecki: yes, finishing something else and then I will look [06:54] pedronis: cool, thank you === pstolowski|afk is now known as pstolowski [07:00] morning [07:00] Hey [07:00] Overslept [07:00] Including kids [07:01] Sorting out stuff [07:05] hey guys [07:06] hey mborzecki [07:09] mvo: quick question about uc20-snap-recovery test, lsblk is wrapped by retry-tool, the comment says it's a race with udev, i've also seen it fail randomly in the travis runs [07:09] mvo: the race is in udev noticind the new partitions and creating device nodes, while lsblk picks them up, right? [07:12] PR snapd#7652 opened: o/ifacestate,interfaces,interfaces/policy: slots-for-plug: * [07:12] mvo: yeah, I think so, we could call udevadm I guess [07:12] mborzecki: [07:12] mborzecki: I'm doing a spread run right now to validate this [07:15] mvo: all greedy plugs PRs are up now [07:15] pedronis: \o/ [07:15] pstolowski: hi, I asked you a review in #7650, that's definitely an area you worked on/touched [07:15] PR #7650: o/ifacestate: unify code into autoConnectChecker.addAutoConnections [07:16] pedronis: sure, will do [07:16] thx [07:19] mvo: heh, obviously spread -repeat=20 did not catch this [07:19] mvo: repeated that a couple of times, same thing [07:19] mvo: wonder if maybe there's some cross test interaction after all [07:21] mborzecki: could be, I can also just mount the partitions directly in the test and not rely on lsblk [07:22] mvo: hm, otoh, the nodes under /dev are created by the kernel now [07:22] mvo: not all though, but /dev/loop*p* probably is [07:30] it failed again fwiw also in 7650 [07:31] meh, I change from lsblk to mount for now [07:31] * mvo does that right away [07:31] sorry guys [07:37] pedronis: restarted the job in 7650, there's nothing new in the log, tests that ran earlier don't seem to be related [07:41] pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/7647 ? [07:41] PR #7647: gadget: skip structures with MBR role during remodel === pedronis_ is now known as pedronis [07:42] mborzecki: reviewed #7643 [07:43] PR #7643: overlord/devicestate: refactor and split into per-functionality files, drop dead code [07:45] pedronis: thanks! i think we could return to making the split nicer once gadget remodel (and maybe base remodel) land [07:46] yea, I would also like to clean up some stuff in managers_test.go but yes, after more remodel stuff lands seems the right call [07:46] mborzecki: sure [07:54] zyga: tiny suggestion under https://github.com/snapcore/snapd/pull/7623 also the test caught something [07:54] PR #7623: tests: check world-writable and test-owned files [07:54] zyga: probably needs a master merge too [08:03] mborzecki: tests caught real bug, it is discussed in the PR [08:04] PR snapd#7653 opened: tests: do not use lsblk in uc20-snap-recovery test [08:14] mvo: left a little comment https://github.com/snapcore/snapd/pull/7653#discussion_r338436144 [08:14] PR #7653: tests: do not use lsblk in uc20-snap-recovery test [08:15] mborzecki: oh, nice! we probably want both, i.e. know we can mount it but also get all this nice information that file provides [08:16] mvo: surprisingly there's not that many methods for probing the fs type of a block device :/ [08:16] mborzecki: yeah, I'm also puzzled why lsblk is not reliable, I try to look at the code of lsblk [08:54] Chipaca: updated my old store-url branch [08:54] Chipaca: will be pushing that out soonish [08:55] * zyga -> tea [08:58] mborzecki: neat [09:24] Chipaca: hm haven't added it to aux info though, we have a website and media list there [09:27] jdstrand: Ping re https://bugs.launchpad.net/snapd/+bug/1849291/comments/2 – can I be of any help to get this into snapd soon(ish)? Sorry for nagging, but our snap pipeline is currently blocked for pressing updates, I'd need to revert a slew of NM-related changes or cherry-pick to another build branch until core allows Introspectable access for NM. [09:27] Bug #1849291: Interface network-manager should grant access to org.freedesktop.DBus.Introspectable [09:27] (Note to self: Don't assume NM access works on Core when working fine on Classic desktop 🙈) [09:31] PR snapd#7654 opened: cmd/snap, store: include snapcraft.io page URL in snap info output [09:31] Chipaca: ^^ [09:49] mborzecki: yeah that'd be better,but could be a follow-up [09:53] pedronis, mvo sorry for not getting to #7595 yesterday, but I'm looking at it now, is there a design doc for the things there I should be looking at it? is the model assertion doc the only thing? [09:53] PR #7595: seed/seedwriter: support writing Core 20 seeds (aka recovery systems) [10:00] ijohnson: there's that and there's a picture from a sprint, let me turn it into a doc quickly [10:01] ok thanks [10:09] also could someone restart https://travis-ci.org/snapcore/snapd/jobs/601379249 again for me? it failed on store 403s again :-/ [10:16] ijohnson: done [10:16] \o/ [10:17] ijohnson: you don't have travis permission to restart? [10:17] I used to but it went away yesterday, I am still a GitHub member of snapcore but Travis doesn't think so anymore [10:17] ijohnson: maybe you need to log back in? [10:17] hmm I could try clearing the cache too I suppose [10:18] heh [10:18] signing out and back in doesn't do anything unfortunately [10:19] i recently got told to clear all my cookies to debug some website [10:19] "hahaha no" [10:19] "ALL your cookies are belong to us now" [10:19] there's so much state in dem biscuits [10:19] huh well now it works when I go sign in fresh to another browser [10:21] quick errand, back in 30 [10:23] also Chipaca does the meeting time I sent out today work for you? [10:24] ijohnson: both of 'em work :-) but yes [10:24] ijohnson: i didn't check, is pedronis also there? [10:25] Chipaca yes and pedronis accepted the most recently sent invitation [10:26] ijohnson: are you saying i should actually click accept on the invite [10:26] psh [10:26] RSVPs is for frenchies [10:27] next time I'll mail you a physical invite so you can't decline it [10:27] you underestimate my power [10:31] can I enjoy some schadenfreude over the "FIREFOX UPDATED WITHOUT MY KNOWLEDGE AND STOPPED RESPONDING THIS IS TERRIBLE WILL YOU PLEASE FIX IT oh wait it was the apt version never mind" thing [10:43] re [10:49] PR snapd#7655 opened: interfaces: allow introspecting network-manager on core [10:50] dot-tobias: ^ [10:52] PR snapd#7646 closed: tests: update mount-ns after addition of /etc/systemd/user [10:54] PR snapd#7643 closed: overlord/devicestate: refactor and split into per-functionality files, drop dead code [10:55] PR snapd#7656 opened: tests: verify host is not affected by mount-ns tests [10:57] zyga: pstolowski: is either of you aware of / working on #1848516 and #1849564 ? [10:57] Bug #1848516: snap connections/interfaces shows dropped interfaces as connected after refresh [10:57] Bug #1849564: snap connections doesn't work as expected when interface attributes change [10:57] Chipaca: I'm aware of the first one, working on neither [10:57] checking the second one [10:57] mvo: with the pulsaudio transition in progress, these bugs are going to bite [10:58] zyga: variation thereof [11:03] Chipaca: uh, thanks for raising this [11:16] zyga: you commented on this, so I ask you: what other projects have bugs for us? I'm adding them all to the triage doc [11:16] Chipaca: that's a great question, let me think [11:17] yassss finally silly spread tests passed [11:17] PR snapd#7597 closed: overlord/snapstate: add LastActiveDisabledServices, missingDisabledServices [11:17] zyga: I'm stepping outside for a while, feel free to edit the doc :-) hopefully it makes sense wrt where to add [11:17] Chipaca: I will [11:17] enojy [11:18] enjoy* [11:23] pstolowski: we should look at those connections bugs [11:24] pstolowski: let me wrap something up first though [11:26] I see some searching and apt-hooks failures [11:26] zyga: interesting, I will investigate later today [11:26] 403 store side [11:26] I see a bunch of (random?) systemctl restart systemd-journald.service failures. anyone else is seeing this in the tests? [11:27] mvo: yes I see this constantly [11:27] cachio has a PR which should fix this [11:27] snap recovery failures [11:27] mvo: all the time [11:27] zyga: I'm trying to fix those [11:27] mvo: as in every day for as long as I can remember [11:27] mvo: thank you so much [11:27] zyga: but now I see failures in the other ones [11:27] zyga: so I can't land my fix :/ [11:27] mvo: https://github.com/snapcore/snapd/pull/7605 is what cachio had to fix that I think [11:27] reliability of the test suite is such an important part of snapd [11:27] PR #7605: tests: configure the journald service for core systems [11:27] zyga: 7653 fwiw [11:28] mvo: you always see it in ubuntu-core right? [11:28] or elsewhere? [11:28] I always only see it on ubuntu core [11:28] ijohnson: correct [11:30] ijohnson: lookng now [11:31] ijohnson: the PR from sergio needs a core update to writable path or some other mean to set this :/ [11:32] ah darn, is there anyway we could hack around that? [11:32] zyga: i suspect a problem around reloadConnections, afair it is not robust enough. I’ll check later, need to wrap up first pare-Bakun PR [11:32] mvo: assuming https://bugs.launchpad.net/snapd/+bug/1849291 is fixed now, should it be targeting 2.43? [11:32] Bug #1849291: Interface network-manager should grant access to org.freedesktop.DBus.Introspectable [11:32] *pre-bake [11:32] some old milestones on launchpad ought to be closed [11:32] I'll do that [11:35] yay [11:35] two milestones left [11:35] mvo: and if so, what is the estimated release for 2.42 and 2.43? [11:36] mborzecki: can you look at https://github.com/snapcore/snapd/pull/7614 [11:36] PR #7614: cmd/snap-confine: implement snap-device-helper internally [11:36] PR snapd#7585 closed: spread.yaml: drop exclude list, use .gitignore <⛔ Blocked> [11:36] It'd like to start moving this forward a little [11:37] PR snapd#7657 opened: snap: fix default-provider in seed validation [11:38] ijohnson: is the restart issue only happening on core18? [11:38] ijohnson: or did you see it on core16 as well? [11:38] mborzecki: also https://github.com/snapcore/snapd/pull/7656 is very simple and could land soon [11:38] PR #7656: tests: verify host is not affected by mount-ns tests [11:39] mvo: I saw this issue (journal service restart) on pretty much all systems, but this is an unscientific assertion] [11:40] mvo: pretty sure I've seen it on core16 as well [11:41] ijohnson: yeah, I'm just going over the last couple of failures and there it is [11:44] I remember I have a note about the fact that we should simply drop some connections [11:44] on refresh [11:44] ijohnson: let me see if I can fix sergios pr [11:47] core 20 recovery fails [11:47] is that fixed or shall we disable it [11:48] pedronis: yes, we never implemented transitions in any way, where a snap can rename a plug or slot or actual dropping of a plug or slot where that is removed from the state [11:48] I need some coffee, it's such a sleepy day [11:48] sorry, brb [11:50] zyga, jdstand, mvo: Hearty thank you for the fast help + fix re: NM Introspectable! One more thing to look forward to in the 2.43 release 😊 [11:50] *jdstrand ^ [11:51] ijohnson: I pushed a PR to core [11:52] PR core#109 opened: extra-file: make /etc/systemd/journald.conf.d writable === ricab is now known as ricab|lunch [11:59] ack mvo will take a look today then [12:03] dot-tobias: a pleasure :) [12:03] mvo: once that lands I need to adjust mount-ns test to compensate [12:07] PR snapd#7658 opened: cmd/snap-start-pressing: add snap-start-preseed executable (1/N) [12:08] pedronis: i've updated https://github.com/snapcore/snapd/pull/7630 [12:08] PR #7630: overlord/devicestate: check snap handler for gadget remodel compatibility [12:09] ijohnson: hey, 7431 has conflicts [12:32] mborzecki: addd to my queue [12:32] thx! [12:37] pstolowski: yes I finally just merged the dependent one, will resolve asap after my meetings this morning [12:42] Chipaca: pushed changes to aux data too [12:52] mborzecki: reviewed changes to aux data too [12:52] Chipaca: cool, thanks! [12:53] Chipaca: something i haven't checked, when is the aux info refreshed? bc if the store were to change the urls for any reason, would this end up being out of date? [12:54] mborzecki: currently it's bumped on refresh, although my plan is to do it on checking for refreshes also [12:54] Chipaca: so it should be fine then [12:55] mborzecki: that is: the refresh action fetches a bunch of data that we currently completely ignore for snaps that aren't actually refreshed, but that can and should change [12:55] mborzecki: we need to run the numbers but it's quite possible it'll be less expensive to get minimal data on refresh, and do separate info calls for what actually changed === alan_g is now known as alan_g_ [12:56] but that's something to do when we have free time [12:56] * Chipaca looks at work mapped out to 2040 [12:56] … yeah, someday [12:57] is there a forum thread so we can put our wishlist items on your 2040 schedule ? [13:02] ogra: ? [13:03] pedronis, i was reacting to Chipaca's snarky joke :) [13:03] ah [13:03] didnt mean to scare you :) [13:05] snarky? me?!? === ricab|lunch is now known as ricab [13:30] * zyga goes for lunch [13:44] PR snapd#7653 closed: tests: do not use lsblk in uc20-snap-recovery test [13:47] mborzecki: 7640 is ready for a second review [13:56] jdstrand, the new daemon user is hardcoded to "snap_daemon" ? (i.e. i can not set a random username here ?) [14:01] ogra: yes [14:03] morning all! I've been seeing some odd snapcraft errors since late last week. They seem to point at missing libraries, but I get different results part to part and for different libraries. Could anyone lend their eyes please? It's blocking some work. [14:03] ogra: you may only use snap_daemon at this time [14:04] joedborg: maybe #snapcraft ? [14:04] joedborg: or are they snapd-related? [14:05] thanks, i'll try there. i guess it's snappy as it's happening at build time [14:06] off to pick up the kids [14:11] dot-tobias: it might make sense to verify the changes in that PR are enough to solve your issue. the way to do that would be a) install the two snaps b) in /var/lib/snapd/apparmor/profiles/snap.network-manager.networkmanager add the receive rule from the PR, substituting the plug's name (eg, snap.mirros-one.hook.post-refresh) [14:11] dot-tobias: c) in /var/lib/snapd/apparmor/profiles/snap.mirros-one.hook.post-refresh add the send rule from the PR, substituting the slot's name (eg, snap.network-manager.networkmanager) [14:12] dot-tobias: d) load the profiles into the kernel with: sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.network-manager* /var/lib/snapd/apparmor/profiles/snap.mirros-one* [14:14] re [14:15] hello jamie :) [14:15] I'll push forward with bugfixes tonight [14:16] (spread permitting, sigh) [14:23] jdstrand: https://github.com/snapcore/snapd/pull/7655 updates as requested [14:24] PR #7655: interfaces: allow introspecting network-manager on core [14:25] sergiusens: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/u/ubuntu-image/20191022_062343_f039f@/log.gz [14:25] this needs updating for 20.04? [14:25] mvo: ^^^ ? [14:26] doko: looks like it, yes - sergiusens will have a look, he is in a meeting right now [14:26] doko: I worked that out on #ubuntu-release already, or is this a trigger after that? [14:27] sergiusens: is it already fixed? [14:28] PR core#109 closed: extra-file: make /etc/systemd/journald.conf.d writable [14:30] doko: a retrigger should fix it if sil2100 or infinity haven't done so yet [14:31] doko: I reported my side of the fix one hour and 20 minutes ago on #ubuntu-release ... I cannot retrigger tests for ubuntu-image [14:33] zyga, cachio new core with fix for journal.d is building [14:33] mvo, perfect [14:35] zyga: hi and thanks! [14:36] mvo: thank you, I will prepare the fix for mount-ns test [14:41] pstolowski: I skimmed the preseed PR, it looks good overall, my main comment will probably be naming related [14:42] pedronis: ty! [14:48] mvo: hello! Once you find a few cycles today, could you formally sign-off on the uc20 u-i spec in the doc? ;) [14:49] Since I'm getting poked about that constantly, especially now that I want to land the changes [14:59] sil2100: yes, today is meeting crazy, I try [15:01] mvo: thank you! [15:03] mvo: I made a suggestion comment in #7649 [15:03] related to what we discussed [15:03] pedronis: thank you! [15:03] PR #7649: overlord: fix TestRemodelSwitchToDifferentKernel for bootvars [15:09] sil2100: mvo: I could probably look at it an sign it off instead of mvo if that helps [15:10] pedronis: that would be great [15:10] pedronis: *if* you have some minutes, but hopefully straightforward [15:13] mborzecki: any news on the f30 xerrors backport? [15:13] mvo: nope, filed a ticket, pinged the guy irc, but haven't heard back [15:14] pedronis: I'd be +1 on that too ;) [15:14] pedronis: left an initial review on #7595, need to finish reviewing the tests in a followup review I will submit later this morning [15:14] PR #7595: seed/seedwriter: support writing Core 20 seeds (aka recovery systems) [15:14] sil2100: I made two additions, as suggestions, do they match our last understanding [15:14] ? [15:15] pedronis: yes, approved those o/ [15:15] Thanks for adding them [15:20] ijohnson: thanks, I'll answer some of your questions in my morning [15:20] sounds good, have a nice evening (also I scheduled a meeting with you tomorrow morning re: performance things) [15:22] sil2100: I filled in the table at the top with my +1, let me know if I should put in something else, not sure how you do this [15:25] pedronis: that's perfect, thanks a lot o/ I'll inform my manager and land the u-i bits tomorrow - we can always follow up with fixes later when we're able to test it properly [15:30] * zyga is sleepy and goes to bed [15:45] ijohnson: I did a pass of answering already [15:45] pedronis: ack will take a look [15:48] PR snapcraft#2767 opened: rust plugin: add rustup profile [15:55] * cachio lunch [16:06] ppisati, so i just tried to roll a kernel snap from the raspi2 tree of eoan ... using the rsapi2 defconfig (to use it on my pi4 core images) ... and i notice that i end up with an ~800MB snap !! lookinng inside (unsquashed it is even 2GB) i see 976M in modules ! [16:09] do we really need all this on a pi ? [16:10] (and how did it grow so much since bionic ?) [16:12] $ snap info rpi-iptv-player [16:12] Killed [16:12] bah ! [16:14] @saviq is there a way to tell multipass to use all available instead of using a single core ? [16:14] om26er: --cpu `nproc` when launching [16:14] jdstrand, do we suddenly have auto-rebuilds enabled for outdated snaps ? [16:14] om26er: snapcraft has a hidden env var for it, too [16:15] Saviq, cool that was going to be my second question :-) [16:15] https://github.com/snapcore/snapcraft/blob/master/snapcraft/internal/build_providers/_multipass/_multipass.py#L99 [16:15] om26er: vote on https://github.com/CanonicalLtd/multipass/issues/756 to have a setting for the defaults ;) [16:15] hmm, it seems to default to 2 apparently [16:16] jdstrand, i just looked at one of my pi's and had an update this morning of one of my packages that i definitely havent touched in more than a month ... and snap info also shows "edge: 0.2 2019-10-24 (56) 42MB -" [16:16] om26er: that's threads, not necessarily cores [16:16] (i.e. with HT that might end up just one core) [16:17] Saviq thanks for info, I'll subscribe to that issue === pstolowski is now known as pstolowski|afk [16:17] ogra: i don't work on ARM kernels anymore -- juergh and hwang4 are taking care of rpi kernels now [16:18] ppisati, ah .. [16:18] well, let me just express that i'm slightly shocked then ... seeing my kernel snaps triple up in size :) [16:18] ogra: while shrirang and jesse work on snapdragon [16:18] ogra: eh, i see what you mean [16:19] ogra: we can probably reduce the size / number of kmods [16:19] yeah i guess so [16:19] ogra: thow a bug in LP about that, and we can take a look [16:19] will do ... not urgent indeed, i believe official pi4 core support is only on schedule for core20 anyway [16:30] zyga: do you think you could give a second review to 7640 ? [16:30] $ snap stop lxd [16:30] error: cannot communicate with server: Post http://localhost/v2/apps: dial unix /run/snapd.socket: connect: connection refused [16:30] GRRRR !!! [16:36] zyga: when you get back I have some bad news for you :-( https://github.com/snapcore/snapd/pull/7547#issuecomment-546001237 [16:36] PR #7547: many: use a dedicated named cgroup hierarchy for tracking <⛔ Blocked> [16:37] I'm looking into it and I'm hoping there's an easy way out for this [16:38] * Chipaca takes a break [16:47] mvo: I reviewed some of your PRs === ijohnson is now known as ijohnson|lunch [17:02] roadmr: hey, can you pull 20191024-1700UTC? [17:02] jdstrand: sure! [17:03] roadmr: thanks :) [17:04] 1700 how precise ;) [17:04] roadmr: just overrides and a change to the README. I used the new debian/changelog methodology as discussed before [17:04] roadmr: yeah :) [17:04] thanks! jdstrand fwiw the guys who had complained about that said having the changelog from source was fine, so yay [17:05] roadmr: cool :) [17:07] ogra: if something is rebuilding that, it isn't something I am aware of [17:09]