[05:14] o/ [07:02] mornings [07:32] Hey Paweł [07:32] Quite a quiet [07:32] Morning :-) [07:33] Making coffee, I somehow could not wake normally [07:34] seems there's a new gccgo version in bionic and we are hitting a bug that is fixed only in newer dh-golang [07:35] gccgo tests have started failing on 18.04 [07:42] mmh [07:48] pedronis: oh, do you have a bug reference? [07:49] shall we disable 18.04 for the weekend so that we can keep pushing things? [07:49] we get this: https://paste.ubuntu.com/p/GVvz4qq4hY/ [07:49] there's a debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=907263 [07:49] pedronis: perhaps we can pin the previous version of the package if it is still in the archive [07:50] thank you, looking [07:50] that mentions dh-golang 1.35 to have the fix, but bionic has still only 1.34 [07:50] or we can add a ppa with dh-golang 1.35 perhaps [07:51] Have you reported the critical SRU regression? [07:52] not yet [07:52] it took me a while to understand it was not new code [07:52] but new package [07:52] version [07:53] doko: ^^ [07:53] hi all [07:54] Do all snap packages need manual review or is it only because the warning showing? [07:55] joelkraehemann: hello, what warning is that? [07:55] joelkraehemann: normally snap package do not require a manual review [07:56] zyga: great to know. I was using experimental layouts but for now I pass the environment variable $AGS_SNAP_PATH [07:56] ... and no more layouts [07:56] joelkraehemann: yeah that _may_ trigger a manual review for now, I'm trying to get that to be enabled by default [07:57] what do you think, when is it enabled by default? [07:57] I'd like to do that for the 2.36 release [07:57] there's a PR for that [07:58] some people are going to be off today so I don't know if it will land yet [07:58] probably monday [07:58] nice [07:58] pedronis, wgrant: so shall we file a bug for this and wait? [08:00] zyga: File a bug, tag it with regression-update, and ping the uploader and/or release team on IRC [08:00] s/release/SRU/ [08:00] kk [08:00] https://wiki.ubuntu.com/StableReleaseUpdates#Regressions [08:01] thank you! [08:04] moin moin [08:05] mwhudson: o/ [08:05] pedronis: so gccgo is the faulty package? [08:05] * zyga tries to phrase the bug report [08:05] mwhudson: if we have 1.10 in 16.04 etc, that still wouldn't pull gccgo equivalent in, right? [08:05] zyga: https://bugs.launchpad.net/ubuntu/+source/dh-golang/+bug/1794936 [08:05] Bug #1794936: new gccgo version update () is triggering dh-golang bug

[08:06] ah, thanks, [08:06] morning [08:07] zyga: it's not a bug on gccgo but things will fail with dh-golang unless we also get a new dh-golang [08:07] pedronis, wgrant: I added the regression-update tag to your bug [08:07] starting later than usual, had to run an errand [08:07] mwhudson: maybe I should ask this question on the bug [08:07] zyga: I tried to mention both packages, but gcc packages are a maze, not sure I picked the right one [08:08] yeah, I was lost at gcc-defaults, dh-golang is easier [08:08] Retargeted at gcc-8 [08:08] Which was the SRU that broke things [08:11] pedronis: should we skip gccgo tests on 18.04 until this is sorted? [08:11] PR snapd#5868 closed: tests/main/snap-env: extend to cover parallel installations of snaps [08:11] PR snapd#5872 closed: tests/main/parallel-install-local: rename from *-sideload, extend to run snaps [08:12] Chipaca: I suppose [08:12] zyga: can you take a look at https://github.com/snapcore/snapd/pull/5870 ? [08:12] PR #5870: tests/main/parallel-install-services: add spread test for snaps with services [08:13] is michael off today? [08:13] pedronis: I think so [08:13] mborzecki: yes === KaitoDaumoto is now known as Guest13744 [08:14] pedronis: I've asked him in private [08:14] pedronis: confirmed [08:14] Chipaca: a 2nd review of #5824 would be appreciated [08:14] PR #5824: fetch the device store assertion together and in the context of interpreting snap-declarations [08:16] zyga: thanks! [08:18] ppisati, your kernel patch works great ! [08:20] PR snapd#5880 opened: interfaces/repo: two helper methods for houtplug [08:24] pedronis: should the assertion fetchers get a context at some point? [08:24] to be cancelable i mean [08:24] maybe, at some point [08:25] pedronis: maybe as part of the move to bulk [08:26] tbh this code with functions getting functions that get passed functions is not super easy to follow :-) [08:26] Chipaca: we have a bigger problem, in this sense that we fetch assertions from sync places [08:26] PR snapd#5870 closed: tests/main/parallel-install-services: add spread test for snaps with services [08:26] (for good reasons but it will not end well at some point) [08:27] mhmm [08:27] not that I'd spotted it, but yes it sounds like trouble [08:27] it's totally preexistent [08:28] and it's a potential issues with hundreds of snaps [08:28] * Chipaca hopes popey isn't listening [08:28] *ears prick up* [08:28] anyway we really have a problem that is either sync or changes [08:29] we don't have an inbetween to do things [08:29] what's up? [08:30] popey: talking about things that can't ever happen, ever ever [08:30] popey: unless somebody had hundreds of snaps [08:31] alan@hal:~$ snap list | wc -l [08:31] 234 [08:31] pedronis: which sync things do this? [08:31] Who would do such a thing? [08:31] Chipaca: snap refresh [08:31] popey: IKR [08:31] pedronis: ah, it does it as part of pre-changes work? [08:31] yes [08:32] since forever [08:32] shouldn't be too hard to move it into a change then [08:32] it is [08:32] he says, like an idiot [08:32] * Chipaca grins [08:32] remember why we moved pre-flight checks out of changes [08:32] it would undo that [08:32] TBH, I don't remember why, but I'm sure it's a good reason [08:33] (and I can re-remember it by reading, probably) [08:33] anyway, this isn't landing PRs [08:33] to be fair it was specifically about downloads [08:33] so maybe there's a why [08:33] but is not a simple change [08:33] s/why/way/ [08:34] hmm ... does the prepare_device hook of a gadget snap only run at the end after all snaps have been processed ? ... when i set a hostname from that hook and have avahi installed, the name resolution only works after a reboot [08:35] ogra: it was not defined exactly [08:35] ah [08:35] it might have been changed recently tough [08:35] I need to go look [08:35] should the install hook run earlier ? [08:36] which install hook? [08:36] of the gadget [08:36] ? [08:36] yeah [08:36] i see the gadget beinmg processed before the app snaps [08:36] yes [08:36] so perhaps thats the better choice [08:36] ok [08:36] it's core kernel gadget apps [08:37] that is well defined [08:37] somehow the name tricked me that prepare_device runs before anything else :) [08:37] will use install then [08:43] ogra: with 2.35 or so prepare-device is strictly after seeding [08:43] yeah, that explains the behaviour [08:44] Chipaca: zyga: do you remember whether we have anywhere bash code that make choices based on comparing snapd version in the spread tests? [08:45] * zyga thinks [08:45] apart from debian control scripts I don't think we do [08:46] pedronis: we have tests that fail if the version is not what's expected, but I don't think that's what you mean [08:46] I would like to do something in upgrade/basic only if the old snapd is newer than 2.35 [08:46] it actually might be the prepare bit of the test tbf [08:47] but writing might be a pain [08:47] the issue really affects only debian in practical terms (the relevant change is actually older than 2.35) [08:49] pedronis: i thought we had a compare-versions in there but can't find it [08:49] thought the same [08:49] but not clear how to write portably except with python or something [08:49] (we can't assume dpkg anymore) [08:54] pedronis: 'snap version --greater-than "$otherversion"'? [08:54] maybe python is easier :-) [08:55] by definition the old snap will not have that [08:55] anyway I might just do something blunter [08:56] Chipaca: thx for the review [08:56] np [09:24] PR snapd#5881 opened: tests: disable gccgo tests on 18.04 for now, until dh-golang vs gccgo are fixed [09:24] Chipaca: zyga: ^ [09:27] ack [09:50] pedronis: Error executing google:ubuntu-16.04-64:tests/main/interfaces-accounts-service [09:50] Error: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Type of message, '(ssssa{ss})', does not match expected type '(sssa{sv}a{ss})' [09:52] Chipaca: looks like the elusive error we see very sporadically on spread tests [09:53] mhmm [09:53] pstolowski: go away you're sick [09:53] (i saw it a couple of times over last few months) [09:56] travis is slooow [09:59] wow, I hadn't realise we were doing ~1k commits per release [09:59] sounds like a lot === scroll is now known as hfjvjffju [10:01] zyga: commented on #5867 [10:01] PR #5867: many: enable layouts by default [10:01] * Chipaca takes a break [10:05] pstolowski: thanks [10:13] * zyga -> walk [10:27] snappy is using golang? [10:34] doko: yes [10:49] zyga: please ask mwhudson about that [10:55] doko: about what? [10:56] PR snapd#5881 closed: tests: disable gccgo tests on 18.04 for now, until dh-golang vs gccgo is fixed [10:57] woohoo [10:57] * Chipaca merges master [10:59] yess [11:00] travis now experiencing stampeding herd of snapd PRs [11:45] pedronis, (not sure you are the right person, but i'll ask anyway) is there a way for me to obtain the model name from a hook in the gadget (i would like to set the default hostanme to the model name on frst boot) ... without having snapd-control ? [11:46] (would "snapctl known model" work without extra interface ?) [11:51] PR snapcraft#2302 closed: requirements.txt: stop using pymacaroons-pynacl [11:51] PR snapcraft#2303 closed: [legacy] requirements.txt: stop using pymacaroons-pynacl [11:55] ogra: not atm [11:55] there are some argument that we could have snapctl ack and snapctl known at least for gadgets but they are not there atm [11:56] hmm, ok [11:56] thats sad ... since it means i'll need one gadget per image type [11:57] thanks anyway ! [12:13] sergiusens: so in theory we have the new logic in place in both edge and stable snaps now, the next upgrade will still be bad for you (because it's running the old stop logic) but hopefully you'll be good after that [12:14] stgraber: nice, can I see the PR that implements this? Just for curiosity [12:14] I am glad I raised it then too [12:14] sergiusens: https://github.com/lxc/lxd-pkg-snap/commit/82950207244f445ad9c59ab22967ab20c002bd25 [12:31] hmmm, snapcraft tells me this: The linker version '2.23' used by the base 'core' is incompatible with files in this snap: [12:33] kyrofa: hey, any idea ^? i'm not doing anything with bases in the snap i'm building. snapcraft installed from snap, i presume it's the base of the snapcraft? [12:34] stgraber: looks much more robust [12:35] pstolowski: it means you are probably not building on 16.04 [12:35] PR snapd#5882 opened: many: support and consider store friend-stores when checking device scope constraints [12:37] sergiusens: yep and the type field really shouldn't be translated :) [12:37] sergiusens: indeed, i'm on 18.04 [12:47] popey, Wimpress: hi! can one of you comment on https://forum.snapcraft.io/t/requesting-approval-for-goreleasers-classic-snap/6571/9? [12:47] jdstrand: hey [12:47] jdstrand: gentle ping for https://github.com/snapcore/snapd/pull/5395 [12:47] PR #5395: interfaces: generalize writable mimic profile [12:47] zyga: I know, sorry [12:48] sure, I understand [13:01] Chipaca: standup? [13:01] omw [13:28] jdstrand: I commented [13:29] jdstrand: fwiw, it was elopio and myself who were involved in that [13:31] PR snapd#5824 closed: many: fetch the device store assertion together and in the context of interpreting snap-declarations [14:06] sergiusens: ah, thanks :) [14:07] Wimpress, popey: sergiusens responded to https://forum.snapcraft.io/t/requesting-approval-for-goreleasers-classic-snap/6571/12, but can one of you vet the publisher? [14:10] done [14:16] zyga: found a snapd bug in the cosmic live session https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1794953 [14:16] Bug #1794953: GNOME Calculator snap has wrong theme in live session

[14:16] zyga: permission denied errors mounting the content interface [14:19] https://pastebin.ubuntu.com/p/nYmtwkYWXp/ :-D [14:20] niemeyer: new category, or should I put them into "other"? ^ [14:20] * niemeyer look [14:20] s [14:20] we still have space for a few more categories, fwiw [14:20] kenvandine: hey [14:20] Chipaca: That seems to fit nicely into a "Snapshots" one [14:20] yerp [14:20] Chipaca: Nice tests, btw [14:21] kenvandine: interesting, can you add the apparmor denials please? [14:21] niemeyer: :) [14:21] sure [14:22] I need coffee, brb [14:24] PR snapd#5833 closed: asserts,interfaces/policy: add support for on-store/on-brand/on-model plug/slot rule constraints [14:24] zyga: added to the bug [14:25] zyga of particular interest is the content interface for gnome-3-26-1604 must be mounting just fine [14:25] it's only gtk-common-themes [14:29] re! [14:29] man [14:29] I put the powdered coffee beans into the mug [14:30] Sep 28 14:08:52 ubuntu kernel: [ 68.310240] audit: type=1400 audit(1538143732.840:235): apparmor="DENIED" operation="open" profile="snap-update-ns.gnome-calculator" name="/upper/snap/" pid=6535 comm="3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [14:30] that's the thing [14:30] well, I can look at that next week [14:31] zyga: thanks, that bug is a cosmic release blocker [14:31] zyga: just FYI [14:31] yeah, no pressure ;) [14:31] it's not an LTS, right? ;-) [14:31] lol [14:32] willcooke: ^^ FYI zyga's on the case [14:32] thanks chaps [14:32] I assigned it to myself [14:33] well [14:33] kenvandine: while I have you [14:33] do you have the session open? [14:33] yes [14:33] can you poke at one file please [14:33] let me give you the details [14:35] can you go to /var/lib/snapd/apparmor/snap-confine [14:35] and tell me what you have there [14:35] there _should_ be a file called "overlay-root" there [14:35] yes [14:35] kenvandine: can you also share /proc/self/mountinfo from the live session [14:35] and if the file is there, pastebin the file please [14:38] http://paste.ubuntu.com/p/PhsbzsCzZ8/ [14:40] zyga: and just to confirm, there is just one file in snap-confine, overlay-root [14:40] hmmm [14:40] that's very cool [14:40] but ... [14:40] something new in the kernel apparently [14:40] what's in the overlay-root file? [14:40] look at the mount table you pasted [14:40] the Wally is /cow [14:40] where's Wally? [14:41] is there a /cow directory? [14:41] overlay-root has "/upper/{,**/}" r, [14:41] there is no /cow [14:42] 29 1 0:24 / / rw,relatime shared:1 - overlay /cow rw,lowerdir=//filesystem.squashfs,upperdir=/cow/upper,workdir=/cow/work [14:42] ! [14:42] right? [14:42] so what's /cow? [14:42] I mooost know [14:42] no idea :) [14:43] a sample denial is: [14:43] Sep 28 14:08:52 ubuntu kernel: [ 68.309885] audit: type=1400 audit(1538143732.840:232): apparmor="DENIED" operation="open" profile="snap-update-ns.gnome-calculator" name="/upper/snap/" pid=6535 comm="3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [14:43] it makes milk usually :P [14:43] but the pattern you pasted should cover that [14:43] (shake it and you have cream !) [14:43] :) [14:43] kenvandine: can you look at /var/lib/snapd/apparmor/profiles and look for /upper there? [14:43] zyga: and the mount for gnome-3-26-1604 is working fine [14:43] in the snap-update-ns.gnome-calculator profile? [14:44] jdstrand: ^ FYI, not sure if you know about any changes to overlayfs between 18.04 and 18.10 [14:45] kenvandine: please collect all those log files in the bug report [14:45] zyga: there is no occurrences of upper in the profile [14:45] maybe as files for easier access [14:45] oh! [14:45] you mean that snap-update-ns.gnome-calculator has no /upper in it? [14:45] right [14:45] zyga: I do not. this is on the livecd? [14:45] yep [14:45] zyga: yes, live session [14:45] kenvandine: that's the bug perhaps [14:45] works fine installed [14:46] well, someone needs to adjust the policy for that. it me, it will be a little while, but someone can submit a pr and I'd look at it [14:46] kenvandine: please add the file you looked at to the bug report [14:46] s/it me/if me/ [14:46] jdstrand: not sure if there's something to adjust yet [14:46] it looks more magic than before [14:46] niemeyer: should I add SnapID to snapshots before, together with, or after landing the commands? [14:47] jdstrand: anyway, I just wanted to check if you were aware of any changes to overlayfs [14:47] kenvandine: I'll break for dinner now, ok? [14:47] zyga: ok [14:47] i'll attach stuff to the bug [14:47] niemeyer: there's already an after-landing tasklet of blocking epochs, and restoring configs, both of which aren't done [14:47] could be part of that [14:48] Chipaca: Sounds fine either way.. as long as we don't release in the wild before the disk format is stable [14:48] Chipaca: We can add an experimental flag for that, if necessary [14:49] niemeyer: I can mark the PR blocked and just not land it until the followup is ready as well [14:49] mmm [14:49] dunno [14:49] niemeyer: I'll do something sensible. Thanks. [14:51] Chipaca: Thanks for that :) [14:58] * Chipaca takes a break [15:04] ogra@localhost:~$ snap interfaces [15:04] error: no interfaces found [15:04] GRRRRRR !!!! [15:04] ogra: eh, you didn't need them anyway [15:04] * Chipaca really tries to take that break now [15:06] Chipaca, well, you guys make setting a hostname from a gadget hook really really hard ... i switched from prepare_device to the install hook of the gadget (because prepare_device only runs after the device hads been prepared which makes it completely useless for that case) ... and that got me into that state [15:06] seems snapd simply stopped seeding the system completely now [15:08] ogra@localhost:~$ snap list [15:08] No snaps are installed yet. Try 'snap install hello-world'. [15:08] :( [15:10] what does snap changes says? and snap tasks for the ones there [15:10] ogra@localhost:~$ snap changes [15:10] ID Status Spawn Ready Summary [15:10] 1 Error today at 14:52 UTC today at 14:54 UTC Initialize system state [15:10] 2 Error today at 14:53 UTC today at 14:53 UTC Initialize device [15:10] 3 Error today at 15:01 UTC today at 15:02 UTC Initialize system state [15:10] 4 Error today at 15:01 UTC today at 15:02 UTC Initialize device [15:10] 5 Error today at 15:09 UTC today at 15:10 UTC Initialize system state [15:10] Error today at 15:08 UTC today at 15:09 UTC Run install hook of "pi-kiosk" snap if present [15:12] pedronis, that install hook calls "/usr/bin/hostnamectl set-hostname ..." the gadget.yaml has the right connect statement and the snapcraft.yaml allows the hostname-control plug for the install hook [15:12] ? [15:12] connect is called later [15:13] ugh [15:13] you really need an auto-connect for that [15:13] connections doesn't make a lot of sense for those kind of interfaces [15:13] any chance the configure hook would be late enough ? [15:13] re [15:13] i really just need to set a hostname before avahi comes up for the first time [15:13] I would need to check but unlikely [15:13] sigh [15:14] we set up connections after everything else [15:14] anyway they are not a good fit for this [15:14] we're making this really hard [15:14] this is really more a auto-connect case [15:14] which would work [15:15]