=== chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk [05:02] morning [06:11] Hey Everton [06:11] Everyone even [06:11] * zyga coffee [06:15] zyga: hey [06:27] zyga: jdstrand: verified that multi arg filtering issue is indeed fixed with the proposed libseccomp-golang package update in fedora 29 === chihchun_afk is now known as chihchun [06:31] mvo: welcome back [06:34] hey mborzecki [06:34] mborzecki: good morning! what did I miss? [06:35] mvo: jdstrand has some interesting findings about libseccomp and go bindings [06:35] mborzecki: oh? in the forum? in a PR? [06:35] mvo: https://bugs.launchpad.net/snapd/+bug/1825052 and forum as well [06:36] mvo: there's also https://github.com/snapcore/snapd/pull/6681#issuecomment-485930543 [06:37] mborzecki: woah! [06:37] mvo: and a relevant forum topic https://forum.snapcraft.io/t/disabling-seccomp-sandbox-where-a-buggy-golang-seccomp-is-used/11054 [06:38] * mvo nods [06:38] mborzecki: anything pending for 2.39 that needs a cherry-pick/backport? [06:40] mvo: this probably https://github.com/snapcore/snapd/pull/6762 [06:41] mborzecki: yes, that looks like it [06:41] mborzecki: doing that now, thank you [06:42] mvo: maybe this one too https://github.com/snapcore/snapd/pull/6748 (spread jobs debugging help) [06:54] Hey mvo [06:54] mvo: yes [06:55] mvo: interfaces have a serious bug [06:55] mvo: also CE waits for firmware fix badly for 2.39 [06:55] zyga: do we need 2.38.2 for the seccomp issue? [06:55] mvo: also snapd panics on 19.04 [06:56] mvo: you need to process all the bugs and decide [06:56] zyga: what is the bugnumber for 19.04? [06:56] zyga: it sounds very much like we need one :/ [06:56] One moment [06:59] mvo: https://forum.snapcraft.io/t/snap-apps-not-working/11000/4 [06:59] https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1825437 [07:00] mvo: people are also affected by https://bugs.launchpad.net/snapd/+bug/1819318 [07:00] mvo: and yesterday we debugged https://bugs.launchpad.net/snapd/+bug/1825883 [07:00] but fixing it is not something easy [07:02] zyga: thanks, looking [07:04] 6720 needs a second review, it touches quite some bits so I would love to merge it soon to avoid conflicts [07:06] * dot-tobias says hi [07:08] mvo: done [07:08] hey dot-tobias [07:09] zyga: ta === pstolowski|afk is now known as pstolowski [07:12] morning [07:12] hey pstolowski ! [07:12] pstolowski: hey === chihchun is now known as chihchun_afk [07:55] 'sup [07:55] (morning!) === chihchun_afk is now known as chihchun [07:56] pstolowski: hey, I wrote some thougts about the interface problem: https://bugs.launchpad.net/snapd/+bug/1825883/comments/6 [07:56] I would appreciate if you could have a look [07:56] hey Chipaca, how are you doing? [07:57] zyga: chipaca integrity is at 73% [07:58] reverse booze polarity [08:01] zyga: thanks, looking [08:05] zyga: nah, lower back still not happy, but i'm a'ight [08:21] mvo: updated https://github.com/snapcore/snapd/pull/6756 [08:21] https://github.com/snapcore/snapd/pull/6758 is super boring, just some code moving from place to place + new tests [08:22] zyga: btw, i was wrong in one aspect re my earlier reconnect PR - disconnect was not run there (I misread the diff), reconnect was only re-executing prepare- and connect- hooks, so it was doing what you're suggesting (expect for prepare- step which was also run). i generally agree with the idea of (re)running interface hook(s) on refresh, but i think Samuele's comment to my original PR still stands. perhaps separate [08:22] 'connection-updated' hook is the aswer to that question (although it still complicates the implementation of hooks for snap devs). also i think not updating dynamic attributes during refresh may be problematic, they may be derived from static attributes of either side of the connection so if static attrs change, dynamic ones can change as well [08:23] pstolowski: re-connect vs initial-connect? [08:23] simple, there's no parepare no disconnect [08:23] pstolowski: hmmm, interesting observation about the dynamic attributes [08:23] pstolowski: is there anything that could be used as an example? [08:30] zyga: example is only theoretical at this point: snap A exposes some content, prepare- hook of snap B reads the exposed path attribute and sets own path attribute (i'm not sure content interface as is would support that, but that's the general idea) [08:31] Question about device intialization, specifically how to shorten its duration: https://forum.snapcraft.io/t/shorten-device-initialization-or-at-least-give-user-feedback/11081 ping ogra / mvo [08:33] dot-tobias: thats an interessting one - we want to improve this (hopefully in the next cycle) by doing as much as we can in the image creation time. right now its slow (and a pain point for many) [08:33] dot-tobias: would it help if we could make some led blink to show the users things are still going? [08:34] dot-tobias: does the rpi has anything like this we could use (pardon my ignorance on this)? [08:37] mvo: perhaps seeding the gadget to run boot splash is the way to go [08:37] mvo: Glad to hear, both that it's on the radar and that it's possible at all to move init to image creation 😊 The RPi has a “status” LED which (I think) already blinks sporadically during init, but I fear a small LED would suffer the same fate as our hint “don't worry if it takes > 8 minutes” on our download page – easily ignored since most of our users are not that tech-savvy [08:37] mvo: I agree with dot-tobias, let is not the good UX here [08:37] ideally 1st boot would allow device makers to setup proper UX on the attached screen (if any) [08:38] or perhaps 1st boot should really be in the factory [08:38] and we should have post factory boot 2nd boot [08:38] that does some custom stuff but is otherwise fast [08:38] zyga: right, I agree :) was mostly wondering what we can do today to help [08:39] dot-tobias: do your users have a screen attached? [08:39] zyga: I like the screen idea [08:41] Chipaca: what came out of this 2.38 snapd rest api issue that was reported on the forum last week? [08:41] mvo: hmm... remind me? [08:43] Chipaca: let me find the forum topic [08:43] mvo: Yes, the image is meant for display devices, so users have a screen. [08:43] Chipaca: https://forum.snapcraft.io/t/snapd-2-28-rest-api-endpoints-with-large-response-issue/10968/3 [08:44] zyga: Yeah, a splash screen would work – BTW, did I misunderstand or do Core18-images show a splash *without* psplash? [08:44] dot-tobias: I wonder if (again - short term) something like a hook or an app that would monitor the snapd socket with progress would help [08:44] mvo: ah! “ some behavior in Nginx and does not necessarily need to be an issue in the Snapd REST API socket” [08:44] mvo: (from the last reply on that) [08:45] Chipaca: aha, thanks. so just user confusion? [08:45] dot-tobias: I don't know about core18 and pi and splash screens I'm afraid [08:45] mvo: not sure if confusion is the right word, but the issue is in how they're proxying snapd, not in snapd [08:45] mvo: AIUI; I half expect them to come back and ask "so when are you going to fix it" [08:45] zyga: maybe some plymouth integration? [08:46] because at no point did they say "oops my bad" [08:46] mborzecki: dunno really, I would not marry the two concepts [08:46] Chipaca: ok, somehting changed on our side with go-1.10 I guess and that confusd nginx? [08:46] mvo: but some people are unable to say that ¯\_(ツ)_/¯ [08:46] Chipaca: did go1.10 switch to http2 by default? [08:46] zyga: Oh sorry – only first part of that message was a reply to you, second one was thinking while writing 😄 [08:46] Chipaca: heh, yeah I guess that might happen (that they ask us to fix something) [08:46] mborzecki: boot splash might as well be shown on a dot-matrix display attached to a gizmo [08:46] Chipaca: to be fair - we did change something :/ [08:47] mvo: http2 is supported, but that's not new between 1.6 and 1.10 [08:47] Chipaca: ok, I have this feeling this will come back but its fine to shelve it for now I think [08:47] mvo: we agree then [08:47] Chipaca: especially with the things we need/want to finish before lyon [08:48] mvo: lyon is next week? [08:48] mvo: re: hook/app to monitor snapd socket: Might be a short-term solution, but that wouldn't be able to output something until at least core, gadget and mir-kiosk are properly installed, if I'm not mistaken? [08:48] mvo: (http2 is only supported if we're doing TLS, and we're not, so actually no http2) [08:49] Chipaca: 13.05 [08:49] ah, phew [08:49] dot-tobias: yeah, it would be late :/ [08:52] I think it must be some gadget specific thing [08:52] before mire and all the stuff is up [08:53] perhaps even a special hook that gets run [08:53] mvo: dot-tobias: this really feels like some custom initramfs, with early boot splash via plymouth or somesuch, effectively device/appliance specific [08:53] seeding-started seeding-done or something of the sort [08:53] mborzecki: yes but it would be nicer if we could have this done via gadgets [08:57] zyga: initrd is part of core though and we don't expect people to build a custom core* [08:58] mborzecki: indeed, they can do custom bases but it would be pretty unfortunate if we would force people to add a lot of boot bases to have a splash screen [08:58] yup [08:59] ogra had boot splashes working [09:00] how'd he do it? [09:00] ogra: yes you :-) [09:00] mborzecki / zyga / Chipaca: My image uses psplash for a custom splash screen right now, see https://gitlab.com/glancr/gadget-snap-pi-kiosk (adapted from ogra 's examples of course 😊 ) [09:02] Problems with this: a) The splash is not visible as soon as mir-kiosk is installed and started → black screen b) Changing the image after the initial boot has completed requires a separate psplash binary in /boot-assets/psplash.img (if I understand things correctly), so an initial “don't touch, doing stuff for a while” screen would confuse users on subsequent boots [09:04] But my knowledge of these things is quite little, so I might just be overlooking simple solutions 😄 [09:04] dot-tobias: instead of an image, just play https://www.youtube.com/watch?v=V4uV3icrmw0 [09:04] simples! [09:04] * Chipaca runs off [09:05] Chipaca: That's plan B 😄 [09:25] is it the default for Elementary to use the LimeNET store? [09:34] dot-tobias: hah nice, looked at the gadget initrd part is loaded into the memory right after the actual initrd, nice trick [09:36] mborzecki: Full credit to ogra for this, his universal pi kiosk gadget snap was a huge inspiration (read: fork and customize) for me 😊 [09:43] mborzecki / zyga / mvo: Re: first boot – FWIW, I didn't mention that I use cloud-init in my image atm, e.g. to work around https://bugs.launchpad.net/snapd/+bug/1820060 . Came across it in https://forum.snapcraft.io/t/how-to-preconfigure-custom-image/4154/15 ; maybe this enables an interim solution for others. Couldn't find an approach for my specific issue though 😄 [10:01] pstolowski: not sure you saw it, looks like 6755 needs a tweak in the spread test [10:03] mvo yes, thanks, im fighting with it this morning [10:03] pstolowski: uh, ok. good luck then .) ! [10:04] the trick with modyfing snapd.service file is a no-no on read only fs on core ;) [10:06] pstolowski: yeah, its fine (for now) to exclude core [10:06] pstolowski: alternatively we could bind mount a modified copy to the place on core [10:07] mvo i’m playing with override.conf [10:09] need to run a quick errand, afk for a while [10:09] pstolowski: no worries [10:10] pstolowski: yeah /run/systemd/system/... should also work [10:10] pstolowski: and much simpler :) [10:10] * mvo really likes this feature of systemd [10:11] sil2100: I was looking at 1825437 just now - the seed.yaml on the image shortly before the release was wrong and that crashed snapd. do you happen to know what writes the seed.yaml and where I can file a bug? mostly for awareness so that we can add some sort of test to ensure the seed.yaml is correct [10:24] zyga: re seed.yaml installer issue> the bugreport is a bit unclear, it sounds like this was a bug on an image before the release and the release image is fine. but that sounds suspicious and someone mentioned a race. do we know more? do we have more reports like this? [10:25] zyga: or pointers to the code that writes the seed.yaml? [10:25] mvo: some more on the forum [10:25] mvo: I don't have that, kenvandine looked before and pointed us at foundations/desktop [10:25] zyga: thanks! all over the place or in a specific thread? [10:25] mvo: there are two more threads AFAIR [10:25] mvo: but no more details [10:25] mvo: people just carry on after removing that extra - [10:28] are people particularly obtuse today, or am I especially grumpy? [10:29] Chipaca: what's up? [10:29] * zyga has a revalation about another bug [10:29] I'm sorry, I'm a bit lost today [10:29] zyga: "I tell it to put something in usr/bin, but then the thing is not in bin/" "did you check usr/bin?" "it's not in bin/" [10:30] pstolowski: Repository.RemoveSnap doesn't remove connections! [10:30] ah, sorry, it's all good : [10:30] * zyga is stumbiling [10:42] mvo: hm, curious, wonder what happened there - all the seeding logic is implemented in livecd-rootfs itself [10:43] mvo: the seed.yaml is created there when seeding happens, so possibly take a look at live-build/functions and look for the snap_preseed family of functions [10:45] sil2100: thanks, so nothing dynamic in the installer? thats cool [10:45] Curious [10:45] sil2100: I have a look [10:46] sil2100: we got reported from media-info 20190413 [10:46] so a bit before the final releease [10:47] sil2100: ha! thanks, I have interessting pointers now [10:47] mvo: not that I'm aware of at least, we get the snap list from the seeds and then work through the list during build time, and I can't remember hearing about that changing [10:47] sil2100: looking deeper, thank you :) [10:47] sil2100: no worries, you helped a lot! [10:47] mvo: yw! Thanks for looking at this, indeed the wild '-' seems like something really really strange ;p [10:48] * mvo looks at code/diffs now [10:48] sil2100: there are changes in livecd-rootfs at exactly the relevant times so I bet its that [10:48] another large'ish gadget update PR coming up, though most of it is tests [10:52] degville: docs for the REST API are still the ones in the github wiki, yes? [10:54] Chipaca: yes. [10:54] k [10:55] Chipaca: I need to bring them over, but also I need to get agreement on where they'll be located - partly as issue with the snap docs outline, and also where Core docs go and fit with the REST API docs. [10:55] degville: as soon as you start doing that somebody'll point out that we shouldn't call them "REST API" anything [10:55] degville: :-) [10:56] not that they're _wrong_, just that names are hard [10:56] degville: of course :) [10:57] degville: talking to yourself already? :-p [10:58] Chipaca: degville: uh oh - too long working from home will do that. [10:58] * degville nods [10:59] degville: run away! run away! to the coffee shop or sth :-) [11:00] * Chipaca is immune to that though [11:00] am not! [11:00] * Chipaca is too! [11:03] * degville checks the ring is safely hiddenses. [11:03] https://github.com/snapcore/snapd/pull/6769 if anyone is interested in volume layout [11:08] mvo: could you cherry-pick https://github.com/snapcore/snapd/pull/6715 to 2.39 too? [11:11] mborzecki: done [11:11] mborzecki: thank you! [11:11] mvo: thanks! [11:13] pstolowski I am close to fixing that attribute issue [11:13] At least enough to buy more time [11:13] zyga: what did you do? setup-profiles fix? [11:16] Yes [11:16] It is not correct but also not more incorrect [11:16] Equivalent to not catching static attractive [11:16] With smoother change [11:39] pstolowski: it's possible to craft actions that will give you different static attrs for the _same_ plug or slot among two connections [12:10] yah [12:10] I found more bugs [12:10] ok [12:10] let's stash those aside and fix one [12:11] kenvandine: as an advice, please don't rely on snapd to create directories in $SNAP [12:11] kenvandine: I found a serious bug in that area now [12:25] mvo: hey, I will be in the standup in ~30 minutes. you don't need 2.38.2 for either issue imo [12:48] erf, I merged from master and now seeing in fedora-29: [12:48] RPM build errors: [12:48] Bad exit status from /var/tmp/rpm-tmp.VZ4h1y (%build) [12:49] * jdstrand discards and tries again [12:50] pstolowski: https://github.com/snapcore/snapd/pull/6770 [12:50] pstolowski: sorry for taking so long, I found a deeper bug in this and spent time exploring it [12:50] mborzecki: the mount backend is hosed, [12:50] mborzecki: I bet the solution is proper topological sort of mount entries [12:50] mborzecki: it sucks to have this mid-refactor too [12:51] zyga: heh, at least we know that now :) [12:51] mborzecki: I have a way to reproduce this [12:51] mborzecki: I'm 100% sure this is the ghost bug people were mentioning all the time [12:51] mborzecki: but were unable to reproduce (which makes sense too btw) [12:51] mborzecki: if I get hit by a bus today: writable mimic on $SNAP misbehaves [12:52] mborzecki: refreshing a snap over itself (same snap reinstalled) shows abnormal outcome [12:52] mborzecki: I will explore some more, I don't have all the answers yet [12:52] mborzecki: but I'm happy to at least take a stab at that elusive bug [12:52] zyga: ty! [12:52] pstolowski: no unit tests, help appreciated [12:57] zyga: s/if I get hit by a bus/if I win the lottery/ :) [12:57] zyga: if you are going away, I'd much rather it be cause you have tons of $$ [12:58] s/\$\$/money/ [12:58] no need for you to be paid in USD [12:58] jdstrand: ¤¤ [13:00] zyga: thank you for that workaround, looks good, i'll help with unit tests after standuo [13:18] zyga: i guess the gnome extension to snapcraft could create those directories [13:18] ~$ snap install snap-with-complex-requirements [13:18] error: https://i.imgur.com/z96dZ0x.jpg [13:18] zyga: i'd rather not expect app developers to know to create all those dirs [13:24] kenvandine: agreed [13:24] kenvandine: I'll let you know more once I have the data [13:33] Chipaca: are those files that we create from snapd? https://bugs.launchpad.net/command-not-found/+bug/1824000/comments/2 [13:34] zyga: yes [13:34] looks like our bug then [13:34] zyga: did 19.04 change ulimit? [13:34] or sth? [13:34] dunno [13:34] maybe :) [13:34] umask I presume [13:35] yeah [13:35] that one [13:35] zyga: but the .metadata might imply that an update crashed? maybe [13:35] not sure [13:36] dunno [13:36] hmmmm [13:36] zyga: actually wait it might not be ours [13:37] zyga: ours are under snapd [13:37] zyga: /var/cache/snapd/commands.db is ours [13:37] zyga: soryr [13:42] zyga: we've had some issues with content interfaces that have been really hard to reproduce [13:42] maybe fixing these things you're finding will fix those :) [13:42] kenvandine: I think what I found will fix that [13:42] kenvandine: not 2.29 material, something that will take some time to do [13:42] like how sometimes the mount is empty [13:42] kenvandine: but I will get it done [13:42] after a refresh [13:42] yes, I reproduced that too [13:42] we've never figured that out [13:43] great === om26er is now known as om26er1 === chihchun is now known as chihchun_afk [14:07] zyga, hey, I see this error sometimes running the failover test [14:07] zyga, https://paste.ubuntu.com/p/2CqKNTfz47/ [14:07] this is on a pi3 [14:07] then the device doesn't boot anymore [14:07] it is stuck [14:10] cachio: I don't know anything about that, perhaps the kernel has crashed, if you see that again try pinging the kernel, it's a simple way to check if the system has failed entirely [14:10] zyga, ok [14:10] thanks [14:26] roadmr Hi! Do you think https://forum.snapcraft.io/t/please-allow-auto-connect-of-display-control-interface-for-deskconn/10831/ is good to go live now ? [14:28] om26er: I think so but usually jdstrand is the one who wrangles auto-connects [14:32] roadmr ah, ok. [14:34] I'm a bit behind but hope to run through everything today [15:14] cachio: do you remember which pr it was that fixed the core18 remove thing? [15:16] ah, #6753 [15:26] zyga: i've unit tests for your fix, how would you like it proposed? [15:29] Hi, I'm a support person for a project and someone has made a snap package I'm trying to understand. Specifically, they're recommending files be owned by root because it runs as root inside. [15:29] https://github.com/albertodonato/sonarr-snap [15:29] Chipaca, correct [15:30] fryfrog: daemons that come in snaps run as uid 0 for now, yes [15:31] :o [15:31] zyga: pushed here https://github.com/stolowski/snapd/tree/lp-1825883-unittests ; please take a look and feel free to incorporate into your PR [15:32] Chipaca: how does one deal w/ that *outside* the snap? [15:32] fryfrog: I'm not sure what you need to deal with [15:32] (i don't know your snap at all) [15:32] s/snap/project/ [15:33] Any files it needs to read/write, would either need to be owned by root, root would need to be in the group or ... i guess at least permissions don't matter :) [15:34] fryfrog: it'll create them as root, but it'll be able to read them anyway yes [15:34] There is no way to specify the UID a daemon runs as? [15:35] fryfrog: no (seccomp arg filtering is only just becoming usable _now_ -- we're working on it!) [15:36] I'm glad to hear it is in the pipeline :) [15:36] Thanks for helping me understand :) [15:37] fryfrog: there's also the issue of no user other than root being 'universal' across distros, but we're tackling that as well [15:37] (initial use of the arg filtering thing will be via enabling a 'daemon' user, which most distros have and those that don't will get a warning on install) [15:37] Docker "fixes" this by not actually caring about the user *name*, they just allow passing in a UID/GID [15:37] yeah [15:38] but then your innocent pvr ends up running as the same uid as postgres or sth [15:38] anyway, yes, there's a bunch of silly gotchas we need to care about [15:38] it's all planned out afaik (there's a forum topic layout out the steps and stages of it) [15:39] laying out* [15:39] Cool :) [15:39] pstolowski: thank you! [16:03] woot, #6755 is green [16:25] Chipaca, when you have time could you please take a look to this one? #6694 === pstolowski is now known as pstolowski|afk [17:21] cachio: LGTM [17:21] cachio: as I said there, some day sergiusens and I will be able to spec out (and implement/use) 'snap download --plz-put-the-snap-here=', but until then, what you've done is best [17:34] Chipaca, nice, thanks for the review [18:10] zyga, please could do take a look to #6694 [18:10] it is almost ready and I need that to validate 2.39