[00:22] jorian, snap would be a good option for packaging your app _together_ with the newer lib === georgeowell_ is now known as georgeowell [01:00] if anyone is familiar with locales in combo with snaps, is the folder usr/lib/locale important for an app, or can I omit it (presumably by not installing locales-all in stage-packages)? [01:01] it's 117MB is all so it's bloating my snap to rather large proportions [01:02] (117MB uncompressed) === chihchun_afk is now known as chihchun [03:48] kyrofa: good point, sorry I didn't say that more explicitly [05:51] PR snapd#3334 closed: Release snapd 2.26.2 [05:51] PR snapd#3336 opened: Reease snapd 2.26.2 [05:58] PR snapd#3292 closed: wrappers: make StartSnapServices cleanup any services that were added if a later one fails [06:09] PR snapcraft#1320 opened: store: send X-Ubuntu-Series, not X-Ubuntu-Release [06:33] PR snapd#3337 opened: errtracker: try multiple paths to read machine-id [06:34] mvo: ^^ [06:41] morphis: cool, thank you [06:42] morphis: is there a leftover debug fmt.Prinln in there ;) ? [06:42] mvo: yeah, could be :-) [06:43] morphis: looks great otherwise, I will do a formal review in a bit, just fixing the image, shadow is out-of-sync with the archive [06:43] mvo: oh wonderful, that just happened recently [06:43] mvo: dropped the Println [06:47] good morning [07:00] mvo: hey, I'd like to trigger a build of the ppa that feeds edge, I'm looking at https://launchpad.net/~snappy-dev but I see quite a few archives there, can you give me a hand plese? [07:00] mvo: perhaps we could close some archives (I suspect we don't need all of them at once) [07:01] CC ogra_ ^^ [07:01] zyga: sure, let me check [07:02] * zyga goes to garden PRs in the meantime [07:02] thank you :) [07:03] zyga: https://launchpad.net/~snappy-dev/+snap/core is what need if you want to trigger a new build [07:04] zyga: please check that shadow build in the ppa:snappy-dev/image, would be good to have that fix in the iamge too [07:05] mvo: I know the snap recipe, I was just struggling with the PPA part [07:05] zyga: aha, ok [07:05] mvo: thank you! [07:05] zyga: ppa:snappy-dev/image is the relevant one [07:06] zyga: I will trigger a new build now unless you want something else in there too? [07:06] mvo: I'll talk to ogra about closing the remaining archives, they feel like just noise [07:06] mvo: no, everyting is ready since yesterday [07:06] mvo: please go ahead [07:06] everything* [07:06] zyga: I deleted some obvious ones, some are 15.04, if we are really sure we don't need 15.04 anymore for the previous snappy incarnation then those can go [07:07] mvo: though when I refreshed core today (from edge) I got 2.26.2 [07:07] mvo: what about this one? https://launchpad.net/~snappy-dev/+archive/ubuntu/edge2 [07:07] or https://launchpad.net/~snappy-dev/+archive/ubuntu/beta ? [07:10] zyga: checking [07:11] zyga: I guess beta is fine to remove, the risk is always that if you delete a ppa you screw all the people who still have it in their sources.list [07:11] zyga: but given that the latest archive in there that is still supported is trusty the risk is relatively low [07:12] zyga: I will delete edge2 after this core build [07:13] mvo: there is something wrong in the image [07:13] https://travis-ci.org/snapcore/snapd/builds/233106199#L3814 [07:13] mvo: this is the 2.26.2 merge back to master [07:13] we lack --extrausers [07:13] it seems like the archive package has higher version number [07:13] zyga: yes, the core build that is currently in process will fix it [07:13] and we're not getting the patched one [07:13] ah, perfect, thank you for handling this :) [07:20] zyga: thanks for pushing to my PR but I will squash that change if you're ok [07:21] was about to push the same [07:25] morphis: no worries, do it [07:25] morphis: I'd squash myself if I knew you wanted it :) [07:25] :-) [07:36] hi mvo: i've started with 2.26.2 validation, tests/main/listing and tests/main/interefaces-log-observe are failing on i386 (the first one that finished) this is expected (it seems that https://github.com/snapcore/snapd/pull/3327 is missong in the release branch), right? [07:36] PR snapd#3327: tests: fix failing tests (snap core version, syslog changes) [07:38] fgimenez: meh, the listing test! we may need to backport that fix to 2.26 too [07:38] fgimenez: interfaces-log-observe is unexpected [07:38] fgimenez: tests/main/listing will also fail in autopkgtest so we also need a new SRU :/ [07:39] mvo: i think interfaces-log-observe is fixed in the same branch, will check in the upgrade-from-stable scenario [07:39] zyga: so moment of truth, running the entire main suite again for fedora, we should now close to 100% [07:40] or better said close to 100% of those tests which apply to Fedora (disabled those which test re-execution or classic confinement which we don't support on Fedora yet) [07:43] PR snapd#3338 opened: tests/main/completion: source from /usr/share/bash-completion [07:44] morphis: I'm very happy to see that :) [07:44] :-) [07:44] zyga: when you're leaving for the suse conference? [07:45] next week [07:45] 25th [07:45] ok [07:58] mvo: why interfaces-log-observe is unexpected? did you build also with an older revision of core stuff [08:01] pedronis: well, maybe it is not, I have not checked the log what exactly is failing. if it has a similar regex as the listing test then it is not unexpected [08:01] * zyga hugs mvo - the SRU dancer [08:01] pedronis: I will check as soon as I'm finished with the current code review [08:01] mvo: well we turned off rsyslog [08:01] yep, that was bundled into chipaca's branch [08:01] at least that's why we had interfaces-log-observe issues on the mainline === didrocks1 is now known as didrocks [08:04] pedronis: aha, I must have missed that, thank you. that will also need to be ported then [08:05] mvo: here is the discussion https://forum.snapcraft.io/t/change-in-logging-behaviour-on-ubuntu-core/591/4 jdstrand still thinks it's bad backward compatibility wise [08:06] pedronis: fwiw, I agree with that [08:07] pedronis: if it changed from 2.26 to 2.26.3 that would also be very bad [08:08] pedronis: thanks for raising it, I will look, it definitely needs to go into the release notes if it goes into 2.26 [08:10] pedronis: I will follow up in the forum, I think if nothing else we should create some hint for people looking at /var/log/syslog that they now need to use journalctl [08:18] zyga: you already had time to look into the bind problem? this really prevents configure hook execution on !ubuntu for all !core snaps [08:18] mvo: can you override merge https://github.com/snapcore/snapd/pull/3332 so that we don't waste resources on re-testing it [08:18] PR snapd#3332: interfaces/seccomp: document Backend.NewSpecification [08:19] pstolowski: hey [08:19] zyga, hey! [08:19] pstolowski: question about https://github.com/snapcore/snapd/pull/3328 - how would you feel starting afresh; take that branch, squash the history so that it makes more sense [08:19] PR snapd#3328: many: snapctl outside hooks v2 [08:19] zyga: 87% pass rate, is what I have now [08:20] pstolowski: I will work with you on the regexp code elimination today [08:20] zyga, np, i can do that if it helps [08:21] morphis: nice job [08:21] pstolowski: as you think but I would recommend it, it's a "new" PR and the history is full of stuff that makes it harder to follow [08:21] PR snapd#3332 closed: interfaces/seccomp: document Backend.NewSpecification [08:22] mvo, zyga: however we need to fix the bind() problem for hooks this cycle, if you have any pointers for me what calls it in the go libs when snapctl is executed from your previous research that would be a great starting point [08:22] morphis: sure [08:22] morphis: who should I talk to about the dropping of rsyslog in the CE team? I would like to make sure none of our existing customers will suffer because of it [08:22] zyga, ok, will do in a few [08:22] morphis: it's easy, one sec [08:23] mvo: I can raise this with Tony later today, it will mean that /var/log/syslog is not being filled anymore, right? [08:23] joc: would that be a problem for you guys in any way? [08:24] morphis: I think so, I'm looking at it right now [08:25] morphis: look at /usr/share/go-1.6/src/net [08:25] morphis: look at /usr/share/go-1.6/src/net/net.go line 99 [08:25] morphis: on startup the net package probes ipv4 and ipv6 [08:25] mvo: i can conform that adding the changes in snapd#3327 to the release/2.26 branch makes the full suite pass using core from beta (amd64 so far) [08:25] PR snapd#3327: tests: fix failing tests (snap core version, syslog changes) [08:25] *confirm [08:25] zyga: hm [08:26] morphis: it creates two sockets and binds them to localhost address [08:26] morphis: pretty sure we have some tests that expect to be able to find events logged in syslog [08:26] morphis: I don't know if seccomp can even express that [08:26] morphis: and even if, I don't know if this is safe [08:26] morphis: after all, you can bind on localhost and that's still IPC [08:27] morphis: off the top of my head, usb insertions [08:27] morphis: apart from allowing any hook to do "socket+bind" [08:27] morphis: or waiting for the bleeding edge kernel that can return EPERM [08:27] morphis: (instead of killing) [08:27] morphis: well, clarification, all kernels can EPERM, we want bleeding edge kernel that does EPERM + audit [08:28] morphis: even if we just return EPERM today, I don't know if we can do that per syscall [08:28] morphis: and even if we can I don't know how that combines with a rule that allows bind (say an actual hook that has the network-bind plug) [08:28] morphis: all in all, it's pretty messy at this level [08:29] morphis: for network filtering I'd feel much better if we wrapped things in a network namespace and used firewall rules [08:29] morphis: but that's a huge chunk of work [08:29] joc: interesting [08:29] mvo: ^^ [08:29] zyga: hm [08:29] mvo: how about postponing this till we can ship a working rsyslog snap? [08:30] mvo: I know it's a tough problem because we have people that want both things _now_ [08:30] zyga: this was suggested by jamie as well, see the forum discussion [08:31] mvo: alternatively allow customers to disable it [08:31] mvo: but don't disable yet and don't remove yet [08:31] I'm trying to make a snap of a prigram that uses git internally, to clone and manage stuff, I'm having trouble getting all of git working from within the snap [08:31] zyga: I personally think even the removal is risky, its a stable 16 core afterall and we enforce updates. if an update is percived to be breaking functionality that will weaken the trust in the system [08:31] mvo: yes, I agree [08:31] looks like it's not finding the stuff that's normally in /usr/lib/git-core/ [08:31] icey: that's expected [08:31] icey: snap execution environment is not like the regular environment [08:32] zyga: I'm aware [08:32] zyga: I've got git installed in the snap [08:32] icey: all the content of your snap is in $SNAP which is something like /snap/yourfancysnapname/123 [08:32] zyga: my program executes git fine, but git can't find any of its handlers [08:32] icey: and the root filesystem is not the one you normally see [08:32] icey: but the core snap itself (see /snap/core/current for some ideas) [08:32] zyga: warning: templates not found /usr/share/git-core/templates [08:32] fatal: Unable to find remote helper for 'http' [08:32] icey: tip: run snap run --shell yoursnapname and look around [08:32] zyga: yes, I've been doing that [08:33] zyga: I'm wondering if anybody has figured out a way to let a program use git (inside the snap) as a dependency [08:33] icey: some people that tried this also set a variable that redirects git to look elsewhere but I don't remember the details [08:34] icey: yes they have, there was a discussion here a few months ago [08:34] icey: but you will still struggle with ssh as ~/.ssh is not going to be available [08:34] zyga: I'm digging through git's manpages now to try to find environment variables [08:34] icey: and $HOME will not retain its value [08:34] icey: dig through git source code [08:34] zyga: I'm only expecting to use the http handlers [08:35] icey: should be good then [08:35] zyga: and I'm not using $HOME [08:35] morphis: what do you think about the bind issue? [08:35] zyga: practically, my program should be fine (and I can change its source ;-) ) [08:35] zyga: it's hard to say what we can do [08:36] zyga: it would be good if you could generate a profile specific for snapctl [08:36] as that is the real issue here and it gets executed under the umbrella of the hook profile [08:36] zyga: do you know if the environment variable is a runtime change or if it would need to be an install time thing? [08:37] icey: I don't know [08:37] icey: may be either, it was long ago, sorry :/ [08:37] fgimenez: mvo: we need to upload some test-snapd-* snap with a configure hook (can be empty, accept anything) [08:37] morphis: interesting [08:37] morphis: that may be a nice way out [08:37] zyga: yes [08:37] morphis: we could have a profile just for snapctl and allow anyone to run snapctl in a way that changes profiles [08:37] zyga: is that something we can do without a lot of refactoring? [08:37] morphis: there's still risk ask overmount must now reject you from changing snapctl [08:38] morphis: perhaps [08:38] morphis: let me try [08:38] pedronis: do we have the snap already? if so its trivial for me to upload [08:38] morphis: thanks for the idea! [08:38] zyga: np :-) [08:38] mvo: no [08:38] zyga: if you have some initial code I can help more with that as we need to get this fixed real soon [08:38] mvo: can make a PR with one though [08:38] pedronis: ok, I can create one - test-snapd-with-configure ? [08:38] yea [08:38] pedronis: whatever is easier for you :) [08:38] zyga: I can open a forum topic to get this more widely discussed [08:39] pedronis: but I can do it too [08:39] morphis: open one please [08:39] morphis: I have no code at all but I can get started quickly [08:39] morphis: it's actually trivial to test [08:39] zyga: would be great! [08:39] mvo: I'll make a PR about that and ping you [08:39] pedronis: thanks a bunch ! [08:40] pedronis: btw, 3211 (the repair assertion) is green and has two +1, anything else I can do there or can I merge? [08:41] mvo: I think it's fine personally, also after further thinking I think it's a good property that if we want we can emit a canonical signed brand_model-# assertion if needed [08:41] (witho some coord with the brand) [08:41] morphis: hmmmmmm [08:42] morphis: I think there's a separate problem [08:42] morphis: we can change the apparmor profile [08:42] morphis: but not the seccomp profile [08:42] morphis: let me read some man pages [08:42] zyga: any chance there's a way to see the snapcraft.yaml used to build a snap? looks like https://uappexplorer.com/app/git-repo.jhodapp may have figured it out [08:42] pedronis: great, I merge it then, it got a +1 from gustavo as well, we can always iterate, its not being actively used yet anyway [08:42] mvo: yup [08:43] icey: no [08:43] icey: I know there's ongoing work for that but I don't think it is ready :/ [08:43] icey: you can just reach to the author of that snap [08:43] mvo: do we have a core that fixe create-user ? or is still building? [08:44] yeah zyga, stalking hiim on GH + Launchpad hasn't led me to useful results [08:45] Chipaca: maybe snapd#3335 is something you can review? [08:45] PR snapd#3335: daemon: teach the daemon to wait on active connections when shutting down [08:45] nah [08:45] :-p [08:45] oh, no battery [08:45] * zyga looks for cable [08:46] zyga: looks like there's a $prefix available during build, I'm going to try building git in my snap ;-) [08:46] fwiw https://codecov.io/gh/snapcore/snapd/pulls [08:47] icey: tip: set prefix to /snap/your-snap-name/current/usr and then install it so that the snap's prime directory only contains the /usr part [08:47] icey: (I think you can do this with organize) [08:47] PR snapd#3211 closed: assertions: add "repair" assertion [08:47] icey: ask kyrofa for hints, I'm not using snapcraft that much and my knowledge can be stale [08:47] zyga: also sorry about the missing comment on 3308 - I added one last night but for some reason gh marked it as pending :/ [08:50] pedronis: 1.6 didn't have Shutdown, i guess [08:50] Chipaca: it doesn't [08:50] with 1.8 [08:50] we wouldn't need this [08:51] but doing it similarly means we if/when we move to 1.8, the change is easier [08:52] zyga, we already have sc_snap_name_validate, which is regexp-free [08:52] there was some idea to do something inside Command.ServeHTTP , but this seems less invasive [08:53] pstolowski: I think we need a few more as well [08:53] pstolowski: but yeah [08:53] pstolowski: that's the way to go [08:53] mvo: no worries, I just replied to your question [08:53] mvo: thank you for responding :) [08:54] Chipaca: I need to merge master and waiting for a fixed core to rerun tests, so I can add a comment about that [08:54] pedronis: +1, anyway [08:55] pedronis: my questions were merely because the godoc i have up right now points to 1.8 [08:55] if it had been pointing to 1.6 i woudln't've had to ask them at all [08:55] code is super clear [08:55] PR snapd#3328 closed: many: snapctl outside hooks v2 [08:55] thx [08:58] offtopic: wow, amd has released some insane chips! [08:58] PR snapd#3338 closed: tests/main/completion: source from /usr/share/bash-completion [08:59] mvo: thanks! [08:59] mvo: do you have a tentative date for 2.27 ? [09:03] mvo: new core built [09:03] pedronis: yes, 23.05 ideally, that would follow the two weeks cadance, I created a milestone [09:03] zyga: another new core? what changed? [09:03] mvo: I was thinking that the release "shadow" issues say we should think about a pair of PPAs and snap recipes [09:04] mvo: I think it was the one you built :) [09:04] mvo: 2.26.2 with fixed --extrausers [09:04] mvo: thank you [09:04] mvo: I'd like to see the new core, from master, with update-ns though [09:04] zyga: that one finished some time ago, maybe the last stranglers or something [09:04] mvo: without interrupting the release [09:04] zyga: reading a bit through the snap-confine code for the seccomp setup .. [09:04] mvo: perhaps, I'm on 1952 now [09:04] morphis: seccomp can only be appended to, I think [09:05] morphis: (constrained) [09:05] zyga: I can't parse the "shadow" issue suggestion, sorry. could you elaborate please? [09:05] mvo: sure, sorry, the conflict of which snapd package is build into the snap [09:05] zyga: the update-ns from core? I think we need to merge the 2.26.2 change first [09:05] zyga: so we can split out the current profile and extend just for snapctl? [09:05] mvo: like when you made 2.26.2 you needed to dput to a ppa and quickly build the core snap [09:05] mvo: so that another daily build doesn't clobber your state [09:05] morphis: "split out the current profile? [09:06] morphis: the problem is that as we can only constrain profiles [09:06] morphis: if you are in a hook/app that doesn't have network-bind (hook) [09:06] morphis: nothing you can do will add that [09:06] morphis: if you allow network-bind in the first place then you'd have to constrain it for _everything_ else (apart from snapctl) but there's no way to do that [09:06] zyga: thats how it works, except it does not need to be "quick" because as long as the version in the ppa is higher than the version in git master we are good, as soon as e.g. 3336 gets merged we get new core builds with the edge snapd. so the workflow is ok for now [09:07] zyga: hm [09:07] mvo: why didn't we get a single build in master that contains edge then? [09:07] mvo: it's not okay because it doesn't work, releases interrupt master and vice versa IMO [09:07] mvo: (I swapped the semantics of master and edge in the previous statement) [09:08] zyga: yeah, that is true, during release the edge is the released version not the edge version, that is annoying [09:08] mvo: not something i wan't to jump at fixing but I want to recongize the issue [09:08] zyga: we can't even patch golang for this .. [09:08] zyga: usually the interruption is short but this time its not because of the various issues (snapd, listing test breakage, rsyslog) [09:08] morphis: nope [09:09] zyga: yeah, agreed, I now understand your point and there is no disagreement [09:09] mvo: I was thinking if we could clone the current setup so we have two of everything: snap recipes, deb recipes and PPA archives [09:09] mvo: then edge just always contains master no matter what [09:09] mvo: and beta is a hand-triggered thing [09:09] mvo: though the list of packages in the ppa is scary :/ [09:09] mvo: I wish we didn't need those [09:09] mvo: for the rarely changing packages we could just copy them from edge ppa once in a while [09:10] mvo: but it'd be better if those didn't exist in the first place [09:10] mvo: and our ppa just contained snapd and nothing else [09:10] mvo: though maybe it can, with the image machinery [09:10] mvo: we might move everything else to a 3rd ppa that is there as a base for all core channels [09:12] PR snapd#3336 closed: Reease snapd 2.26.2 [09:13] zyga: yes, we can build more infrastructure and decouple the two. there is also the suggesiton from Gustavo about decoupling the snapd build and the base image which is probably the first step as we get a more predictable core with it [09:14] mvo: yes, the way the core is assembled also has to change [09:15] zyga, i need to contact all the uploaders first, (i dont want to delete PPAs without consent), but i'll put dropping them on my TODO [09:16] ogra_: hey, good mornign [09:16] ogra_: I think mvo axed some PPAs [09:16] zyga, mvo, hmm, did you guys disable something already ? [09:17] ogra_: I think so [09:17] seems edge is broken [09:17] (the edge PPA) [09:17] ogra_: those looked very much dead anyway [09:17] ogra_: in which way? [09:17] "This PPA depends on disabled archives. it may cause spurious build failures or binaries with unexpected contents." [09:17] ogra_: I killed edge2, I can fix the fallout [09:17] this needs some planning i fear [09:17] ok [09:18] ogra_: edge is fine again [09:18] we should end up with image anbd edge though, all the others should go ... but there might be people or other PPAs still using them somehow [09:18] * ogra_ will start a forum thread [09:18] ogra_: have a look at the discussion just above ^ where I advocate for a beta PPA [09:19] zyga, well, we should finally follow the plan ... [09:19] ogra_: as we (the snapd team) release just beta and edge snaps having a pair makes sense [09:19] it exists since nearly 2 years now and we're still buiolding from random PPAs [09:19] there should only be one PPA ... for edge ... the rest of the snap builds should come from different stages of the actual archive [09:20] * ogra_ points to the QA teams https://wiki.ubuntu.com/QATeam/OSSnapPromotion [09:20] ogra_: did you see my rationale for a beta ppa? [09:21] ogra_: do you think it makes no sense to have it? [09:21] zyga, i think building beta from proposed is the way to go ... but that means a lot of PPA cleanup (packasges to go to the archive) [09:21] like the plan describes [09:22] else you have a PPA for beta builds and *still* need to do the deb release aside ... using proposed as beta stage archive saves that step [09:22] ogra_: interesting, yes, I think that could work [09:23] the problem is that we need to get rid of the image PPA or at least clean up massively in there [09:24] we could start with it enabled, but then we need to make sure the package version for snapd is always higher in proposed before we build [09:24] PR snapd#3339 opened: tests/lib/snaps: add a test store snap with a passthrough configure hook [09:25] mvo: ^ [09:26] pedronis: thanks, on it [09:26] zyga: https://forum.snapcraft.io/t/hooks-calling-snapctl-are-broken-with-just-seccomp-enabled/658 [09:28] morphis: I read it, thank you [09:28] I'm checking what the kernel allows [09:28] ogra_: who is doing the work on upstraming parts of our PPA? [09:29] pedronis: thanks, uploaded and shared with you [09:33] PR snapd#3340 opened: Smany: snapctl outside hooks [09:34] zyga, ^ reopened; I've addressed some of your comments but not all, don't worry, i'll address them based on comments from previous PR [09:35] pstolowski: great, thank you! [09:35] * zyga looks [09:35] * zyga needs some breakfast [09:35] mvo: pi3 finished with just listing and interfaces-log-observe failing, no more crashes it seems, pi2 is good too so far (85/116), i'll trigger now the upgrade-from-stable scenario on pi3 (this one crahsed too on 2.26.1) [09:36] zyga: happy thing, latest golang git master has changed from an initializer to lazy support detection [09:36] so the func init() call is gone [09:36] fgimenez: excellent, thank you. I'm working on 2.26.3 that fixes the listing and observer failures [09:36] mvo: great tahnks [09:38] friendly neighbour drilling again... I need a walk... [09:39] mvo: pedronis should we upload test-snapd-with-configure to snappy-hub too and automate the upload to the store? it is not likely going to change too much, but to align it with the rest of test-snapd-*, i can prepare the lp branch and build [09:39] morphis: nice :) [09:39] zyga: however, doesn't really help us :-) [09:39] morphis: but still meh :( we won't be using that anytime soon [09:40] zyga: LD_PRELOAD would be another way but not sure if that would work [09:40] I guess golang doesn't go through libc for that [09:41] hm, they use syscall() [09:41] PR snapd#3331 closed: tests: remove unit tests task [09:45] morphis: how would LD_PRELOAD help? [09:45] zyga: if it would use libc to call bind we could easily intercept the bind() call [09:45] but we can't [09:45] morphis: ah, right [09:46] morphis: those are stright syscalls I'm afraid [09:46] also sucks for strace [09:46]