[06:08] Good morning [06:19] morning [06:22] Hey [06:22] How are you? [06:23] zyga: fine, thanks [06:24] zyga: it's cold outside :/ [06:24] Yeah [06:24] I was walking with the dog for two hours last night [06:25] All the way until midnight [06:25] zyga: nice [06:31] mborzecki: quick review to start the day? https://github.com/snapcore/snapd/pull/6587 [06:31] PR #6587: interfaces/apparmor: factor out test boilerplate [06:31] zyga: sure, why not [06:34] mborzecki: I changed some of the test function names but that is all sensibile in the follow-up patch [06:48] mborzecki: pushed [06:52] zyga: thanks! [06:52] thanks [06:52] I will push something that enables apparmor on all of suse next [06:53] then I'll look at debian [06:53] I think we can handle debian (and derivatives), suse (and derivatives if any) to cover most of apparmor land [07:05] mborzecki: https://lwn.net/Articles/782786/ [07:05] maybe mvo should apply [07:05] ;D [07:21] zyga: hi, when are you going to address 6360 feedback ? [07:21] pedronis: hi [07:22] pedronis: most likely today [07:22] guys, think we can land this now if everyone is happing with the current phrasing? https://github.com/snapcore/snapd/pull/6556 [07:22] PR #6556: cmd/snap: hide 'interfaces' command, show deprecation notice [07:22] pedronis: I already started yesterday but haven't finished yet [07:25] mborzecki: I didn't see all the bikeshedding there [07:26] good morning [07:27] mborzecki: hey, yay (or :cry:) at 6585 [07:27] mborzecki: I fear the new sentence is too long [07:27] hey pedronis [07:28] mborzecki: I would like John input on htat [07:29] *that [07:29] hey mvo [07:30] mborzecki: I commented there [07:30] pedronis: thans [07:30] hey zyga [07:30] mvo: hi [07:31] mvo: quick review to start the day? https://github.com/snapcore/snapd/pull/6587 [07:31] PR #6587: interfaces/apparmor: factor out test boilerplate [07:31] zyga: sure [07:31] (when typing this the mouse focus was still on mutt - typing random chars into mutt is really scary) [07:32] mvo: lol [07:32] yes [07:32] kind of like vi but then git prevents critical damage [07:32] no such thing for mutt [07:33] mvo: do you want to travel more, apply now for the position of debian leader: https://lwn.net/Articles/782786/ [07:33] zyga: *cough* [07:34] zyga: and no [07:34] :D [07:34] join debian they said [07:34] see the world the said [07:34] zyga: but it would be interessting, pushing some ideas to moderinize things would be nice [07:34] zyga: hahaha [07:34] yeah [07:34] mvo: I see the reference was recognized :) === pstolowski|afk is now known as pstolowski [08:06] morning [08:13] hey pawel [08:37] PR snapd#6587 closed: interfaces/apparmor: factor out test boilerplate [08:38] PR snapd#6580 closed: cmd/snap-confine: drop unused dependency on libseccomp [08:41] PR snapd#6588 opened: interfaces/apparmor: partial confinement on openSUSE [09:14] zyga: did you change 6575 since I reviewed it? [09:15] pedronis: I applied the changes you asked for, fixed a typo that broke he build then merged master and pushed [09:15] I then noticed that I left duplicate include line and force pushed to clear it [09:15] that's all [09:16] zyga: no, sc_init_invocation was not there when I reviewed it [09:16] oh? but it is added before your review? [09:16] no [09:17] please double check, perhaps you reviewed a view that was stale [09:17] it is so according to the github timeline [09:17] zyga: you force pushed, not point debating this [09:17] yes, I said that [09:18] but that was in the morning after some things landed and the branch conflicted [09:18] you can compare the revisions if you want [09:19] anyway I removed my +1 [09:20] zyga: yeah, 6575 changed quite a bit since I looked at it - or so it felt to me this morning but I have not looked at the details [09:20] slightly sad because the previous one was ready for merging afaict [09:21] mvo: I implemented the changes you suggested [09:21] mvo: and made it clear in the log as well [09:22] zyga: my emails says that stuff was pushed 20 hours ago, and my review was 21 hours ago [09:22] but as I said hard to tell with force push [09:22] pedronis: but that force push was done this morning [09:22] and after I pushed yesteray I was happy to see your review [09:23] zyga: sorry, I'm confused, in 6575 I just wrote "looks good" - am I mixing something up? [09:23] I'm 100% sure github showed this the same way before i did that in the morning [09:23] mvo: we discussed this durinig the standup and you suggested that I put additional patches to the same branch [09:23] pedronis: hello, sample output of new timings to discuss - https://paste.ubuntu.com/p/gXrk8sBwwB/ - contains timings of some tasks, ifacemgr startup, snapmgr ensure loop (nb, do we really run ensure so often?). combines doing/undoing times from tasks [09:24] pedronis: this is especially clear since the three patches starting with cmd/snap-confine: tweak ordering of fields in sc_invocation are added in response to your review [09:25] zyga: oh, sorry. my bad then. maybe we can go back to the point in time (via a PR) that had the +1 from pedronis and the "looks good" from me and then land those bits easily? [09:25] zyga hey [09:25] zyga wanna talk about those PR for gadget snaps? [09:26] mvo: but the unclear aspect is what pedronis actually had reviewed [09:26] zyga unless you already had chat with pedronis [09:26] ondra: hey, I asked about it yestereday [09:26] zyga: I didn't review the init helper [09:26] ondra: I don't know what I can do to help [09:26] maybe it was stale [09:26] whatever [09:26] zyga yeah I was already off when I saw you message [09:26] it needs a review from scratch from both of us [09:26] zyga nothing apart of +1 :D [09:27] ondra: I don't fully understand the consequence of that, it is something perdronis should look at first [09:27] zyga I think rest is on foundation [09:27] zyga yeah we already talked with him [09:30] zyga: if I have too be honest I find snap-confine is starting to be too much split out in helpers, it's a maze to follow [09:31] I'm sure they all make sense locally but if in a 2nd time you need to find out what's going on is a chase [09:35] mborzecki: I pushed a tweak to the interfaces deprecation pr [09:35] pedronis: ^ [09:35] pedronis: let me know when/if you have a moment to discuss this [09:35] Chipaca: thanks! [09:36] pstolowski: we call ensure loop each 5 mins but also each time we need to run more task with EnsureBefore(0) [09:36] Chipaca: is fill() aware of a sequence '..' and will not break it? [09:36] mborzecki: it is not [09:37] mborzecki: ¯\_(ツ)_/¯ [09:37] I mean, breaking it isn't ideal but isn't the end of the world either [09:37] Chipaca: fits unboken in my term so :) [09:38] mborzecki: also breaks just right on 80 [09:38] mhm [09:38] Chipaca: I don't understand how using fill helps [09:39] it's going to take up vertical space anyway [09:39] pedronis: your issue was with vertical space, not with improper word-splitting? [09:39] yes [09:39] pedronis: https://paste.ubuntu.com/p/7qzSqPM4Zc/ [09:39] ah, right, EnsureBefore(0) [09:40] pedronis: but the deprecation is only shown on 'snap interfaces' [09:40] pstolowski: we really need to be careful with it [09:40] pedronis: which is not concerned with vertical space [09:41] Chipaca: no, but is a table [09:41] pedronis: so? [09:41] the message goes to stderr [09:42] I know [09:42] we are still eating 3 lines [09:42] seems too much [09:43] grah [09:43] Chipaca: have you by an chance implemented our TabWriter? [09:44] pstolowski: we don't have a TabWriter [09:44] pstolowski: why? [09:44] pstolowski: we do have a tabWriter, I think, that wraps TabWriter [09:44] as a helper function i mean [09:44] not as a type [09:45] Chipaca: 'snap interfaces' is deprecated and replaced by the 'snap connections' command. is 79 [09:45] pedronis: in English [09:45] Chipaca: they we just hide it [09:45] Chipaca: ah, nvm, it's a standard go lib thing [09:45] thne [09:45] Chipaca: then we just hide it [09:46] pedronis: I don't understand why [09:46] I don't understand why 3 lines are too many, on a command that produces 121 lines of output by default [09:47] Chipaca: it doesn always produces 121 lines [09:48] Chipaca: consider snap interfaces gnome-logs [09:48] pedronis: ok [09:48] pedronis: so what's the issue [09:52] Chipaca: we started here "The 'snap interfaces' command is deprecated, try the new 'snap connections'." [09:54] pedronis: yes [09:54] i don't mind going back to the first version [09:55] We could even go with [09:55] 'snap interfaces' is deprecated; use 'snap connections'. [09:55] this incorporates all the feedback so far [09:55] works for me [09:55] pedronis: ^^ wdyt? [09:56] Chipaca: +1 [09:56] mborzecki: I'll push the change, plus adding the notice to the long help [09:56] (fwiw) [09:57] Chipaca: thank you! [09:57] I updated 6401 fwiw [09:58] Chipaca: the last version is very long, but it's all facts, it doesn't recommend anything [09:58] (except implicitly) [09:59] pedronis: the _last_ version? “'snap interfaces' is deprecated; use 'snap connections'.”? [09:59] Chipaca: the last version in the PR [09:59] pedronis: you ok with the last one here? i think it covers all the issues we've come across so far [10:00] Chipaca: it's very terse [10:01] sorry, that's not the problem [10:01] I'd also add [10:01] NOTE this command is deprecated and has been replaced with the 'connections' [10:01] command. [10:01] to the longHelp [10:01] thanks [10:12] Chipaca: have we goon too far the other way around though? is the new one too prompt? [10:14] pedronis: nah, it's fine [10:14] pedronis: adding 'please' and 'try' are both problematic for reasons degville explained in the pr [10:15] Chipaca: wondering if we want "the new" , issue with that is that at some point we would need to remove "new" because is untrue [10:16] pedronis: I really think it's fine as is [10:16] ok [10:28] PR snapd#6589 opened: daemon: support returning assertion information as JSON with the "json" query parameter [10:29] Chipaca: ^ something for you when you have a bit of time [10:29] * Chipaca looks at next year's calendar [10:29] ok [10:30] heh [10:30] :-) [10:54] PR snapcraft#2499 closed: many: support for "base: core" in snapcraft.yaml [10:56] mvo, here is another core18 regression https://bugs.launchpad.net/snappy/+bug/1819629 [10:56] Bug #1819629: no default locale in core18 causes pam errors [10:56] Bug #1819629 opened: no default locale in core18 causes pam errors [10:57] ogra: thanks - that sounds hopefully straightforward to fix, maybe sil2100 can give 1819629 a stab [10:59] yeah, just some "echo C.UTF-8 >/etc/default/locale" somewhere would be enough [10:59] (in the build) [11:00] Oh, indeed [11:01] ogra, sil2100 yeah, the new build system has a "static" dir, we could simply drop static/etc/default/locale into the core18 tree and its fixed :) [11:01] awesome [11:02] sil2100, also https://forum.snapcraft.io/t/cant-find-the-spi-interface/7523 ... you might need to pull the latest upstream into the CanonicalLtd branch for the pi3 gadget SPI interface fix [11:03] i wonder if we could put the snapcore/pi3-gadget tree to rest somehow witout breaking all the forks of it (not sure if github supports re-owning branches and keeps the references) [11:26] zyga what happens when I change base in snap between revisions, will snapd correctly run new revision agains different base? [11:27] ondra: when all processes of that snap terminate, yes [11:28] zyga is there way to check what base it's working [11:28] ondra: yes, sure, just nsenter and look === alan_g_ is now known as alan_g [11:29] zyga hmm for me refresh failed, because I think in post-refresh hook it collided on libc versio [11:30] ondra: can you provide a reproducer? [11:31] zyga testing now, let me make sure I have it all right [11:31] ondra: you can set SNAPD_DEBUG=1 and see what happens [11:31] but I don't know how to set that to see hook output [11:32] ondra: there's a spread test for this feature if you want to see how it behaves [11:32] zyga I get hook error and that indicates there was libc collision, but let me validate clean install first [11:33] https://github.com/snapcore/snapd/blob/master/tests/main/stale-base-snap/task.yaml [11:33] jamesh, hey [11:34] jamesh, yesterday I was working with a test error [11:34] jamon, [11:34] jamesh, the test desktop-portal-filechooser on snapsd [11:34] jamesh, https://paste.ubuntu.com/p/wMhG5vdyzK/ [11:35] the test just fails like this on ubuntu 18.04 [11:35] zyga OK so install opengrok from stable, then try to refresh to edge [11:36] the error itself is that the snap can't write in the mount dir because of lack of permissions [11:36] jamesh, any idea if this has been already seen/reported? [11:37] zyga stable is with core16, edge it base core18, there is libpthread library which depends on libc version, this is used in post-refresh hook, which fails [11:37] jamesh, in ubuntu 18.10 works fine [11:37] zyga clean install from edge works [11:37] zyga so only difference I can think og [11:39] ondra: perhaps something was still running? [11:39] it's interesting [11:39] that post refresh hook can run with old libc [11:39] I mean [11:39] it's possible to craft this [11:39] so that it will happen as you described [11:39] perhaps snapd should disallow base changes that don't discard the mount namespace [11:40] pedronis: ^ [11:40] zyga so post-refresh runs with previous base? [11:40] ondra: please report this [11:40] ondra: yes, it is possible that this happens [11:40] ondra: when an app is still active [11:40] ondra: or perhaps hooks don't depend on their teremination and we start post refresh hook while another hook is running [11:40] zyga doesn't it stop all services before refresh? [11:40] ondra: it does, but there are hooks and apps [11:41] those can run longer [11:41] and there are enduring services [11:41] it warrants a bug [11:41] this is funny actually [11:41] zyga there should not be any app running, so just hooks [11:41] because it means we cannot correctly refresh in some cases [11:41] ondra: perhaps more than one bug then (base awareness across refresh and hook ordering) [11:41] zyga have you managed to reproduce it? [11:41] no [11:42] I have not tried [11:42] I just explained how it is 100% possible to do that [11:42] zyga right :) [11:42] ondra: curious, what was the base change? [11:42] core -> core18 [11:42] or back? [11:44] zyga core->core18 [11:46] zyga a bit related, what is correct base in snap.yaml for core16? core16 or core? [11:51] ondra: and is libc incompatible across that hop? [11:52] zyga I'm using custom libpthread which I ship in snap, so it's missing things from libc [11:54] ahh [11:55] zyga this is error I'm getting mkdir: relocation error: /snap/opengrok/34/lib/x86_64-linux-gnu/libpthread.so.0: symbol __tunable_get_val, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference [11:59] ondra: please report a bug, after an errand later today I will write a test that reproduces thiis [11:59] ondra: it's unfortunate because it's a design omission [12:00] zyga sure I will file bug for it [12:00] thank you for another interesting case :-) [12:01] zyga you are welcome :) [12:11] zyga: we always run at most one hook per single snap [12:11] at a time [12:12] pedronis: and we consider the hook "done" when the initial process exits [12:12] zyga: ? [12:12] pedronis: we probably can craft a hook that leaves a background process behind [12:12] probably [12:12] is that the case here though? [12:13] PR snapd#6562 closed: timings: base API for recording timings in state [12:13] no, I don't think so, unless that was accidentally what had occurred [12:13] but regardless of what happened here the issue of running a revision that clearly demands base snap X with the stale base snap Y needs addressing [12:13] yes [12:13] shouldn't stale detection find that out? [12:14] different base, different device? [12:14] it is different [12:14] I bet it's not changed because the namespace is still used [12:14] howerver here I'd argue that different base snap name should be detected [12:14] but we have no logic to recover base snap name from the mount namespace [12:15] (or if it can be done, in general, reliably, apart from parsing path names) [12:20] pstolowski: the doc comments in the branch you merged still say Timing a lot [12:20] instead of Span [12:20] unless I'm confused [12:22] pedronis: uhm you're right, sorry.. i'll address in a followup, this doc will be updated anyway with new helpers [12:22] pstolowski: yes, fine for a follow up [12:26] Chipaca: the failure on the deprecate interface PR is real, something going with xgettext-go [12:26] and the warning [12:26] panic: unknown type: interfacesDeprecationNotice [12:27] * Chipaca kills all tests [12:27] pedronis: I'll fix [12:27] after lunch tho [12:27] :) [12:29] niemeyer: https://pbs.twimg.com/media/D1dQcvqXcAAaQKs?format=jpg&name=large [12:30] since you were asking about this yesterday [12:30] mvo, is it ok to move 2.37.4 to stable right? [12:30] Chipaca: There we go.. :) [12:30] anyone up for a review on #6485 ? [12:30] PR #6485: interfaces/seccomp: regenerate changed profiles only [12:41] mborzecki: it has grown quite a lot [12:41] mborzecki: I can enqueue that for later today [12:42] I need to leave in ~15 minutes [12:42] thanks! [12:42] mborzecki: enqueued :) [12:44] mborzecki: it would be easier to review if the new package added was its own PR [12:45] pedronis: i can split it [12:45] mborzecki: if it's not too much painful, I would prefer, it would go faster on the review side that way [12:46] * zyga goes to the doctor === ricab is now known as ricab|lunch [12:50] cachio: let's chat about 2.37.4 in the standup [12:51] pedronis, ok [13:02] Chipaca: https://github.com/snapcore/snapd/pull/6556#issuecomment-471990121 could that be because of ; in the message? [13:02] PR #6556: cmd/snap: hide 'interfaces' command, show deprecation notice [13:02] mborzecki: no, we use ; elsewhere [13:02] mborzecki: it's probably the i18n.G() of const === nessita_ is now known as nessita [13:03] checking now [13:03] off to pick up the kids [13:04] how [13:04] how does update-pot work any more [13:04] ogra / mvo: I'm trying to rebase my fork of ogra's kiosk gadget snap (https://github.com/ogra1/pi-kiosk-gadget) onto the core18-related changes of the “universal Pi gadget snap” (https://github.com/snapcore/pi-gadget). Building the core18 gadget works fine, but my Pi 3B does not boot at all … green LED flickers shortly and then dies, no HDMI output whatsoever. I tried to selectively apply changes from both repos from the point where [13:04] they diverted (commit 013c2931230a4b4de419080f8499fca27c51727d), but no success. Do you know if the current revision of the universal Pi gadget works? [13:04] 'go install' now installs binaries in ~/bin/ [13:04] :-| [13:04] dot-tobias, got a link to your current code ? [13:05] oh wait, no, i have GOBIN set for some reason [13:05] hrmm [13:05] arent you in england ? shouldnt that be GOWASTEBASKET instead ? :P [13:06] ogra: https://en.oxforddictionaries.com/definition/bin [13:07] ogra: 'rubbish bin' is british (vs 'trash') [13:07] oops [13:07] ogra: bloody europeans, coming over here, stealing our language [13:08] lol [13:11] ogra: https://gitlab.com/glancr/gadget-snap-pi-kiosk/ -> master builds fine with LXD container without base (though the attempt to feed mir-kiosk the whole config file does not validate in ubuntu-image); branch core18-WIP has the core18 changes in snapcraft.yaml: https://gitlab.com/glancr/gadget-snap-pi-kiosk/commit/d91cc2386af0da24a8942e52053469f4b920ec7b (other files can be ignored) [13:13] dot-tobias, the linux-modules package contains dtb files ?!? [13:14] (i highly doubt that) [13:14] is that a change made by you or did you copy it from the universal branch ? [13:15] ogra: Got that from the snapcore/pi-gadget upstream, see https://github.com/snapcore/pi-gadget/blame/988ee427691177de5cee84370b096be3c6a07951/snapcraft.yaml#L89 [13:16] but that would indeed explain why it's not booting … (I have close to zero knowledge about this stuff, so apologies and thank you for beginner explanations in advance 😊 ) [13:18] dot-tobias, ok, i checked, devicetrees are in that package, so this is fine [13:20] o dont see anything else significant that could prevent booting apart from the fact that you use the distro compiler, but that should be v7 or newer anyway in bionic [13:20] s/o/i/ [13:21] how does your prime folder look like after build ? [13:21] you should fine the RPi bootloader blobs in there as well as uboot2.img and uboot3.img [13:21] re: compiler: yeah, that's what I gathered from https://github.com/snapcore/pi-gadget/pull/9 [13:21] PR pi-gadget#9: RFC: use ubuntu cross gcc to build uboot [13:21] let me check [13:22] actually in prime/boot-assets/ [13:23] $ ls ../../pi3/pi3-gadget/prime/boot-assets/ [13:23] bcm2709-rpi-2-b.dtb bcm2710-rpi-cm3.dtb cmdline.txt COPYING.linux fixup.dat fixup_x.dat overlays start_cd.elf start.elf uboot.bin [13:23] bcm2710-rpi-3-b.dtb bootcode.bin config.txt fixup_cd.dat fixup_db.dat LICENCE.broadcom psplash.img start_db.elf start_x.elf [13:23] thast what you want to see [13:23] *that's [13:23] Ok, thanks. Rebuilding the snap just to be sure, BRB [13:26] ogra: Side-note – Do you happen to know how I can feed a multiline file to a “defaults” entry in gadget.yaml? I'm trying to accomplish setting mir-kiosk's display config file through the device hook (see https://forum.snapcraft.io/t/prepare-device-hook-questions/9626/5?u=tobias , current attempt at https://gitlab.com/glancr/gadget-snap-pi-kiosk/blob/master/gadget.yaml#L12) [13:27] dot-tobias: > will mangle things for you [13:27] dot-tobias: also the quotes are spurious [13:27] dot-tobias: paste your yaml into http://yaml-online-parser.appspot.com to see what works :-) [13:29] dot-tobias: note these are comments about the yaml itself, I don't know if what you're trying to do should work :-) [13:29] i'm also not sure if the mir team has even tested this [13:29] probably ask in #mir-server [13:30] I tried with a pipe before and the parser complained while building the snap, but maybe I did something else wrong. The online parser shows valid JSON with \n linebreaks at least, will test with ubuntu-image 😊 Yeah, but having a valid yaml result is at least one step closer to the goal 😄 Thank you! [13:32] cachio: 2.37.4 stable> yes - I see no reason why not [13:33] ogra: Snap build fore core18 version is complete, here's the boot-assets folder: [13:34] ➜ ls boot-assets/ [13:34] bcm2708-rpi-0-w.dtb bcm2709-rpi-2-b.dtb bcm2835-rpi-a.dtb bcm2835-rpi-b-rev2.dtb bcm2837-rpi-3-b.dtb COPYING.linux fixup_x.dat start_cd.elf uboot2.bin [13:34] bcm2708-rpi-b.dtb bcm2710-rpi-3-b.dtb bcm2835-rpi-a-plus.dtb bcm2835-rpi-zero.dtb bootcode.bin fixup_cd.dat LICENCE.broadcom start_db.elf uboot3.bin [13:34] bcm2708-rpi-b-plus.dtb bcm2710-rpi-3-b-plus.dtb bcm2835-rpi-b.dtb bcm2835-rpi-zero-w.dtb cmdline.txt fixup.dat overlays/ start.elf [13:34] bcm2708-rpi-cm.dtb bcm2710-rpi-cm3.dtb bcm2835-rpi-b-plus.dtb bcm2836-rpi-2-b.dtb config.txt fixup_db.dat psplash.img start_x.elf [13:34] dot-tobias, you are missing the files from the config dir (config.txt and cmdline.txt) [13:34] oh, no, i'm just blind [13:35] mvo: standup update: I’m working on splitting the big refactoring PR; going well and I have several patches that can be proposed but I’m waiting till the whole complete “split” branch is functioning. Apart from that I briefly interacted and tested MX Linux where there is some desire to run snapd. I will try one more time with init system set to systemd. I plan to re-review Maciej’s seccomp optimization PR. [13:36] zyga: [13:36] And as I shared I’m at a doctor undergoing small surgery (all is good, no limbs lost) [13:36] ogra: I don't see uboot{3}.img however, or did you mean uboot.bin? [13:36] zyga: uh get well! [13:36] No no, all is good, nothing to worry about [13:36] dot-tobias, yeah [13:37] if this same stuff ends up properly in the system-boot partition of your image i dont really know what to say ... this should definitely boot [13:37] do you see these files in the toplevel of the boot partition if you plug the SD into your desktop ? [13:40] Chipaca: "there are files left in the git tree after the tests:" ?? cmd/snap-seccomp/... you talked aobut this before, didn't you? [13:41] mvo: where are you getting this? [13:41] Chipaca: at the end of a unit test spread run [13:41] Chipaca: I thought I saw you talking about something similar earlier [13:41] Chipaca: did I misremember? [13:41] mvo: not today, not for ages [13:42] Chipaca: ok, then please ignore me [13:42] mvo: 'git status' should list things [13:42] Chipaca: yeah, it happend on spread [13:42] Chipaca: maybe it was a hickup, no worries [13:43] pedronis: about that branch from this morning. I can split it exactly where it was approved before [13:43] zyga so I can only reproduce it on one machine, on other machines when I do clean install and then refresh it just works [13:43] ondra: very interesting [13:43] Is one machine slower? [13:43] zyga very likely much slower [13:43] pedronis: I didn’t mean to introduce bumps into the review process [13:44] zyga I can try to repro on other slow machine [13:44] ondra: can you please tell me more about the hooks in play [13:44] What kind of hooks are used [13:44] And how are the hooks implemented [13:45] zyga https://github.com/kubiko/opengrok-snap/blob/master/snap/hooks/install [13:45] zyga install and post-refresh are same [13:45] ondra: it’s hard for me to look now (I only have my phone and cannot move) [13:45] So please walk me through this [13:46] zyga nothing special there, couple of cp and mkdir and then snapctl get and snapctl set [13:46] nothing else [13:46] Ah [13:46] Hmmm [13:46] Anything using &? [13:46] (Shell job control and stuff) [13:47] zyga nope [13:47] zyga all running in straight line [13:47] So there are install and post-refresh only, right? [13:47] zyga also error is from first line of that hook [13:47] Are there any services? [13:47] Perhaps we don’t wait enough for service shutdown [13:48] We now kill just the main process, right? [13:48] zyga there is configure which does nothing [13:48] Inside services [13:48] zyga yes there is service [13:48] (Or am I mistaken?) [13:48] zyga I have stopped service with systemctl and it still happened [13:48] Aha [13:49] That is very useful data [13:49] Can you reproduce it on the slow machine if you remove the service from snap yaml? [13:49] zyga there is one time activated service and tomcat, I stopped both and same thing [13:50] I wonder if you can reduce this to just the hooks [13:50] Or if services play a role [13:55] zyga but if I remove service it will be new revision, so would that make difference as service from new snap should not be running before post-refresh hook, or am I missing sometning? [13:55] zyga so I cannot repro on another slow machine with clean istall [13:58] zyga BTW unrelated bug: ERROR cannot discard snap namespace "opengrok", will retry in 3 mins: cannot discard preserved namespace of snap "opengrok": cannot unlink opengrok.mnt: Device or resource busy [13:58] zyga not able to remove snap which was installed with try [13:58] zyga Remove security profile for snap "opengrok" (x4) (cannot find installed snap "opengrok" at revision x4: missing file /snap/opengrok/x4/meta/snap.yaml) [13:59] zyga but /snap/opengrok is empty dir, so something out of sync [14:01] cachio: was out an event earlier. I haven't seen that error before. It looks like it is coming from the xdg-document-portal FUSE file system though === epod is now known as luk3yx [14:05] mvo: I'm having trouble connecting to the stand up! I'll keep trying (I'm in London) [14:05] zyga removing service made no difference [14:06] cachio: I suspect the ENOSYS is this one: https://github.com/flatpak/xdg-desktop-portal/blob/master/document-portal/document-portal-fuse.c#L730 -- but I'm not sure why the test would trigger that code path: the test should be running with a clean document portal DB [14:06] ogra: So I rebuilt the core18 gadget from scratch, added “base: 18” to my model assertion and dd'd the image to an SD card. All files show up in the root folder, but there's an additional folder “pi2-kernel_83.snap” which I didn't notice before. Contains additional dtbs. And unfortunately, the Pi still won't boot 😞 HDMI output turns the screen on, but nothing visible and green LED doesn't blink once. IIRC that means “no no” … [14:07] ondra: excellent [14:07] PR snapd#6556 closed: cmd/snap: hide 'interfaces' command, show deprecation notice [14:07] ondra: please report the try issue as well [14:07] zyga but as I said, this was service in snap to be installed, so it'd not affect already installed snap [14:08] I’m happy that the hook is sufficient to reproduce [14:08] Ah [14:08] Did you try removing the service from both revisions? [14:08] dot-tobias, anything on serial (i forgot if u-boot actually prints to the display or serial only) [14:09] ogra: Bummed to say that I don't really know (or even have the necessary hardware?) how to do that. Will read up a bit on that and get back to you. Thank you! === Kamilion|ZNC is now known as Kamilion === aleksander_ is now known as aleksander [14:10] jamesh, I suppose it is something not being cleaned up corrently in that case [14:10] dot-tobias, https://www.amazon.de/SainSmart-USB-TTL-Console-Raspberry/dp/B00KVUSI30 you want something like this [14:11] jamesh, it is weird because the package is installed/uninstall as part of the test [14:11] cachio: I used to have some paranoid clean up in the prepare for these tests, but was asked to remove it and trust other tests to leave things in a pristine state [14:11] degville: thanks for letting me know [14:11] * dot-tobias tips hat to ogra [14:12] cachio: I suspect it is some user-level configuration rather than something that would be affected by package install/uninstall [14:14] jamesh, it is weird that I can't reproduce that error on ubuntu 18.10 [14:14] with the same code === ricab|lunch is now known as ricab [14:15] cachio: yeah. We've got 1.0.3 on both 18.04 and 18.10 [14:16] cachio: scratch that. On bionic 1.0.3 is still waiting in proposed [14:17] jamesh, mmm, I'll try to test with 1.0.3 on bionic to see it that fix the issue [14:17] jamesh, I'll let you know the results} [14:18] cachio: we definitely had problems when Cosmic had xdg-desktop-portal 1.0.2, which were fixed by 1.0.3. Bionic seemed fine with the older 0.11, but we've been trying to get 1.0.3 rolled out to all the old releases. [14:19] It wouldn't surprise me if there is some other corner case in the older version that I didn't hit when testing [14:23] zyga and when I return 0 as result of the hook, then snapd is tricked and refresh works [14:23] ondra: well, that's expected [14:23] zyga though of course there are all the errors there [14:23] the beauty(*) of shell [14:23] zyga yep [14:24] zyga so I have only one machine I'm able to reproduce this one [14:26] zyga ha, was did you say was trick to check what core I'm running against if I can snap run --shell [14:26] ondra: it's easier to use nsenter [14:26] ondra: nsenter -m/run/snapd/ns/foo.mnt [14:26] then see what / is [14:27] zyga nsenter: neither filename nor target pid supplied for ns/mnt [14:27] -marg, not -m ar [14:28] zyga because even now installed, it still seems to run with wrong base [14:28] nesenter cli parsing is very strict [14:29] zyga nsenter: reassociate to namespace 'ns/mnt' failed: Invalid argument [14:29] what is the command you ran exactly? [14:29] zyga when running nsenter -m/run/snapd/ns/opengrok.mnt [14:29] sudo? [14:29] actually [14:29] no [14:29] zyga same with sudo [14:29] EINVAL means it's not mounted anymore [14:29] that's ... odd [14:29] we used to unmount but keep the file [14:29] but recently we always unlink it for clarity [14:31] ondra: note that snap-confine may not save a new mount namespace if construction fails [14:31] ondra: and may discard a mount namespace when it is stale [14:32] ondra: but that should not happen here (construction should not fail) [14:33] zyga so I still get into that namespace despite the error [14:33] zyga and it's indeed using core16 base [14:34] ondra: uh? [14:34] the erorr is ... od [14:34] either setns worked or it did not work [14:35] ondra: I'm at risk of getting lost here, can you please collect this information in the bug report [14:35] zyga https://paste.ubuntu.com/p/99Pz3wQ79P/ [14:35] ondra: I think the bug may be separate from the core -> core18 bug [14:35] ondra: that is, there is something wonky with hook dependency (CC pstolowski) [14:35] zyga on machine where it works I get: https://paste.ubuntu.com/p/jhCTxk7mwH/ [14:36] ondra: that nsenter looks ok [14:36] no EINVAL anymore [14:36] zyga see the difference in library version [14:36] 2.23? [14:36] zyga yep, that is wrong glibc [14:36] zyga should be 2.27 [14:36] ondra: please refresh my memory, core16 has 2.23 and core18 has 2.27? [14:37] ondra: can you run findmnt and tell me what / is please? [14:37] zyga correct [14:37] ondra: so, one observation, it would be neat to always have logging in snap-confine and log that to syslog or something [14:37] ondra: this way we would know what happened for sure [14:37] zyga / /dev/loop0 [14:37] losetup -l? [14:38] oh [14:38] one sec [14:38] so [14:38] zyga /dev/loop0 0 0 1 1 /var/lib/snapd/snaps/core_6130.snap (deleted) [14:38] SNAP_CONFINE_DEBUG=yes snap run --shell ... [14:38] oh? [14:38] deleted is certainly interesting [14:38] (the plot thickens) [14:38] ondra: that will now tell you what snap-confine detected [14:39] it should tell you that the mount namespace is stale but occupied [14:39] zyga on other machine it looks correct [14:39] is that snap really deleted on the fs? [14:39] zyga already seeing loop0 indicates something is hanged :) [14:39] deleted is very nasty :/ [14:40] wait, this is core! [14:40] right? [14:40] ooooh [14:40] man I know [14:40] so [14:40] we have a bug [14:40] * zyga slows down to think [14:40] zyga /var/lib/snapd/snaps/core_6130.snap does not exist at all [14:40] ondra: specifically this is ubuntu-core 16 right? [14:40] a core system [14:40] zyga classic [14:41] aha [14:41] in that case it's back to regularly interestngi [14:41] zyga 18.04 [14:41] I suspected there are bugs in core->core18 sans reboot refreshes [14:41] but let's think [14:41] do you have snap changes that indicate something happening to revision 6130? [14:42] I suspect it was just garbage collected [14:42] zyga yeah nothing in changes [14:42] ok [14:43] SNAP_CONFINE_DEBUG=yes snap run --shell ... [14:43] what does that say? [14:43] (use the real snap name there) [14:44] ondra: (suspense music plays) [14:45] zyga https://paste.ubuntu.com/p/7w6nRX4Y4J/ [14:45] DEBUG: sending information about the state of the mount namespace (discard) [14:45] jamesh, same error with 1.0.3 [14:45] DEBUG: the mount namespace is stale and should be discarded [14:45] jamesh, I have a debug session open, do you want to take a look? [14:45] when you do this you must keep something in there running [14:46] now you should see core18 / [14:46] ondra: as I said, when the error happens there are still some processes lingering [14:46] ondra: so snap-confine knows it's supposed to discard but chooses not to [14:47] ondra: can you confirm that / is now core18? [14:47] zyga no / is core16 still in that name space [14:48] ondra: DEBUG: found process 6540 belonging to user 0 [14:48] I missed that [14:48] what is that process [14:49] zyga funny ps -aux does not show anything running from old revision [14:49] ondra: coin on the edge [14:49] ondra: it's a thread [14:49] can you find that in /proc please? [14:49] ondra: a bit of go thread is running and it inhabits that namespace somehow (wild guess) [14:51] zyga I assume reboot will fix this as whatever is hanging will die, that stale core revision seems pretty old [14:51] ondra: don't reboot [14:51] ondra: find out what process 6540 is [14:51] what's /proc/6540/exe ? [14:52] ondra: inspect it from the outside please) [14:53] zyga I refreshed back to base core18 snap, debug log was from revision running against core16 and that was functioning [14:53] ondra: can you reproduce this enough to see the dangling process and find out what it was please? [14:54] ondra: you can look at /sys/fs/cgroup/freezer/snap.$SNAP_NAME/cgroup.procs for the list of inhabitants [14:54] ondra: (to avoid using snap-confine) [14:54] zyga now same on broken revision, process id I see from snap confine is ssh-agent [14:55] oh [14:55] how did that get there? [14:55] Chipaca: thanks for the review [14:55] zyga snap is running ssh-agent [14:55] ondra: so there's a ssh-agent running in that mount namespace [14:55] ondra: and that is a "service" (but not) [14:55] ondra: who starts it? [14:55] zyga it's forked [14:55] is that a service / app in the snap>? [14:56] zyga service will check if it's running and starts it [14:56] the service that is a part of the snap? [14:56] zyga I suppose I can add it as service [14:56] ondra: if you forgete refreshes for a second [14:56] if you stop the service who started i [14:56] does it stop ssh-agent? [14:56] or is that a process that is not stopped across restart (effectively) [14:56] zyga no it will not stop it when you stop service [14:56] ondra: so that's that [14:56] ondra: it's a feature [14:57] ondra: we don't discard the mount namespace when there are apps or hooks running that would then observe old/new fs together with other processes in that snap [14:57] zyga shall I kill the process and test refresh? [14:57] ondra: refresh-app-awareness would refuse to refresh your snap [14:57] ondra: please [14:57] ondra: we do have a bug [14:57] ondra: about core / core18 [14:57] zyga not yet [14:57] or in general where base snap name changes [14:58] that should behave in a different (yet undesigned way) [14:58] PR # closed: snapd#5644, snapd#5822, snapd#5915, snapd#6098, snapd#6108, snapd#6177, snapd#6238, snapd#6258, snapd#6270, snapd#6325, snapd#6327, snapd#6329, snapd#6341, snapd#6347, snapd#6360, snapd#6367, snapd#6401, snapd#6404, snapd#6410, snapd#6418, snapd#6436, snapd#6485, snapd#6491, [14:58] snapd#6502, snapd#6541, snapd#6553, snapd#6559, snapd#6564, snapd#6575, snapd#6577, snapd#6583, snapd#6585, snapd#6588, snapd#6589 [14:59] zyga lol different error cannot unlink opengrok.mnt: Device or resource busy [14:59] ondra: uname -r? [14:59] ondra: that's a known kernel behavior on old (3.10) kernels [14:59] PR # opened: snapd#5644, snapd#5822, snapd#5915, snapd#6098, snapd#6108, snapd#6177, snapd#6238, snapd#6258, snapd#6270, snapd#6325, snapd#6327, snapd#6329, snapd#6341, snapd#6347, snapd#6360, snapd#6367, snapd#6401, snapd#6404, snapd#6410, snapd#6418, snapd#6436, snapd#6485, snapd#6491, [14:59] snapd#6502, snapd#6541, snapd#6553, snapd#6559, snapd#6564, snapd#6575, snapd#6577, snapd#6583, snapd#6585, snapd#6588, snapd#6589 [15:00] zyga I'm on 4.15 [15:00] ondra: that's new then :) [15:00] ondra: new and not fun [15:00] ondra: can you run SNAPD_DEBUG=1 snap-discard-ns [15:00] with the snap name [15:00] well [15:00] specifically [15:00] unlink is expected to fail [15:01] but we should have unmounted it before [15:01] run stat on that file [15:01] and stat -f please [15:01] perhaps before you discard it [15:01] zyga but since I killed that process I'm now running core18 [15:01] to see what's currently there [15:03] zyga nothing with stat -f [15:03] nothing as in? [15:06] zyga https://paste.ubuntu.com/p/7DmG45TYnz/ [15:06] ID: 0 Namelen: 255 Type: nsfs [15:07] this is relevant [15:07] it's a mounted namespace [15:07] so it cannot be unlinked [15:07] the question is: why did snap-discard-ns not unmount it> [15:07] zyga so I will redo snap to run ssh-agent as service, I guess this should then fix it [15:07] which version of snapd are you on? [15:08] zyga 2.37.2 [15:09] I forgot when snap-confine started using snap-discard-ns but I suspect it's been before than that [15:09] what happens if you want to discard the mount namespace and use SNAPD_DEBUG=1 SNAP_CONFINE_DEBUG=yes snap-discard-ns? [15:10] (note: make sure to use the right binary, reexec wise) [15:11] zyga where do I get snap-discard-ns [15:11] ondra: it's right next to snap-confine [15:11] either in /usr/lib/snapd or in the core snap in the same path [15:12] zyga https://paste.ubuntu.com/p/fGPY9WNRrp/ [15:12] ondra: what's the version of snapd on the hostt? [15:12] woah [15:12] that's neat [15:12] PR snapd#6590 closed: daemon/api: filter connections with hotplug-gone=true [15:12] so it unmounted it, supposedly without erorrs [15:12] zyga 2.37.2 [15:13] and then it ... failed to unlink it because it is still mounted [15:13] can you re-run [15:13] with strace please? [15:13] (ditch the debug options) [15:13] and report this as a new bug [15:13] it's a weird kernel behavior, unless we're missing something [15:15] * zyga cannot wait for the strace [15:15] zyga https://paste.ubuntu.com/p/SkBMzTPpHX/ [15:15] zyga so you want bug with which exactly? :) [15:15] zyga I will paste those longs in, jut to know how to call it [15:15] woah [15:15] I see [15:15] fstatfs(5, {f_type=TMPFS_MAGIC, f_bsize=4096, f_blocks=255810, f_bfree=255557, f_bavail=255557, f_files=1279048, f_ffree=1278375, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NOEXEC|ST_RELATIME}) = 0 [15:16] so... [15:16] findmnt? [15:16] it looks like a kernel bug [15:17] we look at the .mnt file, see it's a TMPFS (not nsfs), try to unlink it and hit a EBUSY error [15:17] can you repeat the stat -f call on the file agani [15:17] you reported it was nsfs before [15:17] is it still? [15:18] PR core18#119 opened: Add a default locale [15:20] ondra: ? [15:20] * zyga is dying from suspense [15:20] PR snapd#6590 opened: daemon/api: filter connections with hotplug-gone=true [15:22] zyga yeah now it's tmpfs [15:22] ondra: findmnt [15:22] is it a bind mount? [15:23] or /proc/self/mountinfo if you fancy that instead [15:23] zyga findmnt in the namespace? [15:23] outside, all of discard should be done outside [15:23] inside that _IS_ expected to happen [15:23] (AFAIK) [15:23] zyga https://paste.ubuntu.com/p/4S3VJyJJCv/ [15:24] │ └─/run/snapd/ns/opengrok.mnt nsfs[mnt:[4026532246]] nsfs rw [15:24] outside it is mounted [15:24] were you running discard from within the mount namespace by any chance? [15:24] no from outside [15:24] and that strace where it showed to be a tmpfs, that was outside as well? [15:25] yes [15:25] ondra: please report this [15:25] all from outside [15:25] ondra: one bug for base core -> core18 design change on running processes [15:26] ondra: one bug on discard that we just talked about [15:26] ondra: I think there are no other things to report at this time, do you agree? [15:26] zyga agree [15:26] zyga I will do bugs tonight, now meetings [15:26] ondra: *thank you* [15:27] this is super valuable [15:27] * zyga hugs ondra [15:27] zyga: sorry, i saw you ping but haven't followed the long discussion you had with ondra, is there anything i should look at re hooks? [15:27] pstolowski: no, it's all clear now [15:27] there was a hidden app running [15:27] all is good [15:28] * zyga heads home [15:28] afk for some time [15:28] ok [15:29] zyga: I'm not sure that PR needs to be resplit but I made some comments there [15:29] zyga you are welcome and thank you! this was intense :D [15:36] mvo, we have core 2.37.4 on stable now [15:36] pedronis, ~ [15:36] thx [15:37] cachio: thank you! [15:39] * cachio lunch === cjwatson_ is now known as cjwatson [15:48] PR core18#120 opened: Backport wpa_supplicant.service.d/snap.conf from core [15:50] PR core18#119 closed: Add a default locale [15:50] sil2100: thanks! [15:51] mvo: yw! The locale one was very trivial so I just ekhm, self-merged it [15:51] We'll make sure it goes out with the nearest snap [15:52] xnox: do you mind if I do a systemd SRU? with the timedatectl fix and the new 1819728 fix? [15:52] sil2100: thank you! [15:52] mvo, go go go =) [15:53] mvo, but also if mark files a proposed regression bug report, it will be on you to revert ;-) [15:53] xnox: cool, thank you [15:53] xnox: sure, its perfectly safe ;) [15:53] xnox: already in the upstream systemd etc [15:54] yeah yeah, ddstreet thought so too =) [15:54] (since 1y almost) [15:54] yet got caught up by missing pieces of things =))))) [15:54] mvo, most of them do go through fine. [15:55] xnox: heh - thats fine, I need to prepare the details and then I will show you the final debdiff [15:55] xnox: so tomorrow [15:56] xnox: still thanks for the green light [16:06] * zyga is home [16:17] PR snapd#6591 opened: [RFC] managers: basic measurements === devil is now known as Guest10579 [16:22] PR snapcraft#2498 closed: python plugin: graceful ret when no packages set === pstolowski is now known as pstolowski|afk [18:36] pedronis: hey, just for my own prioritizing, when are you shooting for a 2.38 upload to disco? [18:39] mvo: hey, maybe that is a question for you: just for my own prioritizing, when are you [18:39] shooting for a 2.38 upload to disco? [18:41] PR snapcraft#2500 opened: DRAFT: Add remote build [18:42] jdstrand: probably end of this week, why? anything pending? [18:43] mvo: nothing for you. I can't upload apparmor 2.13 without 2.38 is all [18:43] (2.13 will break snap removes with system where apparmor is enabled) [18:47] jdstrand: ok [19:04] * cachio afk [19:17] PR snapcraft#2501 opened: nodejs pluging: support for type str bin entries === Eleventh_Doctor is now known as Pharaoh_Atem [19:26] PR snapcraft#2496 closed: many: support the use of build-aux/snap [19:26] I was hoping that 2.38 would have the selinux stuff :( [20:50] jdstrand: is it possible personal-files could grow exec to go with read and write? [20:50] jdstrand: i have an app which uses a go library called byteexec which writes go byte arrays as executables in ~/.byteexec, but fails for me because while it can write, it can't exec [20:55] popey: it isn't really meant for that and that could easily lead to exec transition conflicts. why can't it write to SNAP_USER_DATA (which is $HOME to the snap)? [21:00] jdstrand: no, this doesn't seem to work https://github.com/getlantern/byteexec/blob/master/byteexec.go#L111 [21:01] popey: patch that or file a bug to have them look at HOME? [21:03] popey: alternatively, the snap could use personal-files for ~/.bytecode and then create a symlink from there to SNAP_USER_DATA. it is a hack, but should work [21:04] huh, that's interesting [21:04] * popey breaks out the wrapper scripts [21:05] jdstrand: layouts! [21:06] user.Current opens up /etc/passwd to find out the user, so a code patch could work too [21:07] jdstrand: if we are going to commit to HOME being SNAP_USER_DATA perhaps it would be a good idea to have the bind mounted /etc/passwd path to that as home [21:08] sergiusens: it won't work for the user's home atm [21:08] better than bind mounting is likely an nss module that is snap aware [21:13] jdstrand: yeah, but the go std lib opens /etc/passwd and reads straight from there (I haven't looked in detail or seen how they solve nis and such) [21:14] that a dumb way to do it. the home may not even be in /etc/passwd [21:14] maybe they have a fallback that does nss [21:20] should $HOME point to the users outside home dir? [21:20] because now I'm wondering how my launcher can know what the home dir outside the snap is named [21:22] popey: $HOME should be set to SNAP_USER_DATA when the snap starts. you can use 'getent passwd $(id -u) | cut -d : -f 6' to see what the system has to say about it [21:23] hah, that's memorable [22:13] jdstrand: ok, moving things and symlinking helped my next issue is that the binary which runs is trying to set the system proxy. I don't see any interfaces which can do that. Is it something I should request? [22:21] popey: that is a surprising request for your app but I see no reason why we couldn't have an interface for it. of course, it depends on what it is trying to actually do, but a forum post is good [22:21] What should I capture? strace? :) [22:22] it's a blob which tries to set the proxy [22:26] hmm, looks like it rummages in gsettings. I'll start a thready, thanks [23:23] hi all. i'm trying to build a snap that depends on mono. the version of mono from apt is too old, so trying to work out how to pull in the latest versions directly [23:23] something roughly equivalent to => echo 'deb https://download.mono-project.com/repo/ubuntu stable-bionic main' | sudo tee /etc/apt/sources.list.d/mono-official-stable.list [23:24] from what i've read so far, this doesn't seem to be possible?