[00:07] Did you see Tino is gone as of today? [06:20] morning! [06:32] good morning :) [06:44] PR snapd#3167 opened: interfaces: fold network bind into core support with tests [06:45] pedronis: so tyhicks is proposing /run/snapd as the place to put the temporary key files [06:46] pedronis: is that reasonable for you? [06:46] good morning [06:46] tvoss: what's up? :-) [06:47] zyga: aiming to finalize the key generation PR [06:47] tvoss: mhm [06:47] mvo: hey [06:48] zyga: tyhicks is proposing /run/snapd to place the temporary key files, I'm fine with that. Do you have any objections? [06:48] mvo: I merged one of the approved PRs last night [06:48] tvoss: who will place them there? [06:49] zyga: snapd itself, when generating the device key. cleaning them up once the operation finishes [06:49] tvoss: /run/snapd/lock and /run/snapd/ns are touched by snap-confine but I think this is fine [06:49] tvoss: maybe you can use a sub-directory, not sure what your plan is fully [06:50] zyga: I just need a place to put the key files temporarily [06:50] tvoss: and /tmp is not good? [06:50] zyga: so readable/writable by snapd only is pretty much the only requirement [06:50] aha [06:52] zyga: so this static/non-static build of snap-confine is really tricky now, why did you drop the the --enable-partially-static thing and switched to individual --enable-static-* switches? [06:52] zyga: with that, /run/snapd seems like a good choice [06:52] morphis: because --enable-partially-static is a no-op in the new scheme, it's always like that [06:52] morphis: you can pick any partially static part of snap-confine [06:53] morphis: and it is not possible to build a fully static snap-confine anyway, because of udev [06:53] tvoss: and /tmp? [06:53] zyga: does snapd get its own /tmp? If not, I would rather put it in a more specific directory [06:54] tvoss: no, snapd doesn't get any magic treatment, /tmp is shared [06:54] zyga: so I would rather go for /run/snapd then [06:54] * zyga is still somehow confused, why /run vs /tmp for temp file [06:54] it's not that /run is bad, I just want to understand why it was picked over /tmp [06:55] zyga: following securities advice. /tmp is world readable, right? [06:55] tvoss: as is run [06:55] tvoss: it's just a matter of giving the right mode to the file [06:56] zyga: it get's the right mode from ssh-keygen already [06:56] tvoss: so there's no difference [06:56] zyga: feel free to comment on the PR, too. [06:58] tvoss: yep, thanks [06:58] tvoss: is that 2.24 material? [06:58] tvoss: or 2.25? [06:58] zyga: well, ideally, we would have it in 2.24 [07:00] * zyga nods [07:05] pedronis: hey [07:05] pedronis: so I'd like to understand how we are going to proceed [07:06] pedronis: (fold and fix later vs fix now) [07:06] pedronis: do I understand correctly that the root issue is how auto-connect works? [07:07] https://forum.snapcraft.io/t/unblocking-2-24/238 is my proposal to get 2.24 out and then fix the deeper issues [07:10] morning [07:11] pstolowski: hello :) [07:11] zyga: so for whatever libtool/autotools reason https://github.com/snapcore/snapd/pull/3162 disables linking libcap statically, it ends up in _STATIC but is linked against the .so [07:11] PR snapd#3162: cmd: use libtool for the internal library [07:12] pstolowski: morning! [07:12] morphis: because the .la files now dictate everything [07:12] morphis: kind of suspected this would happen though, did we merge that already by any chance? [07:12] sure, but shouldn't we be able to override that? [07:13] zyga: we didn't [07:13] tests are detecting that libcap is dynamically linked [07:13] morphis: yes but the way is not the same, I suspect [07:13] mvo: zyga: I answered there [07:13] morphis: look at the resulting makefile [07:13] pedronis: thanks! [07:14] zyga: I do [07:14] zyga: .._STATIC = -lcap is what we have there [07:14] pedronis: so just picking selected bits from 3145 ? let me see [07:16] pedronis: "this part" is fixDisconnectedCorePlugs or the hack further down in autoConnect? [07:16] morphis: those are just variables, look at rules [07:16] zyga: the hack in autoconnect, after thinking it seems we don't need fixDisconnectedCorePlugs ... when we intall the new core we run autoconnect logic again [07:17] pedronis: aha, interesting [07:17] * zyga thinks [07:17] zyga: remember that for core we run setup-profiles twice, once in the old snapd and once in the new one [07:18] yes, that's right [07:18] and the 2nd try will already have the hack and will connect things [07:18] (the plugs that is) [07:19] mvo: I think it will work too [07:19] I worry about folding (a little) that it's a slippery slope we will have to climb out of for some reason we cannot forsee yet [07:19] this is not true for very old snapds but then those have other bugs around running configure too [07:20] (that we will end up needing the privilege separation in core0 [07:20] mvo: I think this is your call ultimately, while we discuss more and more there will be more and more branches proposed/merged [07:21] mvo: so if you want to limit the extent of the changes we put into 2.24, I think the way is to release now [07:21] PR snapd#3168 opened: .travis.yml: add option make raw log less noisy [07:21] zyga, pedronis: I am fine with adding a special case for core/core-support in auto-connect, if its small and targeted, but I really want to get on with 2.24, that is my overriding priority right now (of course while still making sane decisions) [07:22] zyga: release now with 3167? or release now now :) ? [07:22] mvo: "now" as in quickly [07:22] +100 [07:25] pedronis: so your second suggestion is to use folding + special case, a special case like http://paste.ubuntu.com/24359539/ ? [07:25] mvo: well if we also fold the special case could be just about core-support but yes [07:27] pedronis: do you have something simpler in mind already, if so, would you mind sketching it out? I'm fine with fold+special-case as it feels like we at least have everything ready when autoconnect is fully fixed this way. as long as its straightfoward :) [07:28] * zyga would like to know what the fully fixed autoconnect is like [07:29] zyga: thats for later, lets not get distracted ;) [07:29] that's a bigger thing [07:29] * mvo puts on his tunnel vision glasses [07:29] not doing to think about that now (though I have ideas) [07:30] mvo: I think just taking that bit and plug.Interface == "core-support" first would be enough for now , if we fold ... we can improve/change it later [07:33] do you want me to prepare that branch? [07:35] zyga: if you don't mind, as small as possible please :) [07:36] mvo: OK [07:37] zyga: ok, problem is that libtool needs the full-path to the .a we want to statically link [07:39] PR snapd#3164 closed: tests: ensure network-bind and core-support are connected [07:41] PR snapd#3165 closed: interfaces: adjust shm accesses to use 'm' for updated mmap kernel mediation [07:43] I need reviews for 3167 - at leat one, the underlying branch (without the spread test) got a +1 from gustavo already [07:52] mvo: running tests locally [07:53] zyga: great, looking forward for this branch [07:53] to this branch [07:55] mvo: I included the small yaml cleanup as I didn't want to tear that out and it is harmless (just variable rename for correctness) [07:55] mvo: (same that was there originally) [07:56] zyga: sounds good, push it [07:56] * mvo is being pushy [07:57] PR snapd#3168 closed: .travis.yml: add option to make raw log less noisy [08:03] we don't seem to have any spread tests which exercises gadget config defaults, do we? [08:03] pstolowski: no, also that functionality is broken in other ways [08:04] pedronis, how do you mean? [08:04] pstolowski: that at first boot it will not always be applied because of order issues [08:04] of things [08:06] pedronis, oh i see. ty [08:08] something to fix for 2.25 (I think) [08:10] PR snapd#3169 opened: many: fix plug auto-connect during core transition [08:11] pedronis, mvo: ^ [08:12] zyga: could you have a look at 3167 please? [08:12] morning eurofriends [08:12] anything particularly lit today? [08:12] mvo: certainly, looking now [08:13] ta [08:13] Chipaca: morning separatistic islander, nothing particularly on fire [08:13] hey Chipaca! good morning [08:14] Chipaca: I'm having slight tunnel vision to get 2.24 out but otherwise things are fine [08:14] s/slight/huge/ [08:14] mvo— as well you should [08:14] mvo— as long as it's not from getting up too quickly or something :-) [08:14] mvo: the regexpes there feel too generic, if we grow core-support-apparatus they will still match [08:15] mvo: but since tests are passing ignore me for now :) [08:15] zyga: :) happy to do a followup [08:15] mvo: in 2.25 :) [08:15] mvo: I just approved it [08:16] * zyga is getting hungry [08:16] I'll talk to you after some 1-2-1s [08:16] Chipaca: if you don't mind a review of 3167 would be great, should be very simple [08:17] zyga: sure, thanks for your help! [08:17] zyga— my whole house is going to be smelling of slow-cooking beans, onions, peppers and bay leaves today. I think I'm having another breakfast. [08:20] mvo— +1; there's a typo in a comment, but you can fix later [08:21] Chipaca: hahahaa - thats such a freudian slip - cure-support! [08:21] :-) [08:21] mvo— maybe you're just missing '80s rock [08:22] Chipaca, ogra_: and now - https://github.com/snapcore/core/pull/33/files [08:22] PR core#33: the core snap only needs core-support (not network-bind) [08:22] PR snapd#3166 closed: interfaces/builtin: fold network-bind into core-support [08:22] PR snapd#3167 closed: interfaces: fold network bind into core support with tests [08:22] Chipaca: lol [08:22] pretty please [08:22] * mvo is still pushy [08:22] approved [08:22] ta! [08:22] yeah, +1. Bring on the two-line merges. [08:23] heh [08:23] PR core#33 closed: the core snap only needs core-support (not network-bind) [08:24] zyga: does #3169 work from your tests? [08:29] PR snapd#3162 closed: cmd: use libtool for the internal library [08:39] ogra_: and one more for core (sorry for that) [08:39] PR core#34 opened: really remove network-bind-plug [08:41] zyga: mvo: core-support is interesting because is probably the only? one of the only interfaces where plug and slot that connect are on the same snap? [08:41] pedronis: yeah, I think so [08:41] pedronis: network-manager and modem-manager have some smilar properties [08:42] or maybe something related to location support, I recall really wacky archtecture [08:42] but core-support is certainly most common [08:42] mm i'm getting a lot of "2017-04-11 04:37:02 Discarding linode:ubuntu-core-16-64 (Spread-06), cannot connect: cannot connect to linode:ubuntu-core-16-64 (Spread-06): ssh: handshake failed: ssh: unable to authenticate, attempted methods [password none], no supported methods remain" this morning https://travis-ci.org/snapcore/spread-cron/builds/220818086#L382 [08:42] aha [08:42] maybe some other spread collected that machine since it was started? [08:43] zyga: if we have more maybe (I say maybe) it would be interesting to consider the concept of self-plugging , instead of abusing autoconnect for this [08:44] pedronis: implicit or somehow explicitly declared in yaml? [08:45] well somehow/somewhere we would need to say that core-support plug on core is supposed to plug into the same snap slot [08:45] Does anyone know what barriers there are to getting tincd running as a snap? [08:45] anyway if it's just core-support if might be easier to just special case (also because the issue is mostly transition) [08:47] DonAlex: we don't know what tincd is so probably you need to tell us more [08:47] pedronis: I agree we should explore how connecting should work [08:48] pedronis: including a way to say "this snap requires connection before it is functional" [08:48] pedronis: something that is currently not possible [08:51] Chipaca: one more trivial review please https://github.com/snapcore/core/pull/34 - silly me for overlooking this earlier [08:51] PR core#34: really remove network-bind-plug [08:52] mvo— hah [08:52] +1'ed [08:52] zyga: https://tinc-vpn.org/ [08:52] If I am using a snap on a server I want to be able to put it on my VPN [08:56] PR core#34 closed: really remove network-bind-plug [08:58] PR snapd#3170 opened: interfaces/builting: allow read-only access to /sys/module [08:59] DonAlex: we don't have support for user services [08:59] DonAlex: not sure if that is required [08:59] DonAlex: I'd recommend building it as a snap and checking what is missing [08:59] DonAlex: we will gladly take feedback [08:59] DonAlex: as for permissions, you may need to run in devmode initially (not sure) [09:00] DonAlex: devmode means that confinement is advisory, not mandatory [09:00] DonAlex: but we can add new interfaces quickly [09:00] DonAlex: (or extend existing interfaces where appropriate) [09:00] DonAlex: stay in the loop, post on forum.snapcraft.io, try things out [09:00] we are close to user services, there is a PR, let me try to find it [09:01] https://github.com/snapcore/snapd/pull/2752 [09:01] PR snapd#2752: snap: add support user-sessions from snaps [09:01] mvo: so 2.25 is probably more likely to be around the 25th, after yesterday discussion? [09:01] * zyga is done with breakfast [09:01] pedronis: correct [09:01] and heads to 1-2-1 [09:01] mvo: changing the date on github [09:01] pedronis: \o/ [09:02] zyga: I guess the biggest problem will be with creating the device or loading the tun module. [09:02] zyga: That is normally a system action. Though if you use user permission on some system it works. [09:02] DonAlex: tun module can be loaded with the kmod interface [09:02] DonAlex: essentially snapd will load the module for you [09:03] DonAlex: does it take any parameters that need to be tweaked / set [09:03] DonAlex: what devices do you need to create? [09:03] zyga: hmm no i don't think so .. just needs the device to be created [09:03] DonAlex: does the module do that or is that done from userspace? [09:04] zyga: it creates and iterative interface names per network so tun1 tun2 etc. [09:07] mvo, hmmm [09:07] unknown plugs interface name reference 'network-bind-plug' lint-snap-v2_hook_plugs_plug_reference (configure, network-bind-plug) [09:07] ogra_: yeah, see above [09:07] mvo, thats what the store mailed me for the last round of core buillds [09:07] ah, k [09:07] ogra_: the fix is in, Chipaca was kind enough to review. silly me for overlooking a plug reference [09:08] too much tunnel vision :( [09:08] PR snapd#3171 opened: snapstate: normalize gadget defaults [09:08] mvo, well, i'D appreciate a tunnel vision that distracts me from that awful spreadsheet [09:09] ogra_: :( [09:11] oh [09:11] darn, I forgot [09:11] mvo, ogra_, zyga, all the unknowns there are killing me (and there's 2+ weeks to go for those) [09:12] Choh, i thought it was supposed to be done by friday [09:12] Chipaca, [09:12] nope, 2-3 weeks [09:12] sigh [09:12] yep [09:13] mvo, this build got through ! [09:14] ogra_: yeah, I'm running a local test now and fire spread [09:14] up [09:14] Chipaca: In case you didn't see my note yesterday, I've put up that promised demo at https://github.com/mabnhdev/snappy-daemon-demo. [09:15] mikeb_— I didn't, no [09:15] network's been iffy and irc doesn't cope well with that [09:17] mikeb_— why is 'devmode' needed? [09:17] mvo: hi [09:18] mikeb_— anyway, thanks, I'll look into this [09:18] hey woodrowshen [09:18] Ok .. gonna have a go at building it myself. Will let you know what happens via the forums. [09:19] Chipaca: devmode is force of habit - I mostly play with code that violates containment. [09:19] mikeb_— cheeky :-) [09:19] mikeb_— but seriously, thank you, and I'll be playing with this [09:20] might get something into 2.25 if it's concise enough [09:20] Chipaca: I appreciate your time. Thanks. [09:20] (and if we all agree on it) [09:21] mvo: i built snappy image with armhf, and just found core snap rev. is 1580, is it current rev. in stable ? [09:23] woodrowshen: that sounds correct [09:24] woodrowshen: 1580 is the stable core on armhf with snapd 2.23.6 [09:24] mvo: ok, thanks. i met the weird case of failure to exec `snap list` [09:25] re [09:25] how can I help? [09:27] woodrowshen: what is the failure? can you please pastebin it [09:29] zyga: mostly waiting for tests now, the new core breaks the interfaces-network-bind spread test testing a fix locally and then we should be good again [09:31] mvo: http://paste.ubuntu.com/24359921/ [09:32] mvo: just output "error: cannot list snaps: cannot communicate with server: Get http://localhost/v2/snaps: EOF" [09:32] woodrowshen: let me help you [09:32] mvo: focus on the fix :) [09:32] woodrowshen: Apr 11 08:46:07 localhost snapd[1994]: runtime: out of memory: cannot allocate 258801664-byte block (510001152 in use) [09:32] woodrowshen: what system is that? how much ram? [09:32] zyga: ha! thank you [09:32] woodrowshen: how about "snap version" [09:32] zyga: even better [09:32] woodrowshen: and systemctl status snapd.service [09:33] zyga: mvo: thanks a lot [09:33] zyga: ooo... [09:34] PR snapd#3172 opened: tests: fix interfaces-network-bind to match the new core [09:34] zyga: all snap commands can't work [09:36] woodrowshen: how much memory do you have there? [09:36] zyga: 1G [09:37] woodrowshen: what is that system, is it a VM? [09:37] woodrowshen: what dis [09:37] distribution? [09:38] zyga: roseapple-pi, a devel board [09:38] woodrowshen: aha [09:38] woodrowshen: which kernel are you running [09:38] zyga: 3.10.37 [09:39] woodrowshen: and how did you install the system; did you build an all-snap (aka core) image or is this a distro running on your board and you just installed/compiled snapd there [09:39] woodrowshen: with 1GB of ram, are you running anything like a graphical UI there? [09:41] zyga: https://github.com/xapp-le/SnappyUbuntuCore/blob/master/builder/build-snappy.sh [09:41] zyga: i used this script to build it [09:41] woodrowshen: let me see [09:41] zyga: no any graphics satck [09:41] zyga: s/satck/stack/ [09:42] woodrowshen: aha, so it's an all-snap/core image [09:42] zyga: yes, and previous images i had built didn't happen that. [09:43] woodrowshen: can you pastebin "top -b -n 1" [09:43] zyga: i wonder if we have pastebin snap XDDD [09:44] woodrowshen: you can ssh in [09:44] woodrowshen: run that command [09:44] woodrowshen: and copy-paste [09:45] woodrowshen: (into something like pastebin.ubuntu.com [09:45] zyga: http://paste.ubuntu.com/24359952/ [09:47] woodrowshen: does snap version run? [09:47] woodrowshen: or is that crashing too? [09:48] zyga: no, the system still works normally. just snap commands will get stuck [09:48] pedronis: mind having another look here: https://github.com/snapcore/snapd/pull/3130 [09:48] PR snapd#3130: overlord/devicestate: switch to ssh-keygen for device key generation [09:48] zyga: even snap version [09:49] pedronis: switched back ssh-keygen :) [09:49] zyga: sorry fix the description, snap commands will get stuck for a while, and output "error: cannot list snaps: cannot communicate with server: Get http://localhost/v2/snaps: EOF" [09:51] right [09:52] woodrowshen— journalctl -u snapd.service [09:52] please [09:52] hm, unreleated - I think we need spread tests for the core options, I just got a failure trying to snap set core ystem.power-key-action - looks like we have no /etc/systemd/logind.conf.d directiory [09:54] woodrowshen: and after you get that log, can you systemctl restart snapd.service [09:54] woodrowshen: and see if memory usage drops [09:54] woodrowshen: and if you can run snap changes or what not [09:54] Chipaca: http://paste.ubuntu.com/24359985/ [09:55] zyga: do you know if there is a way to allow only a single "mkdir" on a directory via apparmor. i.e. not full write but just mkdir? [09:55] woodrowshen— there you go then [09:56] woodrowshen— this is probably bug #1674778 [09:56] Bug #1674778: snapd hangs with 100% CPU usage when image has a custom gadget snap [09:56] woodrowshen— IOW you probably have a bug in your gadget snap, and we have a bug in snapd where it OOMs on gadget snap weirdness [09:57] to be clear: “runtime: memory allocated by OS (0x7286e000) not in usable range [0x965c2000,0xffffffff)” [09:57] on a 32-bit system [09:57] means OOM [09:57] mvo: yes, directoy and stuff inside are separate [09:57] mvo: e.g. /path/to/dir w, [09:57] er [09:57] zyga: thanks, excellent [09:57] mvo: e.g. /path/to/dir/ w, [09:57] the trailing slash matters [09:58] try that as I suspect you need more than w in practice [09:58] mikeb_— you around? [09:58] zyga: http://paste.ubuntu.com/24359994/ [09:59] right [09:59] zyga: snapd cna't free the memory [09:59] zyga: ok, i see. [09:59] zyga: thanks, running it locally now. I'm adding a test to ensure that snap set core after the transition *actually* really works and hit a bug :) [10:00] mvo: good call, outch, what bug? [10:00] Chipaca: what's the reason for the extra memory use, do you know? [10:00] zyga— no [10:01] somebody should look into that bug at some point :-) [10:01] Chipaca: maybe the hook hangs and we buffer something it says into memory? [10:01] mikeb_— I'm trying to reproduce the issues you're seeing, and I'm not being able to [10:01] zyga— we've seen the OOM also because the gadget asks for an interface it hasn't been granted [10:01] zyga: I push a PR once my local test finishes, but its not super critical, just adding it to really ensure all angles are tested [10:02] zyga— and because its assertion is signed with the wrong key [10:02] zyga— these things happen before the hook afaik [10:02] zyga— also afaict it's not spamming changes; there was a relatively low amount of changes at the time [10:02] aha [10:02] as i say, somebody needs to dig [10:02] yes, for sure [10:04] mikeb_— ah, i might be understanding one of the issues you mention now [10:04] * Chipaca looks further [10:05] Chipaca: do we know if it's reproducible on amd64? undoing a first boot install and OOM ? [10:06] PR snapd#3173 opened: tests: add extra test after the core transition for snap get/set core [10:06] zyga: ^- thats the one I talked about [10:07] aha [10:11] Chipaca: the last comment on bug said removing the interface on hook can fix the issue, but my gadget didn't use that like him [10:11] bye bye [10:11] I'll rejoin if I'll create snappy products [10:11] woodrowshen— that comment comes from a different thread and was posted to that bug in error [10:12]