[00:31] <solrac> hello
[05:05] <mborzecki> morning
[05:30] <mup> PR snapd#5664 closed: interfaces: workaround for activated services and newer DBus <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/5664>
[05:36] <mborzecki> mvo: morning
[05:37] <mvo> hey mborzecki
[05:37] <mborzecki> mvo: regarding experimental.<flag>= we seem to have a problem there
[05:37] <mvo> mborzecki: uh, ok, tell me more
[05:37] <mborzecki> mvo: it's actually quite silly https://paste.ubuntu.com/p/tVGVZjstcm/
[05:38] <mvo> mborzecki: ohhh, i see
[05:39] <mvo> mborzecki: its because of the bool i guess
[05:39] <mborzecki> mvo: yeah, should be a quick fix
[05:39] <mvo> mborzecki: funny, so it accepts null as false
[05:39] <mvo> mborzecki: but not ""
[05:39] <mvo> mborzecki: thanks
[05:39] <mborzecki> mvo: null == nothing to unmarshal, so a bool stays in its default value
[05:40] <mvo> mborzecki: aha, nice
[05:40] <mvo> mborzecki: wasn't aware of this
[05:41] <mborzecki> mvo: even funnier, the corecfg code unmarshals to a string and allows ""
[06:23] <mup> PR snapd#5675 opened: overlord/snapstate: improve feature flag validation <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5675>
[06:23] <mborzecki> mvo: ^^
[06:31] <mborzecki> mvo: updated #5671 too
[06:31] <mup> PR #5671: tests: basic test for parallel installs from the store <Parallel installs> <Simple> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5671>
[06:43] <mvo> mborzecki: yay, thank you
[07:00] <zyga> good morning
[07:00] <zyga> I'm not really here, just checking which office to go to handle the car paperwork
[07:01] <mvo> zyga: good morning
[07:01] <mborzecki> mvo: that initialiazation is for the case when there is nothing to unmarshal or null (cause json, null is untyped :))
[07:01] <mvo> mborzecki: aha, ok
[07:01] <mborzecki> zyga: hey, so you're off-really-off today or just regular off?
[07:02] <zyga> mborzecki: I'm swapping for the weekend
[07:02] <mborzecki> zyga: if the former we can chat about s-c on monday
[07:02] <zyga> mborzecki: but after I handle the paperwork I will return
[07:02] <zyga> mborzecki: and we can chat
[07:02] <zyga> or we can chat straight away now since I'm here
[07:02] <mborzecki> haha so the 'regular off' ;)
[07:02] <zyga> haha, so that's what you meant by "regular off" :D
[07:03] <zyga> I guess that's only fair :D
[07:03] <mborzecki> off-but-not-off
[07:09] <pstolowski> morning
[07:23] <mborzecki> pstolowski: heyah
[07:31] <Chipaca> moin moin
[07:32] <Chipaca> is the lxd snap busted?
[07:32] <mborzecki> google:ubuntu-16.04-32:tests/main/lxd seems to be failing
[07:32] <Chipaca> yeah
[07:32] <Chipaca> same here
[07:32] <mborzecki> Chipaca: pedronis: hellos
[07:32] <Chipaca> mborzecki: hi
[07:32] <mvo> hey pstolowski and Chipaca
[07:33] <Chipaca> mvo: o/
[07:34] <mvo> Chipaca: I added some comments to the dump-db PR, I think a slightly more generic format for the output would be nice. I'm thinking about the field spearator, \ff is used right now
[07:34] <Chipaca> mvo: yep, and wotsisname said it'd be fine
[07:34] <mvo> Chipaca: do you think ":" is reasonable? or shall we go with something else?
[07:35] <Chipaca> mvo: an emoji would be frouned on, i  guess
[07:36] <Chipaca> : is reasonable
[07:36] <mvo> Chipaca: heh, ok
[07:40] <pedronis> mvo: I'm looking again at the changes in  device_asserts.go and now I'm very confused
[07:41] <mvo> pedronis: can I help fix that somehow?
[07:41] <pedronis> mvo: this should go away no:   https://github.com/snapcore/snapd/blob/master/asserts/device_asserts.go#L248 ?
[07:42] <mvo> pedronis: yes, sorry, that was a oversight, let me kill it (with fire)
[07:42] <pedronis> mvo: why is this here and not one level up:  https://github.com/snapcore/snapd/blob/master/asserts/device_asserts.go#L163 ?
[07:42] <pedronis> mvo:  the new branch needs a similar check for gadget, no?
[07:43] <pedronis> mvo: checkModel means check the "model" header
[07:43] <mvo> pedronis: indeed, let me fix that too
[07:44] <mvo> pedronis: plus some gadget track error tests are missing (which of course would have found the issue)
[07:45] <pedronis> mvo: yea,  I found these because I had the nagging feeling that something was missing in the new PR, it was too short :)
[07:45] <pedronis> and so I went back to see what we did for kernel
[07:46] <mvo> pedronis: thanks for noticing! I will generalize it a bit, I get the feeling that this will come again
[07:46] <mvo> pedronis: should I split the PR up?
[07:46] <pedronis> mvo: as your prefer
[07:47] <mvo> pedronis: ok, I think I do that then
[07:47] <mvo> pedronis: do you have an opinion about "Gagdget() SnapWithTrack" vs "Gadget() string and GadgetTrack() string" ?
[07:52] <pedronis> mvo: I think I prefer the latter until we can do something outside of asserts
[07:53] <mup> PR snapd#5669 closed: asserts,image: support gadget tracks in the model assertion <Core18> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/5669>
[07:55] <mvo> pedronis: ok, thats fine, its easy enough to fix later especially if/when we get support for this in snap install
[08:03] <mup> PR snapd#5676 opened: asserts: add support for gadget tracks in the model assertion <Created by mvo5> <https://github.com/snapcore/snapd/pull/5676>
[08:03] <mvo> pedronis: the first part -^
[08:10] <pedronis> mvo: reviewed
[08:12] <mvo> pedronis: yay, thank you
[08:15] <mup> PR snapd#5654 closed: cmd/snap-confine: establish snap directory mappings for parallel instances <Parallel installs> <Created by bboozzoo> <Closed by bboozzoo> <https://github.com/snapcore/snapd/pull/5654>
[08:16] <mup> PR snapd#5677 opened:  image: add support for "gadget=track" <Created by mvo5> <https://github.com/snapcore/snapd/pull/5677>
[08:16] <mup> PR snapd#5678 opened: snapstate: add support for gadget tracks in model assertion <Created by mvo5> <https://github.com/snapcore/snapd/pull/5678>
[08:17] <pedronis> mvo: this is are all for 2.35, right?
[08:18] <mvo> pedronis: correct
[08:18] <mvo> pedronis: I added tags now
[08:19] <Chipaca> so
[08:20] <Chipaca> selftest is failing in lxd in 16.04-32
[08:21] <mvo> Chipaca: uh, what is the error?
[08:21] <mvo> Chipaca: I mean, what part of the selftest fails?
[08:21] <Chipaca> error: cannot start snapd: cannot mount squashfs image using "squashfs": mount: /tmp/selftest-mountpoint-487148902: mount failed: Unknown error -1
[08:22] <Chipaca> lxd fails to start the first time with an error, and it restarts and doesn't print the logs leading up to the error either, which is suspicious
[08:22] <Chipaca> Aug 17 09:12:18 autopkgtest lxd.daemon[18103]: ==> Setting up persistent shmounts path
[08:22] <Chipaca> Aug 17 09:12:18 autopkgtest lxd.daemon[18103]: [08:22] <Chipaca> Aug 17 09:12:18 autopkgtest lxd.daemon[18103]: ln: failed to create symbolic link '/var/snap/lxd/common/lxd/shmounts': No such file or directory
[08:23] <zyga> ohhh drat, I need to remove the lxd quirk in 18
[08:23] <zyga> or maybe I did
[08:24] <zyga> the lxd quirk is applied on all classic systems
[08:24] <zyga> _hmmm_
[08:24]  * Chipaca goes for more coffee
[09:03] <mborzecki> Chipaca: root@my-ubuntu:~# systemd-detect-virt --help
[09:03] <mborzecki> bash: /usr/bin/systemd-detect-virt: Numerical result out of range
[09:03] <mborzecki> Chipaca: that's inside lxc container
[09:04] <Chipaca> mborzecki: why does that start with bash:?
[09:04] <Chipaca> mborzecki: i mean, that's a bash error?
[09:04] <mborzecki> Chipaca: heh, beats me, no clu
[09:04] <Chipaca> yes
[09:05] <Chipaca> mborzecki: it's an error from bash
[09:05] <Chipaca> because
[09:05] <Chipaca> *EXECVE RETURNED THAT*
[09:05] <mborzecki> Chipaca: root@my-ubuntu:~# /usr/bin/systemd-detect-virt --container
[09:05] <mborzecki> bash: /usr/bin/systemd-detect-virt: Numerical result out of range
[09:05] <mborzecki> omg
[09:05] <Chipaca> execve("/usr/bin/systemd-detect-virt", ["/usr/bin/systemd-detect-virt"], [/* 12 vars */]) = -1 ERANGE (Numerical result out of range)
[09:05] <mborzecki> Chipaca: so if that fails, useFuse() => false, mount is done with -t squashfs which fails
[09:05] <mborzecki> root@my-ubuntu:~# mount -t squashfs $PWD/data.squashfs /mnt/
[09:06] <mborzecki> mount: /mnt/: mount failed: Unknown error -1
[09:06] <mborzecki> Chipaca: like this ^^
[09:06] <ogra> do you have squashfuse installed ?
[09:06] <Chipaca> mborzecki: it gets more interesting
[09:06] <Chipaca> mborzecki: do a getcap of systemd-detect-virt
[09:07] <mborzecki> Chipaca: Failed to get capabilities of file `/usr/bin/systemd-detect-virt' (Numerical result out of range) ?
[09:08] <Chipaca> mborzecki: yes
[09:43] <Chipaca> stgraber: in 16.04 i386 (only), installing lxd from stable and launching an unprivileged container results in weirdness: /usr/bin/systemd-detect-virt fails to execve, returning ERANGE
[09:50] <Chipaca> sparkieg`: is that a typo for a german war on spas
[09:50] <sparkiegeek> Chipaca: heh, glitch in the matrix, combined with a non-friendly unique-naming scheme in my IRC client :)
[09:51] <Chipaca> sparkiegeek: you could've gone with 'yes'
[09:54] <mborzecki> guys, how abou we disable tests/main/lxd on *-32 until this is resolved?
[09:54] <pedronis> Chipaca: ah, interesting, I was wondering if systemd-detect-virt coul fail when ways that weren't just I'm not a container
[09:55] <pedronis> s/when/in/
[09:55] <pedronis> Chipaca: I might have even asked mvo at some point to put more defensive code around it
[09:55] <Chipaca> pedronis: I doubt it's systemd-detect-virt itself
[09:55] <Chipaca> pedronis: it never gets to have a say in the matter
[09:56] <Chipaca> pedronis: (the execve call fails)
[09:56] <pedronis> Chipaca: ok, but our code anyway assumes it means we are not virtualized ?
[09:56] <Chipaca> yes, yes it does
[09:56] <pedronis> that was more my point
[09:56] <pedronis> anyway
[09:57] <Chipaca> we should probably bail there instead of assuming tbf
[09:57] <mborzecki> maybe we should bubble the error up
[09:57] <mborzecki> otherwise it's rather cryptic while anything fails at this stage
[09:59] <Chipaca> mborzecki: dunno, stgraber is often up really early
[10:00] <mborzecki> Chipaca: i can open a PR and we can close it if a solution is found soon(ish)
[10:01] <Chipaca> what's VGAuthService
[10:02] <Chipaca> nm
[10:02] <Chipaca> mborzecki: sure
[10:07] <mborzecki> damn, that test has a whitelist of systems
[10:10] <Chipaca> fun fact: byobu-config will lock up the whole everything
[10:10] <mup> PR snapd#5679 opened: tests/main/lxd: run ubuntu-16.04 only on 64 bit variant <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5679>
[10:11]  * Chipaca was looking to see if any other binaries failed to exec in the same way
[10:18] <mborzecki> Chipaca: anything else failed?
[10:18] <Chipaca> mborzecki: my patience
[10:18] <mborzecki> heh ;)
[10:20] <Chipaca> I should probably step away from the forum for a bit
[10:24] <mborzecki> anyone feels like looking at https://github.com/snapcore/snapd/pull/5614 ?
[10:24] <mup> PR #5614: interfaces: parallel instances support, extend unit tests <Parallel installs> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5614>
[10:25] <mup> PR snapd#5680 opened: [RFC] hotplug: handling of simple add/remove scenario <Blocked> <Hotplug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/5680>
[10:25] <pstolowski> uh, what
[10:26] <pstolowski> (about byobu-config)
[10:28] <Chipaca> pstolowski: inside a snapped lxd, inside kvm, inside spread, running 'byoby-config --help </dev/null >/dev/null' locks the whole thing up
[10:28] <Chipaca> byobu*
[10:29] <pstolowski> finny
[10:29] <pstolowski> *funny
[10:29] <Chipaca> viry finny. hilirius, ivin
[10:30] <pstolowski> just checked the dictionary in case the word finny exists and means something. not in my dict ;)
[10:30] <Chipaca> pstolowski: 'abounding in fishes', fwiw
[10:31] <Chipaca> pstolowski: http://www.dict.org/bin/Dict?Form=Dict1&Query=finny&Strategy=*&Database=*&submit=Submit+
[10:32] <pstolowski> aha
[10:32] <Chipaca> sergiusens: poing
[10:51] <Chipaca> why do people sell things they call "Ubuntu" with just random crap running as the kernel
[10:51] <Chipaca> >:-(
[10:52] <ogra> well, its a massive improvement ... the last openvz servers i saw (when deugging something similar together with zyga) was 2.x or early 3.x
[10:53] <ogra> surprisingly openvz finally supports 4.x kernels :)
[11:00] <Chipaca> ogra: that's why snapd ships a test squashfs :-)
[11:00] <Chipaca> ogra: https://github.com/snapcore/snapd/blob/master/selftest/squashfs.go#L55
[11:01] <niemeyer> Chipaca: So looking at snapshotstate, the last missing point is the last id name
[11:01] <niemeyer> Chipaca: "last-snapshot-set-id"?  It's a mouthful, but has precedence in other options
[11:01] <Chipaca> niemeyer: what do you mean?
[11:02] <niemeyer> Chipaca: The "snapshots.last-id" thing, and the comment from me and pedronis in the PR
[11:02] <Chipaca> ah
[11:02] <Chipaca> snapshots.last-set-id is what it is now
[11:04] <Chipaca> which seems alright to me, if we need to add more info about snapshots it won't be out of place in there
[11:04] <Chipaca> dunno
[11:07] <Chipaca> niemeyer: I think both approaches are fine (I mean: snapshots.last-set-id is fine, and a toplevel last-snapshot-set-id is fine; anything more structured and I'm going to call YAGNI on it)
[11:07] <Chipaca> exactly which names are best, I don't know
[11:08] <niemeyer> Chipaca: I think pedronis had a point about "snapshots" generally being a map of actual snapshots per other cases
[11:08] <niemeyer> Chipaca: And we indeed have the last-foo-bar-id case already in other places
[11:09] <niemeyer> last-refresh, last-refresh-hints, last-change-id, last-task-id
[11:09] <niemeyer> "ubuntu-core-transition-last-retry-time"
[11:10] <niemeyer> /o\
[11:12] <mvo> did we figure out more about the lxd issue btw?
[11:12] <Chipaca> mvo: <Chipaca> stgraber: in 16.04 i386 (only), installing lxd from stable and launching an unprivileged container results in weirdness: /usr/bin/systemd-detect-virt fails to execve, returning ERANGE
[11:12] <mvo> Chipaca: heh, woah, ERANGE
[11:12] <Chipaca> mvo: getcap of the file _also_ fails with ERANGE
[11:13] <Chipaca> mvo: so we're about to learn something about _something_
[11:13] <mvo> Chipaca: what a surprising error
[11:13] <mvo> Chipaca: yeah, its amazing
[11:13] <niemeyer> Chipaca: I've +1d assuming that's tuned per agreement.. someone else needs a final +1 too
[11:13] <niemeyer> pedronis ^
[11:14] <Chipaca> man, i'm shaking
[11:14] <Chipaca> whoa
[11:14] <Chipaca> ok
[11:14] <Chipaca> niemeyer: thanks
[11:14] <pedronis> Chipaca: mvo: yes, ERANGE usually makes me think of math libraries
[11:14] <Chipaca> I didn't expect the emotional response from myself ¯\_(ツ)_/¯ good thing i'm going on holidays next wednesday :-)
[11:14] <mvo> Chipaca: uhhhh, snapshot going in?
[11:15] <Chipaca> mvo: got a +1 from niemeyer
[11:15] <mvo> pedronis: heh, exactly
[11:15] <Chipaca> ooooh, somebody's jealous :-p
[11:15] <ogra> Chipaca, dmesg -H means he needs to understand that he is in a pager ... i'd have suppressed the -H
[11:15] <Chipaca> ogra: maybe TERM isn't set or something stoopid
[11:16] <ogra> or that, yeah
[11:16] <pedronis> mvo: sorry for being annoying about context,  but is really mostly meant to have  a place talk back to itself or connected places talk between unrelated or user layers
[11:17] <pedronis> I mean it's Value feature
[11:18] <pedronis> niemeyer: yes either me or mvo need to do a 2nd pass of snapshotstate
[11:20] <pedronis> any consistent reason why all newish PR seems to be red ?
[11:20] <Chipaca> pedronis: lxd
[11:20] <Chipaca> pedronis: 16.04-32 lxd errors
[11:21] <pedronis> the ERANGE issue ?
[11:21] <Chipaca> pedronis: yes
[11:21] <Chipaca> EDERANGED
[11:21] <pedronis> fun :(
[11:21] <Chipaca> ERANGE is not even listed as a return value for execve
[11:21] <Chipaca> :-)
[11:22] <Chipaca> (but it's probably something to do with the xattrs)
[11:22] <Chipaca> if I had to guess, I'd guess that
[11:22] <Chipaca> because systemd-detect-virt is one of the very few files in 16.04 that uses caps (via xattrs)
[11:22] <Chipaca> in fact, i should look at the other ones, d'oh
[11:22]  * Chipaca does that
[11:24] <Chipaca> pedronis: dunno if you noticed but mvo wasn't online when you were apologising to him
[11:24] <pedronis> no
[11:24] <Chipaca> pedronis: maybe you just wanted to get it off your chest :-D
[11:25] <mborzecki> https://github.com/snapcore/snapd/pull/5679 shall we pull the trigger?
[11:25] <pedronis> maybe I didn't complete his nick
[11:25] <mup> PR #5679: tests/main/lxd: run ubuntu-16.04 only on 64 bit variant <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5679>
[11:28] <ogra> "The linux beginners course with ogra and Chipaca"
[11:28] <ogra> session 1 ...
[11:28] <ogra> :)
[11:29] <Chipaca> ogra: "If you survive with both your kidneys, [...]"
[11:29] <ogra> lol
[11:30] <Chipaca> ogra: as an aside, what on earth have they done to that poor "Ubuntu" that dmesg doesn't work
[11:31] <ogra> good question
[11:32] <ogra> lol
[11:32] <ogra> "no entries"
[11:32] <ogra> probably he run no kernel at all !!!!
[11:33] <Chipaca> ogra: it's secretly just running WSL
[11:38] <Chipaca> YESS
[11:38] <Chipaca> it's the capabilities
[11:38] <Chipaca> mtr fails in the exact same way
[11:39] <Chipaca> as does traceroute6.iputils
[11:40] <sergiusens> Chipaca: what's up?
[11:41] <mvo> Chipaca: should I merge 5679 or is the solution so close that its not worth adding the workaround?
[11:41] <Chipaca> mvo: NFI about the solution -- merge away
[11:41] <Chipaca> sergiusens: WHY DIDN'T I GIVE MYSELF MORE CONTEXT WITH THE PING :-(
[11:41] <Chipaca> sergiusens: now I have _no_ idea what it was about
[11:41] <Chipaca> it was, like, six context switches ago
[11:42] <mup> PR snapd#5679 closed: tests/main/lxd: run ubuntu-16.04 only on 64 bit variant <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/5679>
[11:42] <Chipaca> sergiusens: I hope I'll remember and ping you again
[11:42] <Chipaca> oh!
[11:42] <Chipaca> sergiusens: i remembered :-D
[11:42] <Chipaca> sergiusens: were you aware of 'snap watch --last=auto-refresh?'
[11:43] <Chipaca> sergiusens: coming in 2.35
[11:45] <sergiusens> Chipaca: yes we are https://github.com/snapcore/snapcraft/blob/master/snapcraft/internal/build_providers/_snap.py#L263
[11:45] <sergiusens> but I think I might want to disable refreshes for a lot longer to not get killed mid build :-)
[11:45] <Chipaca> sergiusens: no no
[11:45] <Chipaca> sergiusens: the question mark at the end of the change type
[11:46] <Chipaca> sergiusens: means "no error if none found"
[11:46] <Chipaca> sergiusens: or from --help, “A question mark at the end of the type means to do nothing (instead of returning an error) if no change of the given type is found.”
[11:46] <sergiusens> Chipaca: oh, then I can get rid of the suppress. I must say though that that syntax is hard to spot given you phrased the sentence as a question :-)
[11:47] <Chipaca> sergiusens: mbuahaha
[11:47] <Chipaca> sergiusens: (sorry)
[11:47] <sergiusens> and I did an improper quote match
[11:47] <Chipaca> tut tut
[11:47]  * sergiusens needs to put his glasses on
[11:47] <Chipaca> sergiusens: anyway, 2.35+, so you probably can't use it yet
[11:47] <sergiusens> Chipaca: still good to know!
[11:47] <Chipaca> sergiusens: but, with our conversation abour --check-skeleton  from the other day i thought i'd call it out to you
[11:48] <sergiusens> all is good :-)
[11:51] <Chipaca> ogra: wasn't there a file in proc to tweak the kernel log level? we could ask this person to try that
[11:52] <ogra> Chipaca, not sure if in /proc ... there is a sysctl setting you can apply though
[11:52] <ogra> mvo, this is an interesting one https://forum.snapcraft.io/t/set-system-proxy-from-custom-snap-service/6926
[11:52] <Chipaca> ogra: sysctl is just writing to /proc/sys/<thing>
[11:52] <Chipaca> ogra: :-)
[11:52] <ogra> ah, indeed
[11:53] <ogra> so yeah, there is one, i just dont know the node then ;)
[11:53] <Chipaca> ogra: but yes better to present it with sysctl
[11:53] <ogra> should be something about log_level
[11:54] <Chipaca> ogra: sys.kernel.printk ?
[11:54] <ogra> Chipaca, /proc/sys/kernel/printk
[11:54] <ogra> yeah
[11:54] <Chipaca> heh
[11:55] <ogra> $ cat /proc/sys/kernel/printk
[11:55] <ogra> 4	4	1	7
[12:04] <Chipaca> ogra: I did not point them to https://i.imgur.com/Pfr9dj0.jpg !  I think I deserve a cookie.
[12:05]  * ogra hands Chipaca a well deserved cookie :D
[12:06] <ogra> i wonder if he paid for that carp ...
[12:06] <Chipaca> ogra: at lunch time? are you mad?!?
[12:06] <ogra> *crap
[12:06] <Chipaca> :-)
[12:06] <ogra> hahaha
[12:06] <Chipaca> ogra: 8GB of ram? you betcha it was paid for
[12:06] <Chipaca> although given they said it was Ubuntu and it wasn't, maybe it's "8GB" of "RAM"
[12:07] <Chipaca> (actually just a big swap file on a 1GB netbook)
[12:07] <ogra> haha
[12:12] <mborzecki> zyga: snap-update-ns is looking good, did a change, it's actually surprisingly simple
[12:12] <zyga> woot, that's great
[12:14] <mborzecki> zyga: you know, i might have screwed something up there too :P
[12:14] <zyga> I'll check next week :)
[12:15] <jdstrand> zyga: hey, fyi, responded to https://github.com/snapcore/snapd/pull/5644
[12:15] <mup> PR #5644: interfaces: add audio-playback/audio-record and make pulseaudio manually connect <Created by jdstrand> <https://github.com/snapcore/snapd/pull/5644>
[12:15] <zyga> jdstrand: thank you
[12:15] <zyga> jdstrand: I'm swapping today, doing office move and legal paperwork
[12:15] <mborzecki> zyga: https://paste.ubuntu.com/p/YZ7w7RKw5d/ mountinfo (does not include $SNAP_USER_DATA yet)
[12:15] <jdstrand> zyga: np. I suspect you'll just agree with me and ack
[12:16] <zyga> I'll look quickly now :)
[12:16] <jdstrand> zyga: but by all means, exercise your day off :)
[12:16] <mup> PR snapd#5675 closed: overlord/snapstate: improve feature flag validation <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/5675>
[12:16] <jdstrand> Chipaca: fyi, I think that hostnamectl issue will be resolved if the PR merges from trunk
[12:17] <jdstrand> Chipaca: and hi :)
[12:17] <Chipaca> jdstrand: I was assuming as much :-) hi
[12:18] <Chipaca> jdstrand: excitement now is about lxd on 16.04-32 being unable to execve files that use capabilities
[12:19]  * Chipaca ~> lunch
[12:23] <ogra> pedronis, Chipaca https://imgur.com/a/ZFNu5pV ... finally able to reproduce ... capturing logs now
[12:23] <zyga> jdstrand: +1
[12:24] <jdstrand> zyga: thanks :)
[12:24] <zyga> marked as such on the PR
[12:26] <mvo> ogra: aha, you reproduced the shutdown hang?
[12:28] <mborzecki> damn, the trackpoint left click in my x220 is starting to fail :(
[12:29] <ogra> mvo, yeah
[12:30] <ogra> mvo, pedronis Chipaca https://pastebin.canonical.com/p/DGKBDMzQ2r/ logs (filtered out binder and anbox audit messages since they ake it unreadable)
[12:30] <ogra> *make
[12:33] <pedronis> internal shutdown seems correct
[12:33] <pedronis> so it would some handshake with systemd problem
[12:34] <pedronis> seems we get a sigterm but don't do the right thing:  snapd.service: State 'stop-sigterm' timed out. Killing.
[12:36]  * mvo wonders if anything has chnaged here
[12:37] <pedronis> mvo: well it might have been like since a while
[12:38] <pedronis> mvo: it's related to the waiting we do on reboot and signal unhandling
[12:38] <ogra> note this is all edge plus a devmode daemon (anbox) though i see the daemon being killed several lines before the snapd timeout shows up
[12:38] <ogra> (also not sure if a misbehaving snap could actually make snapd not stop)
[12:38] <pedronis> mvo: we also added the watchdog
[12:39] <mvo> pedronis: aha, thats a good one
[12:39] <mborzecki> pedronis: Aug 17 12:19:55 localhost.localdomain snapd[1005]: daemon.go:577: Waiting for system reboot ?
[12:39] <pedronis> mborzecki: ?
[12:39] <mborzecki> isn't snapd waiting in a long sleep here?
[12:39] <pedronis> yes
[12:39] <pedronis> as I said signal handling is not quite right over shutdown
[12:39] <mborzecki> so any signals are not really handled
[12:39] <pedronis> yes
[12:39] <pedronis> what I'm trying to say
[12:39] <pedronis> not sure it's related to the timeout though
[12:39] <pedronis> if it's really much later
[12:41] <pedronis> mvo: we need to call Reset or Stop for the signal handling we setup in main.go, not sure exactly where
[12:42] <ogra> if you want to repro: create a qemu VM with an image from edge ... leave it off over night so there is a new core ... make sure to start it only after core has updated in the store... boot it and watch it to do an auto-refresh with that hang
[12:42] <ogra> i have never seen it when doing a manual refresh
[12:44] <mborzecki> pedronis: i'd assume it's related, then things start to make sense, term get queued, if systemd gets a request to restart the process it would make sense to ignore it since the system is going down anyway, systemd timeouts waiting for snapd to exit, snapd would timout waiting for reset :)
[12:45] <pedronis> well, not from the logs  it seems systemd then kills snapd
[12:46] <pedronis> so snapd doesn't timeout
[12:46] <mvo> yeah, I'm puzzled
[12:46] <pedronis> ogra timeout is systemd saying something about snapd again
[12:46] <pedronis> but maybe I'm confused
[12:46] <mvo> if snapd does not handle sigterm correct I should be able to simulate this by simply kill -TERM $(pidof snapd)
[12:46] <mvo> and that exist normally
[12:46] <pedronis> mvo: this is about reboot mode
[12:46] <pedronis> not normal running snapd
[12:47] <ogra> pedronis, i'm seeing exactly whats in the imgur png
[12:47] <pedronis> we are inside daemon Stop at that point
[12:47] <pedronis> mvo ^
[12:47] <mvo> pedronis: ok
[12:47] <pedronis> I mean,  I know what we need to do about sigterm (except exactly how to place it)
[12:47] <pedronis> not sure it fixes what ogra sees
[12:48] <mvo> pedronis: that makes sense, we are in stop at this, so yeah
[12:49] <pedronis> ah, it says A Stop Job so yes, it's related
[12:49] <Chipaca> pedronis: sorry to sidetrack you, but did you see Gustavo's comment about last-snapshot-set-id?
[12:49] <pedronis> yes
[12:49] <Chipaca> pedronis: +1?
[12:49] <pedronis> I agree with that
[12:49] <Chipaca> pedronis: k
[12:56] <pedronis> mvo: something like this perhaps (untested):  https://pastebin.ubuntu.com/p/rYWJ8PR5HH/
[12:58] <mvo> pedronis: yeah, that looks sensible
[13:13] <mup> PR snapd#5318 closed: interfaces/builtin: add new cuda-support interface <Created by anonymouse64> <Closed by jdstrand> <https://github.com/snapcore/snapd/pull/5318>
[13:42] <pedronis> mvo: I can try to cut a PR from that pastebin on monday I suppose unless you want to
[13:44] <mvo> pedronis: I can look at this while waiting for travis to biuld my gadget-track PRs
[13:44] <pedronis> ok
[13:44] <mvo> pedronis: this mostly needs tests, right? the feature itself looks reasonalbe
[13:45] <pedronis> yes, some kind of tests (not sure it's easy to test)
[13:46] <mvo> pedronis: yeah, it looks tricky, maybe I can manage some sort of integration test at least, I will poke a bit at it
[13:47] <pedronis> a 2nd review for #5676 would be great
[13:47] <mup> PR #5676: asserts: add support for gadget tracks in the model assertion <Core18> <Created by mvo5> <https://github.com/snapcore/snapd/pull/5676>
[13:48] <mvo> yeah, was about to ask for this - should be straightforward (now)
[13:55] <Chipaca> stgraber: ping?
[13:56] <stgraber> Chipaca: pong
[13:56] <Chipaca> stgraber: I don't know if you saw that we're having issues with lxd since the release yesterday?
[13:57] <stgraber> Chipaca: I just read your comment about systemd-detect-virt on i386, will take a look
[13:58] <Chipaca> stgraber: it's any executable that uses capabilities
[13:59] <Chipaca> stgraber: systemd-detect-virt, mtr, and traceroute6.iputils
[13:59] <Chipaca> stgraber: but, on 64 bits, snapcraft is also having trouble with snapd dying, that goes away with switching to 3.0
[14:00] <stgraber> Chipaca: what kernel are you on?
[14:00] <Chipaca> stgraber: e.g., on travis, «apt install snapd; snap install snapcraft --classic; snap install lxd; mkdir project; cd project; snapcraft init; snapcraft cleanbuild» fails
[14:00] <Chipaca> not sure what travis is using
[14:00] <Chipaca> I'm on 4.4.0-131-generic; mborzecki is on something newer I think
[14:01] <Chipaca> and the spread runs on 16.04-32 are on fresh cloud images, whatever's shipped there
[14:01] <Chipaca> (I can track it down if it's important)
[14:01] <mborzecki> i was on whatever we used in spread instances
[14:02] <Chipaca> oh wait my machine is 4.4.0-131-generic but the test was run in kvm
[14:02] <Chipaca> I'd have to check what was there :-)
[14:02]  * Chipaca checks
[14:03] <mborzecki> i'll start the test and check what's used in gce
[14:04] <Chipaca> stgraber: 4.4.0-133-generic
[14:04] <Chipaca> running on a 4.4.0-131-generic
[14:07] <stgraber> Chipaca: reproduced the issue
[14:08] <Chipaca> stgraber: should we be installing lxd from candidate in our integration suite, to catch this family of errors before they hit stable?
[14:09] <stgraber> Chipaca: that may be useful, yeah, this is definitely related to fscaps in this case so our own tests didn't trip on that
[14:09] <Chipaca> stgraber: also mborzecki noticed that a privileged container didn't have this issue
[14:10] <stgraber> yeah, that part would be expected
[14:10] <Chipaca> ok
[14:10] <Chipaca> stgraber: does it only failing on 32 bit intel make sense to you?
[14:11] <Chipaca> (this particular failure i mean -- the snapcraft one i am yet to dig into)
[14:12] <stgraber> Chipaca: no, got it failing the same way on amd64
[14:13] <Chipaca> that's very strange
[14:13] <mborzecki> that's good news, right?
[14:13] <Chipaca> mborzecki: how are our tests working on amd64?
[14:13] <mborzecki> Chipaca: they are 'passing'
[14:15] <Chipaca> mborzecki: that's my point: if the bug is there, they shouldn't
[14:15] <Chipaca> mborzecki: I'll dig
[14:15] <mborzecki> stgraber: Chipaca: got a debug shell on i386, do you want to check anything?
[14:15] <stgraber> tracked it down and working on a fix now
[14:15] <mborzecki> stgraber: what's the issue?
[14:16] <stgraber> tl&dr is your kernel doesn't support unprivileged file capabilities, yet it lets us write an xattr that uses that new v3 fscap format
[14:16] <stgraber> but then blows up when reading the file
[14:16] <mborzecki> stgraber: heh, nice
[14:16] <Chipaca> stgraber: so it's a bug in the image itself?
[14:17] <Chipaca> s/bug/regression caused by a change/
[14:18] <stgraber> Chipaca: it's a combination of things, LXD 3.4 introduced logic to remap file caps rather than just strip them and unsquashfs-tools was fixed yesterday to not drop xattrs in 16.04
[14:19] <stgraber> Chipaca: so even in our candidate and edge channels, everything was good until the last snap rebuild yesterday which picked up the fixed unsquashfs
[14:19] <stgraber> and most of our manual testing is done on 4.15 kernels which do support the v3 caps or on 4.4 systems with -proposed enabled that again do support v3 caps (next kernel SRU has a backport of the feature)
[14:19] <Chipaca> ＭＡＧＩＣ
[14:20] <Chipaca> sigh
[14:20] <Chipaca> I'll push a pr that adds --candidate to the lxd integration test
[14:20] <stgraber> and even though we do have snap tests that use the broken kernels, our test image doesn't use file caps (it's just a tiny busybox image)
[14:21] <Chipaca> stgraber: we woudln't've caught it if we hadn't just happened to be using one of the three binaries that have caps in xenial
[14:22] <stgraber> Chipaca: yeah, I expect that detect-virt is what's going to break most users so trying to rush a fix now
[14:29] <Chipaca> stgraber: as soon as the fix is in candidate let me know, and I'll push a pr that turns the test back on and fetches lxd from candidate
[14:32] <Chipaca> mborzecki: I'm guessing amd64 just happened to get a newer kernel or something
[14:32] <Chipaca> dunno
[14:35] <mborzecki> Chipaca: ubuntu-16.04-32 uses 4.4.0-131-generic
[14:35] <Chipaca> mborzecki: I got 133 here, but ok
[14:36] <Chipaca> and indeed if I rebooted I'd have 133 as well
[14:36] <Chipaca> mborzecki: mirror sync
[14:43] <mborzecki> Chipaca: ubuntu-16.04-64 image users different kernel indeed, it's 4.13.0-1019-gcp
[14:44] <Chipaca> haha, hehe. Ha ha ha ha, he he he he
[14:44] <Chipaca> mborzecki: ok
[14:44] <pedronis> big difference
[14:44] <Chipaca> so there was a little tiny window of hitting the bug, and we *nailed* it
[14:44] <Chipaca> :-)
[14:45] <mborzecki> flawless victory
[14:45] <Chipaca> next question: should we make sure -32 and -64 are running similarish kernels?
[14:45] <mborzecki> heh :)
[14:45] <mborzecki> cachio__: ^^ ?
[14:47] <cachio__> mborzecki, yes
[14:48] <cachio__> mborzecki, I think the problem with the function
[14:50] <cachio__> is that "if apt-cache policy "linux-image-extra-$(uname -r)" >/dev/null 2>&1; then" is going always through the then
[14:50] <cachio__> with match or without
[14:52] <mborzecki> cachio__: yeah, something to figure out
[14:52] <mborzecki>  the current pr will effectively switch the kernel to something else
[14:53] <mborzecki> cachio__: i have 4.13.0-1019-gcp and the proposed code wants to install linux-image-extra-4.13.0-31-generic, this will pull in the non-gcp kernel
[14:54] <cachio__> mborzecki, yes, but that linux-image-extra-4.13.0-31-generic works well with the current kernel
[14:54] <cachio__> mborzecki, at least the tests pass
[14:55] <cachio__> so, what should we use when there is no linux-image-extra pkg for the current kernel?
[14:58] <mborzecki> cachio__: i see that there's linux-module-extra for the newer kernels, and linux-image-extra seems to be gone
[14:58] <mborzecki> cachio__: the real issue is that apt-cache policy <nonexistent-package> returns 0 :(
[14:58] <cachio__> mborzecki, yes
[14:59] <mborzecki> damn, and so does apt list
[14:59] <cachio__> let me try installing linux-modules-extra-4.15.0-1017-gcp and running the tests
[15:00] <pedronis> need to look at output it seems
[15:00] <mborzecki> cachio__: can you replace apt-cache policy with apt-cache show?
[15:02] <cachio__> mborzecki, that works
[15:05] <mborzecki> cachio__: something like this perhaps https://paste.ubuntu.com/p/QVSbPQcqWC/
[15:06] <cachio__> mborzecki, yes, makes sense
[15:07] <Chipaca> niemeyer: in spread, is there a reason why I can't say “systems: [ubuntu-*, -ubuntu-14*]”?
[15:08] <cachio__> mborzecki, I pushed that
[15:08] <cachio__> I'll test this with the new gce images to see if it works
[15:14] <mup> PR snapd#5681 opened: overlord: handle sigterm during shutdown better <Created by mvo5> <https://github.com/snapcore/snapd/pull/5681>
[15:14] <mvo> pedronis: ^- this is your earlier pastebin with tests, hope the tests make (some) sense, its a bit tricky
[15:14] <mvo> ogra: ^- this should hopefully fix the reboot timeout
[15:14] <mvo> ogra: I mean the wait
[15:15] <mvo> a second review for 5676 and 5677 would be great, should be easy
[15:25] <ogra> mvo, awesome, will happily test once it landed
[15:26] <t1mp> is there a way to speed up snap building if I already have all the .deb dependencies installed in the system?
[15:27] <ogra> mvo, btw, kyle told me it is likly he had the SD mounted when dd'ing to it ... (yay, nautilus) ... you really need to promote godd more ;)
[15:27] <t1mp> I have a full pipeline on Jenkins now, which runs inside a Docker image that has all the dependencies. Still, the snapcraft step takes a long time because it is re-downloading all the deb files
[15:28] <mup> PR snapd#5676 closed: asserts: add support for gadget tracks in the model assertion <Core18> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/5676>
[15:28] <pedronis> mvo: #5678 has conflicts now, merging master into it would be nice either way
[15:28] <mup> PR #5678: snapstate: add support for gadget tracks in model assertion <Core18> <Created by mvo5> <https://github.com/snapcore/snapd/pull/5678>
[15:30] <mvo> pedronis: yeah, I updated it
[15:30] <mvo> pedronis: and also the snapstate one
[15:31] <mvo> pedronis: when you say "before adding the close probably it would have worked anyway" in the review of 5681 - do you mean that without the close the test I added would behave the same? or am I misunderstanding?
[15:31] <mvo> pedronis: I will add the nil checks, I like that
[15:31] <pedronis> mvo: no, I mean the code would not have explode passing nil in, but with the close it does
[15:32] <mvo> pedronis: aha, got it - will fix it
[15:32] <niemeyer> Chipaca: The logic is simpler.. you start with the fixed list for the file at hand, and can either add or drop.. it's not sequential
[15:32] <mvo> pedronis: I just need the close for my test, I could do it differently but this looked most natural (next to sending sigkill itself in the test which does not work)
[15:32] <pedronis> mvo: the close is ok, but I think making sigCh optional is also natural
[15:33] <niemeyer> Chipaca: At least from my vague memories.. it's been a while
[15:33] <mvo> pedronis: yeah, we are in agreement :)
[15:34] <mvo> pedronis: updated, thanks for the feedback
[15:37] <pedronis> mvo: I would have put the if around the whole bit using sigCh,  Stop supporting nil works but is not super clear from the docs why it should
[15:38] <mvo> pedronis: indeed
[15:47] <Chipaca> mvo: any idea what failing to reset devices.list means?
[15:49] <mvo> Chipaca: do you have more context
[15:50] <pedronis> what is devices.list
[15:51] <mvo> Chipaca: is that an lxc error? related to the lxd devices cgroup?
[15:51] <mvo> Chipaca: or to our deivices cgroup?
[15:55] <mup> PR snapd#5660 closed: wayland: add extra sockets that are used by older toolkits (e.g. gtk3) <Created by gerboland> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/5660>
[15:55] <plars> sergiusens: how often do you normally do a release to stable of the snapcraft snap? Do you spend any time in beta/candidate? it looks like the current set of snaps is stable==candidate==beta, so I'm guessing you just do testing in edge?
[15:56] <plars> sergiusens: the reason I ask, is we'd like to do some testing of our snaps with upcoming releases of snapcraft, trying to figure out which channel would be the best to pull from
[16:02] <sergiusens> plars: when we tag, we put the snap in beta once it passes all our automated tests, then we do internal testing and if it all works we move it to candidate and make an ANN for a call for testing
[16:09] <plars> sergiusens: ok cool, how long do you normally give it in candidate? sounds like that's where we should aim for
[16:19] <Chipaca> mvo: I'm afraid we moved on, but yes related to lxd (not sure if it's lxd or snapd printing that)
[16:29] <Chipaca> pedronis: are you around?
[16:31] <noise][> he EOW'd a bit ago
[16:32] <Chipaca> ah well
[16:33] <Chipaca> pedronis: it was to avoid having to reflash a device to undo a serial assertion, but it'd only save us ~5 minutes :-)
[16:33] <Chipaca> noise][: thanks
[16:44] <t1mp> is there a way to cache a snapcraft pull when I run it in fresh docker images on Jenkins?
[16:44] <t1mp> that stage always takes 10 minutes of installing new deb packages
[16:45] <t1mp> :%s/fresh docker images/fresh docker containers/
[16:54] <sergiusens> plars: 3 days, but we can negotiate on that with something reasonable (like a week)
[16:55] <sergiusens> s/reasonable/expectable/
[16:59] <kyrofa> t1mp, two thoughts: 1) If we're talking build-packages, generate a docker image with them already installed and use that, should speed things up. 2) if we're talking stage-packages, snapcraft caches them in ~/.cache/snapcraft/, you can preserve that between runs
[17:00] <kyrofa> Consider that each of those steps makes the build less "clean"
[17:01] <pedronis> Chipaca: having dinner
[17:02] <t1mp> kyrofa: thanks. That's useful info. I'm snapping a python app, so I don't really need to build anything.
[17:03] <t1mp> kyrofa: maybe it would be nice to have a 'snapcraft create-cache' command that downloads the .debs. I can include that in my Dockerfile since it doesn't change often.
[17:04] <t1mp> basically I only need the 'dump' plugin to copy some files that are already generated in previous steps (using PyInstaller)
[17:04] <t1mp> but setting up the stage takes 20 minutes :(
[17:05] <kyrofa> t1mp, I don't quite understand. If you're only using the dump plugin, why is snapcraft fetching a bunch of stuff?
[17:06] <plars> sergiusens: I don't have any concerns about that. 3 days sounds reasonable. I don't expect we would normally have any problem
[17:10] <t1mp> kyrofa: it has dependencies, see https://pastebin.ubuntu.com/p/tbvGycDvvt/
[17:11] <kyrofa> Ah, the remote part is probably taking a chunk of time
[17:12] <t1mp> yes, and it is repeating every time I make a small change somewhere
[17:12] <t1mp> I really only need to build the snap before publishing a new version, but for now I'm building it for each commit to make sure I don't break it. And to test that the snap building works properly
[17:12] <kyrofa> t1mp, every time you make a small change you fire up a clean docker container?
[17:13] <kyrofa> Ah, commits, okay that's fair
[17:13] <t1mp> yes, Jenkins does
[17:13] <t1mp> for each push
[17:15] <kyrofa> t1mp, if you run `snapcraft define desktop-gtk3` you'll see that it's mostly stage-packages as well
[17:15] <kyrofa> t1mp, stage-packages are not installed on the host, which means creating a new docker image with them installed won't help you
[17:15] <kyrofa> t1mp, but you can try preserving that cache between runs (point (2)) so they don't need to be fetched again
[17:15] <sergiusens> kyrofa, t1mp: you can mount the cache directory into the container
[17:16] <kyrofa> sergiusens, you're stealing my thunder
[17:16] <sergiusens> exactly that
[17:16] <t1mp> I realized that :) I meant to have an image that includes ~/.cache/snapcraft. That would be kind of ugly though. Probably better to mount it.
[17:16] <kyrofa> Indeed
[17:16] <sergiusens> yeah, that (thunder) too :-)
[17:16] <kyrofa> :D
[17:16] <t1mp> sergiusens, kyrofa: right :)
[17:17] <t1mp> that caches only the deb files right? They still need to be installed
[17:17] <kyrofa> t1mp, they're just unpacked into the snap, yeah
[17:17] <kyrofa> Should be quick compared to fetching them
[17:19] <t1mp> yeah pull takes 3 minutes now https://pastebin.ubuntu.com/p/PJqfM6ndQW/
[17:19] <t1mp> (the total time went down from 20 to 5.. I think the server was overloaded before)
[17:19] <t1mp> err.. to 8 min, not 5. :)
[17:20] <t1mp> I'll look into keeping the cache. On Monday :)
[17:20] <kyrofa> t1mp, we'll be here!
[17:20] <t1mp> great, thanks :)
[17:20] <kyrofa> Have a lovely weekend
[17:21] <t1mp> you too
[18:14] <stgraber> Chipaca, sergiusens: we have a fix (https://github.com/lxc/lxd/pull/4943), should land upstream in ~30min, then in snap another 30min or so later
[18:14] <mup> PR lxc/lxd#4943: shared/idmap: test fcaps support <Created by stgraber> <https://github.com/lxc/lxd/pull/4943>
[18:14] <sergiusens> thanks for the update
[19:31] <Chipaca> stgraber: so I could push the PR about now?
[19:32] <stgraber> Chipaca: yeah, we should have the fix in candidate in the next 10min or so
[19:32] <stgraber> Chipaca: once Jenkins is happy on our side, I'll promote to stable
[19:32] <Chipaca> stgraber: the pr I'd push would bring it back for i386, but also pull from candidate
[19:33] <Chipaca> stgraber: (is candidate the right place to pull from?)
[19:34] <stgraber> Chipaca: the issue isn't arch-specific though so you shouldn't make anything specific to i386
[19:34] <stgraber> yeah, we only use edge, candidate and stable
[19:34] <Chipaca> stgraber: the difference is our i386 images have 4.4.x, whereas amd64 has 4.13.x
[19:34] <Chipaca> stgraber: so the issue only manifested on i386
[19:35] <stgraber> Chipaca: looks like we're seconds away from having the fix in candidate now
[19:35] <stgraber> that snap takes so long to build...
[19:36] <stgraber> 52min for that build
[19:36] <Chipaca> stgraber: via snapcraft?
[19:40] <stgraber> Chipaca: yeah on LP, but it's not a simple snap :)
[19:41] <mup> PR snapd#5682 opened: tests/main/lxd: pull lxd from candidate; reënable i386 <Created by chipaca> <https://github.com/snapcore/snapd/pull/5682>
[19:42] <Chipaca> stgraber: tadaa
[19:52] <stgraber> sergiusens, Chipaca: promoting to stable now
[19:52] <Chipaca> woop
[20:19] <kyrofa> Thank you, stgraber