[00:31] hello [05:05] morning [05:30] PR snapd#5664 closed: interfaces: workaround for activated services and newer DBus [05:36] mvo: morning [05:37] hey mborzecki [05:37] mvo: regarding experimental.= we seem to have a problem there [05:37] mborzecki: uh, ok, tell me more [05:37] mvo: it's actually quite silly https://paste.ubuntu.com/p/tVGVZjstcm/ [05:38] mborzecki: ohhh, i see [05:39] mborzecki: its because of the bool i guess [05:39] mvo: yeah, should be a quick fix [05:39] mborzecki: funny, so it accepts null as false [05:39] mborzecki: but not "" [05:39] mborzecki: thanks [05:39] mvo: null == nothing to unmarshal, so a bool stays in its default value [05:40] mborzecki: aha, nice [05:40] mborzecki: wasn't aware of this [05:41] mvo: even funnier, the corecfg code unmarshals to a string and allows "" [06:23] PR snapd#5675 opened: overlord/snapstate: improve feature flag validation [06:23] mvo: ^^ [06:31] mvo: updated #5671 too [06:31] PR #5671: tests: basic test for parallel installs from the store [06:43] mborzecki: yay, thank you [07:00] good morning [07:00] I'm not really here, just checking which office to go to handle the car paperwork [07:01] zyga: good morning [07:01] mvo: that initialiazation is for the case when there is nothing to unmarshal or null (cause json, null is untyped :)) [07:01] mborzecki: aha, ok [07:01] zyga: hey, so you're off-really-off today or just regular off? [07:02] mborzecki: I'm swapping for the weekend [07:02] zyga: if the former we can chat about s-c on monday [07:02] mborzecki: but after I handle the paperwork I will return [07:02] mborzecki: and we can chat [07:02] or we can chat straight away now since I'm here [07:02] haha so the 'regular off' ;) [07:02] haha, so that's what you meant by "regular off" :D [07:03] I guess that's only fair :D [07:03] off-but-not-off === pstolowski|afk is now known as pstolowski [07:09] morning [07:23] pstolowski: heyah [07:31] moin moin [07:32] is the lxd snap busted? [07:32] google:ubuntu-16.04-32:tests/main/lxd seems to be failing [07:32] yeah [07:32] same here [07:32] Chipaca: pedronis: hellos [07:32] mborzecki: hi [07:32] hey pstolowski and Chipaca [07:33] mvo: o/ [07:34] Chipaca: I added some comments to the dump-db PR, I think a slightly more generic format for the output would be nice. I'm thinking about the field spearator, \ff is used right now [07:34] mvo: yep, and wotsisname said it'd be fine [07:34] Chipaca: do you think ":" is reasonable? or shall we go with something else? [07:35] mvo: an emoji would be frouned on, i guess [07:36] : is reasonable [07:36] Chipaca: heh, ok [07:40] mvo: I'm looking again at the changes in device_asserts.go and now I'm very confused [07:41] pedronis: can I help fix that somehow? [07:41] mvo: this should go away no: https://github.com/snapcore/snapd/blob/master/asserts/device_asserts.go#L248 ? [07:42] pedronis: yes, sorry, that was a oversight, let me kill it (with fire) [07:42] mvo: why is this here and not one level up: https://github.com/snapcore/snapd/blob/master/asserts/device_asserts.go#L163 ? [07:42] mvo: the new branch needs a similar check for gadget, no? [07:43] mvo: checkModel means check the "model" header [07:43] pedronis: indeed, let me fix that too [07:44] pedronis: plus some gadget track error tests are missing (which of course would have found the issue) [07:45] mvo: yea, I found these because I had the nagging feeling that something was missing in the new PR, it was too short :) [07:45] and so I went back to see what we did for kernel [07:46] pedronis: thanks for noticing! I will generalize it a bit, I get the feeling that this will come again [07:46] pedronis: should I split the PR up? [07:46] mvo: as your prefer [07:47] pedronis: ok, I think I do that then [07:47] pedronis: do you have an opinion about "Gagdget() SnapWithTrack" vs "Gadget() string and GadgetTrack() string" ? [07:52] mvo: I think I prefer the latter until we can do something outside of asserts [07:53] PR snapd#5669 closed: asserts,image: support gadget tracks in the model assertion [07:55] pedronis: ok, thats fine, its easy enough to fix later especially if/when we get support for this in snap install [08:03] PR snapd#5676 opened: asserts: add support for gadget tracks in the model assertion [08:03] pedronis: the first part -^ [08:10] mvo: reviewed [08:12] pedronis: yay, thank you [08:15] PR snapd#5654 closed: cmd/snap-confine: establish snap directory mappings for parallel instances [08:16] PR snapd#5677 opened: image: add support for "gadget=track" [08:16] PR snapd#5678 opened: snapstate: add support for gadget tracks in model assertion [08:17] mvo: this is are all for 2.35, right? [08:18] pedronis: correct [08:18] pedronis: I added tags now [08:19] so [08:20] selftest is failing in lxd in 16.04-32 [08:21] Chipaca: uh, what is the error? [08:21] Chipaca: I mean, what part of the selftest fails? [08:21] error: cannot start snapd: cannot mount squashfs image using "squashfs": mount: /tmp/selftest-mountpoint-487148902: mount failed: Unknown error -1 [08:22] lxd fails to start the first time with an error, and it restarts and doesn't print the logs leading up to the error either, which is suspicious [08:22] Aug 17 09:12:18 autopkgtest lxd.daemon[18103]: ==> Setting up persistent shmounts path [08:22] Aug 17 09:12:18 autopkgtest lxd.daemon[18103]: ====> Making LXD shmounts use the persistent path [08:22] Aug 17 09:12:18 autopkgtest lxd.daemon[18103]: ln: failed to create symbolic link '/var/snap/lxd/common/lxd/shmounts': No such file or directory [08:23] ohhh drat, I need to remove the lxd quirk in 18 [08:23] or maybe I did [08:24] the lxd quirk is applied on all classic systems [08:24] _hmmm_ [08:24] * Chipaca goes for more coffee [09:03] Chipaca: root@my-ubuntu:~# systemd-detect-virt --help [09:03] bash: /usr/bin/systemd-detect-virt: Numerical result out of range [09:03] Chipaca: that's inside lxc container [09:04] mborzecki: why does that start with bash:? [09:04] mborzecki: i mean, that's a bash error? [09:04] Chipaca: heh, beats me, no clu [09:04] yes [09:05] mborzecki: it's an error from bash [09:05] because [09:05] *EXECVE RETURNED THAT* [09:05] Chipaca: root@my-ubuntu:~# /usr/bin/systemd-detect-virt --container [09:05] bash: /usr/bin/systemd-detect-virt: Numerical result out of range [09:05] omg [09:05] execve("/usr/bin/systemd-detect-virt", ["/usr/bin/systemd-detect-virt"], [/* 12 vars */]) = -1 ERANGE (Numerical result out of range) [09:05] Chipaca: so if that fails, useFuse() => false, mount is done with -t squashfs which fails [09:05] root@my-ubuntu:~# mount -t squashfs $PWD/data.squashfs /mnt/ [09:06] mount: /mnt/: mount failed: Unknown error -1 [09:06] Chipaca: like this ^^ [09:06] do you have squashfuse installed ? [09:06] mborzecki: it gets more interesting [09:06] mborzecki: do a getcap of systemd-detect-virt [09:07] Chipaca: Failed to get capabilities of file `/usr/bin/systemd-detect-virt' (Numerical result out of range) ? [09:08] mborzecki: yes [09:43] stgraber: in 16.04 i386 (only), installing lxd from stable and launching an unprivileged container results in weirdness: /usr/bin/systemd-detect-virt fails to execve, returning ERANGE [09:50] sparkieg`: is that a typo for a german war on spas === sparkieg` is now known as sparkiegeek [09:50] Chipaca: heh, glitch in the matrix, combined with a non-friendly unique-naming scheme in my IRC client :) [09:51] sparkiegeek: you could've gone with 'yes' [09:54] guys, how abou we disable tests/main/lxd on *-32 until this is resolved? [09:54] Chipaca: ah, interesting, I was wondering if systemd-detect-virt coul fail when ways that weren't just I'm not a container [09:55] s/when/in/ [09:55] Chipaca: I might have even asked mvo at some point to put more defensive code around it [09:55] pedronis: I doubt it's systemd-detect-virt itself [09:55] pedronis: it never gets to have a say in the matter [09:56] pedronis: (the execve call fails) [09:56] Chipaca: ok, but our code anyway assumes it means we are not virtualized ? [09:56] yes, yes it does [09:56] that was more my point [09:56] anyway [09:57] we should probably bail there instead of assuming tbf [09:57] maybe we should bubble the error up [09:57] otherwise it's rather cryptic while anything fails at this stage [09:59] mborzecki: dunno, stgraber is often up really early [10:00] Chipaca: i can open a PR and we can close it if a solution is found soon(ish) [10:01] what's VGAuthService [10:02] nm [10:02] mborzecki: sure [10:07] damn, that test has a whitelist of systems [10:10] fun fact: byobu-config will lock up the whole everything [10:10] PR snapd#5679 opened: tests/main/lxd: run ubuntu-16.04 only on 64 bit variant [10:11] * Chipaca was looking to see if any other binaries failed to exec in the same way [10:18] Chipaca: anything else failed? [10:18] mborzecki: my patience [10:18] heh ;) [10:20] I should probably step away from the forum for a bit [10:24] anyone feels like looking at https://github.com/snapcore/snapd/pull/5614 ? [10:24] PR #5614: interfaces: parallel instances support, extend unit tests [10:25] PR snapd#5680 opened: [RFC] hotplug: handling of simple add/remove scenario [10:25] uh, what [10:26] (about byobu-config) [10:28] pstolowski: inside a snapped lxd, inside kvm, inside spread, running 'byoby-config --help /dev/null' locks the whole thing up [10:28] byobu* [10:29] finny [10:29] *funny [10:29] viry finny. hilirius, ivin [10:30] just checked the dictionary in case the word finny exists and means something. not in my dict ;) [10:30] pstolowski: 'abounding in fishes', fwiw [10:31] pstolowski: http://www.dict.org/bin/Dict?Form=Dict1&Query=finny&Strategy=*&Database=*&submit=Submit+ [10:32] aha [10:32] sergiusens: poing [10:51] why do people sell things they call "Ubuntu" with just random crap running as the kernel [10:51] >:-( [10:52] well, its a massive improvement ... the last openvz servers i saw (when deugging something similar together with zyga) was 2.x or early 3.x [10:53] surprisingly openvz finally supports 4.x kernels :) [11:00] ogra: that's why snapd ships a test squashfs :-) [11:00] ogra: https://github.com/snapcore/snapd/blob/master/selftest/squashfs.go#L55 [11:01] Chipaca: So looking at snapshotstate, the last missing point is the last id name [11:01] Chipaca: "last-snapshot-set-id"? It's a mouthful, but has precedence in other options [11:01] niemeyer: what do you mean? [11:02] Chipaca: The "snapshots.last-id" thing, and the comment from me and pedronis in the PR [11:02] ah [11:02] snapshots.last-set-id is what it is now [11:04] which seems alright to me, if we need to add more info about snapshots it won't be out of place in there [11:04] dunno [11:07] niemeyer: I think both approaches are fine (I mean: snapshots.last-set-id is fine, and a toplevel last-snapshot-set-id is fine; anything more structured and I'm going to call YAGNI on it) [11:07] exactly which names are best, I don't know [11:08] Chipaca: I think pedronis had a point about "snapshots" generally being a map of actual snapshots per other cases [11:08] Chipaca: And we indeed have the last-foo-bar-id case already in other places [11:09] last-refresh, last-refresh-hints, last-change-id, last-task-id [11:09] "ubuntu-core-transition-last-retry-time" [11:10] /o\ [11:12] did we figure out more about the lxd issue btw? [11:12] mvo: stgraber: in 16.04 i386 (only), installing lxd from stable and launching an unprivileged container results in weirdness: /usr/bin/systemd-detect-virt fails to execve, returning ERANGE [11:12] Chipaca: heh, woah, ERANGE [11:12] mvo: getcap of the file _also_ fails with ERANGE [11:13] mvo: so we're about to learn something about _something_ [11:13] Chipaca: what a surprising error [11:13] Chipaca: yeah, its amazing [11:13] Chipaca: I've +1d assuming that's tuned per agreement.. someone else needs a final +1 too [11:13] pedronis ^ [11:14] man, i'm shaking [11:14] whoa [11:14] ok [11:14] niemeyer: thanks [11:14] Chipaca: mvo: yes, ERANGE usually makes me think of math libraries [11:14] I didn't expect the emotional response from myself ¯\_(ツ)_/¯ good thing i'm going on holidays next wednesday :-) [11:14] Chipaca: uhhhh, snapshot going in? === pstolowski is now known as pstolowski|lunch [11:15] mvo: got a +1 from niemeyer [11:15] pedronis: heh, exactly [11:15] ooooh, somebody's jealous :-p [11:15] Chipaca, dmesg -H means he needs to understand that he is in a pager ... i'd have suppressed the -H [11:15] ogra: maybe TERM isn't set or something stoopid [11:16] or that, yeah [11:16] mvo: sorry for being annoying about context, but is really mostly meant to have a place talk back to itself or connected places talk between unrelated or user layers [11:17] I mean it's Value feature [11:18] niemeyer: yes either me or mvo need to do a 2nd pass of snapshotstate [11:20] any consistent reason why all newish PR seems to be red ? [11:20] pedronis: lxd [11:20] pedronis: 16.04-32 lxd errors [11:21] the ERANGE issue ? [11:21] pedronis: yes [11:21] EDERANGED [11:21] fun :( [11:21] ERANGE is not even listed as a return value for execve [11:21] :-) [11:22] (but it's probably something to do with the xattrs) [11:22] if I had to guess, I'd guess that [11:22] because systemd-detect-virt is one of the very few files in 16.04 that uses caps (via xattrs) [11:22] in fact, i should look at the other ones, d'oh [11:22] * Chipaca does that [11:24] pedronis: dunno if you noticed but mvo wasn't online when you were apologising to him [11:24] no [11:24] pedronis: maybe you just wanted to get it off your chest :-D [11:25] https://github.com/snapcore/snapd/pull/5679 shall we pull the trigger? [11:25] maybe I didn't complete his nick [11:25] PR #5679: tests/main/lxd: run ubuntu-16.04 only on 64 bit variant [11:28] "The linux beginners course with ogra and Chipaca" [11:28] session 1 ... [11:28] :) [11:29] ogra: "If you survive with both your kidneys, [...]" [11:29] lol [11:30] ogra: as an aside, what on earth have they done to that poor "Ubuntu" that dmesg doesn't work [11:31] good question [11:32] lol [11:32] "no entries" [11:32] probably he run no kernel at all !!!! [11:33] ogra: it's secretly just running WSL [11:38] YESS [11:38] it's the capabilities [11:38] mtr fails in the exact same way [11:39] as does traceroute6.iputils [11:40] Chipaca: what's up? [11:41] Chipaca: should I merge 5679 or is the solution so close that its not worth adding the workaround? [11:41] mvo: NFI about the solution -- merge away [11:41] sergiusens: WHY DIDN'T I GIVE MYSELF MORE CONTEXT WITH THE PING :-( [11:41] sergiusens: now I have _no_ idea what it was about [11:41] it was, like, six context switches ago [11:42] PR snapd#5679 closed: tests/main/lxd: run ubuntu-16.04 only on 64 bit variant [11:42] sergiusens: I hope I'll remember and ping you again [11:42] oh! [11:42] sergiusens: i remembered :-D [11:42] sergiusens: were you aware of 'snap watch --last=auto-refresh?' [11:43] sergiusens: coming in 2.35 [11:45] Chipaca: yes we are https://github.com/snapcore/snapcraft/blob/master/snapcraft/internal/build_providers/_snap.py#L263 [11:45] but I think I might want to disable refreshes for a lot longer to not get killed mid build :-) [11:45] sergiusens: no no [11:45] sergiusens: the question mark at the end of the change type [11:46] sergiusens: means "no error if none found" [11:46] sergiusens: or from --help, “A question mark at the end of the type means to do nothing (instead of returning an error) if no change of the given type is found.” [11:46] Chipaca: oh, then I can get rid of the suppress. I must say though that that syntax is hard to spot given you phrased the sentence as a question :-) [11:47] sergiusens: mbuahaha [11:47] sergiusens: (sorry) [11:47] and I did an improper quote match [11:47] tut tut [11:47] * sergiusens needs to put his glasses on [11:47] sergiusens: anyway, 2.35+, so you probably can't use it yet [11:47] Chipaca: still good to know! [11:47] sergiusens: but, with our conversation abour --check-skeleton from the other day i thought i'd call it out to you [11:48] all is good :-) [11:51] ogra: wasn't there a file in proc to tweak the kernel log level? we could ask this person to try that [11:52] Chipaca, not sure if in /proc ... there is a sysctl setting you can apply though [11:52] mvo, this is an interesting one https://forum.snapcraft.io/t/set-system-proxy-from-custom-snap-service/6926 [11:52] ogra: sysctl is just writing to /proc/sys/ [11:52] ogra: :-) [11:52] ah, indeed [11:53] so yeah, there is one, i just dont know the node then ;) [11:53] ogra: but yes better to present it with sysctl [11:53] should be something about log_level [11:54] ogra: sys.kernel.printk ? [11:54] Chipaca, /proc/sys/kernel/printk [11:54] yeah [11:54] heh [11:55] $ cat /proc/sys/kernel/printk [11:55] 4 4 1 7 [12:04] ogra: I did not point them to https://i.imgur.com/Pfr9dj0.jpg ! I think I deserve a cookie. [12:05] * ogra hands Chipaca a well deserved cookie :D [12:06] i wonder if he paid for that carp ... [12:06] ogra: at lunch time? are you mad?!? [12:06] *crap [12:06] :-) [12:06] hahaha [12:06] ogra: 8GB of ram? you betcha it was paid for [12:06] although given they said it was Ubuntu and it wasn't, maybe it's "8GB" of "RAM" [12:07] (actually just a big swap file on a 1GB netbook) [12:07] haha [12:12] zyga: snap-update-ns is looking good, did a change, it's actually surprisingly simple [12:12] woot, that's great [12:14] zyga: you know, i might have screwed something up there too :P [12:14] I'll check next week :) [12:15] zyga: hey, fyi, responded to https://github.com/snapcore/snapd/pull/5644 [12:15] PR #5644: interfaces: add audio-playback/audio-record and make pulseaudio manually connect [12:15] jdstrand: thank you [12:15] jdstrand: I'm swapping today, doing office move and legal paperwork [12:15] zyga: https://paste.ubuntu.com/p/YZ7w7RKw5d/ mountinfo (does not include $SNAP_USER_DATA yet) [12:15] zyga: np. I suspect you'll just agree with me and ack [12:16] I'll look quickly now :) [12:16] zyga: but by all means, exercise your day off :) [12:16] PR snapd#5675 closed: overlord/snapstate: improve feature flag validation [12:16] Chipaca: fyi, I think that hostnamectl issue will be resolved if the PR merges from trunk [12:17] Chipaca: and hi :) [12:17] jdstrand: I was assuming as much :-) hi [12:18] jdstrand: excitement now is about lxd on 16.04-32 being unable to execve files that use capabilities [12:19] * Chipaca ~> lunch === pstolowski|lunch is now known as pstolowski [12:23] pedronis, Chipaca https://imgur.com/a/ZFNu5pV ... finally able to reproduce ... capturing logs now [12:23] jdstrand: +1 [12:24] zyga: thanks :) [12:24] marked as such on the PR [12:26] ogra: aha, you reproduced the shutdown hang? [12:28] damn, the trackpoint left click in my x220 is starting to fail :( [12:29] mvo, yeah [12:30] mvo, pedronis Chipaca https://pastebin.canonical.com/p/DGKBDMzQ2r/ logs (filtered out binder and anbox audit messages since they ake it unreadable) [12:30] *make [12:33] internal shutdown seems correct [12:33] so it would some handshake with systemd problem [12:34] seems we get a sigterm but don't do the right thing: snapd.service: State 'stop-sigterm' timed out. Killing. [12:36] * mvo wonders if anything has chnaged here [12:37] mvo: well it might have been like since a while [12:38] mvo: it's related to the waiting we do on reboot and signal unhandling [12:38] note this is all edge plus a devmode daemon (anbox) though i see the daemon being killed several lines before the snapd timeout shows up [12:38] (also not sure if a misbehaving snap could actually make snapd not stop) [12:38] mvo: we also added the watchdog [12:39] pedronis: aha, thats a good one [12:39] pedronis: Aug 17 12:19:55 localhost.localdomain snapd[1005]: daemon.go:577: Waiting for system reboot ? [12:39] mborzecki: ? [12:39] isn't snapd waiting in a long sleep here? [12:39] yes [12:39] as I said signal handling is not quite right over shutdown [12:39] so any signals are not really handled [12:39] yes [12:39] what I'm trying to say [12:39] not sure it's related to the timeout though [12:39] if it's really much later [12:41] mvo: we need to call Reset or Stop for the signal handling we setup in main.go, not sure exactly where [12:42] if you want to repro: create a qemu VM with an image from edge ... leave it off over night so there is a new core ... make sure to start it only after core has updated in the store... boot it and watch it to do an auto-refresh with that hang [12:42] i have never seen it when doing a manual refresh [12:44] pedronis: i'd assume it's related, then things start to make sense, term get queued, if systemd gets a request to restart the process it would make sense to ignore it since the system is going down anyway, systemd timeouts waiting for snapd to exit, snapd would timout waiting for reset :) [12:45] well, not from the logs it seems systemd then kills snapd [12:46] so snapd doesn't timeout [12:46] yeah, I'm puzzled [12:46] ogra timeout is systemd saying something about snapd again [12:46] but maybe I'm confused [12:46] if snapd does not handle sigterm correct I should be able to simulate this by simply kill -TERM $(pidof snapd) [12:46] and that exist normally [12:46] mvo: this is about reboot mode [12:46] not normal running snapd [12:47]