=== KindTwo is now known as KindOne [05:17] morning [05:30] PR snapd#8763 closed: gadget: make ext4 filesystems with or without metadata checksum === franke is now known as Guest56840 [06:03] quick errand [06:25] PR snapd#8758 closed: tests: run ubuntu-20.04-* tests on all ubuntu-2* releases [06:30] zyga: I see "2020-05-27T16:45:02.9982589Z invariant-tool: crashed-snap-confine not-ok" in some tests, most recently in https://github.com/snapcore/snapd/pull/8744/checks?check_run_id=713534659 [06:30] PR #8744: interfaces-ssh-keys: Support reading /etc/ssh/ssh_config.d/ [06:30] PR snapd#8744 closed: interfaces-ssh-keys: Support reading /etc/ssh/ssh_config.d/ [06:41] mborzecki: mvo: hi, #8763 needs to be ported to master? (I didn't notice it was target at 2.45 only) [06:41] PR #8763: gadget: make ext4 filesystems with or without metadata checksum [06:43] pedronis: I think there were two, let me quickly check (one for master, one for 2.45) [06:44] pedronis: oh, you are right, let's fix this :( [06:47] mvo: does the test in #8762 works? [06:47] PR #8762: snap-bootstrap: remove sealed key file on reinstall [06:48] pedronis: yes, I'm reworking it now though to put the files into the right places, i.e. simulate more closely what we really do (/run/mnt/ubuntu-seed/keyfile, /run/mnt/ubuntu-data/system-data/var/lib/snapd/fde/... for the rest) [06:48] ah, good [06:50] PR snapd#8765 opened: gadget: make ext4 filesystems with or without metadata checksum (master) [06:53] mvo: btw the comment in the description was wrong, given that the note mentions reboot, I don't think it's just a creation-time issue [06:54] Good morning [06:55] re [06:55] mvo: pedronis: zyga: hey [06:55] mvo: and sorry for the many rounds of comments on #8762, I wasn't concetrating properly last night [06:55] PR #8762: snap-bootstrap: remove sealed key file on reinstall [07:01] pedronis: thank you for the comments, the opposite - my stuff was not coherent enough :( but I will push an updated version soon that is hopefully good [07:02] morning [07:04] good morning pstolowski [07:17] mvo: I need to look, I think it's not really that but leftover from package upgrade, trying to reproduce now [07:18] offtopic, I was thinking, that 8GB pi may be just able to run run spread with qemu backend [07:33] zyga: finally a arm spread runner? [07:33] mvo: we'll see, but I think given the low cost and 8GB of ram, might be possible [07:37] zyga: do ou know what's the status of kvm and virtio-* on aarch64? [07:37] i would guess it's a major factor for making the tests run smoothly [07:38] i'd appreciate a second review for #8710, test only [07:38] PR #8710: tests: spread test for preseeding in lxd container (3/3)

[07:39] pstolowski: i can take a look [07:39] mborzecki: thanks! [07:45] mborzecki: not really, I think 32 bit kvm was removed but 64 bit should be ok [07:52] mvo: i was talking to Sergio yesterday about uc20 model assertions and the TODO you have in gadget-config-defaults-vitality spread test; apparently it's not possible yet to port this to uc20? i'd like a similiar test for nested uc20 [07:55] pstolowski: I need to look again, but iirc the simulation of firstboot is slightly diffreent on uc20 [08:01] we need to think [08:02] pstolowski: can you explain what you mean with similar but for nested? [08:02] those tests are not nested [08:03] pedronis: i'm repacking the gadget, but not cheating about firstboot (booting for real) [08:03] *i mean [08:05] pstolowski: I see, you need a dangerous model, you need to your unasserted snaps into the seed partition in /systems/SYSTEM/snaps and you need a /systesm/SEED/options.yaml which is a bit like seed.yaml but only for overrides [08:10] pstolowski: about options.yaml see seed/internal/options20.go [08:12] mvo: can you merge https://github.com/snapcore/snapd/pull/8271 ? the failure on 19.10 is unratelated to the change [08:12] PR #8271: interfaces: add hugepages-control [08:12] mborzecki: sure [08:13] pedronis: thanks for pointers, that helps! [08:13] mborzecki: does it have two +1 ? [08:13] mborzecki: nevermind, I have a look [08:14] mvo: ah no ;) jdstrand_ gave +1, but i can't cause i pushed some changes there [08:16] PR snapd#8271 closed: interfaces: add hugepages-control [08:20] pedronis: do you have a moment to talk about sysconfig changes? [08:20] pstolowski: yes [08:23] pedronis: ok, coming to standup ho [08:28] mup is asleep? [08:28] fairly simple fix in daemon: https://github.com/snapcore/snapd/pull/8766 [08:28] PR #8766: daemon: fix filtering of service-control changes for snap.app [08:31] PR snapd#8766 opened: daemon: fix filtering of service-control changes for snap.app [08:36] mborzecki, hey, thanks for taking care of the hugepages PR - I should really have done those changes, but got trapped with other things [08:37] mvo: I spent some time looking at the log you referenced and it does seem like something is randomly failing in that test [08:41] abeato: hey, np, we're here to help land things too :) [08:54] zyga: thanks, yeah, it looks like 19.10 and 20.04 are a bit more unhappy today than earlier this week :/ [08:55] I'll try to do something about that :) [08:55] call in 5, let me prepare [08:56] PR snapd#8767 opened: snap: refactor download code for testing/extending [08:59] ah [08:59] * zyga is dummy [08:59] that's not today :0 [08:59] ok :) [09:00] back to original task [09:22] journal logs don't show failures from snapd-session-agent [09:22] but [09:22] mvo: this is the failure [09:22] May 28 20:06:30 may281929-399859 snapd[27253]: May 28 19:59:05 may281929-399859 systemd[1273]: snapd.session-agent.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted. [09:23] I suspect those are branches without https://github.com/snapcore/snapd/pull/8753 but let me verify [09:23] PR #8753: tests: reload systemd --user for root, if present [09:24] hmm, no, that patch is present [09:24] curious [09:24] let's dig deeper [09:25] hmm [09:25] actually [09:25] just reading the error message [09:25] it seems to suggest the problem is not daemon-reload [09:25] but the fact that the socket needs restarting as wel [09:25] *well [09:26] that might be it [09:26] we'll know soon [09:26] zyga: thanks! [09:31] PR snapd#8749 closed: configcore: show better error when disabling services [09:31] PR snapd#8757 closed: tests: update statx test to run on all LTS releases [09:41] mvo: oops, there was confusion, my proposal was to replace the second line of the comment, not all of it [09:41] in #8782 [09:41] pedronis: oh, sorry. let me fix that [09:41] heh [09:41] #8762 [09:41] PR #8762: snap-bootstrap: remove sealed key file on reinstall [09:44] pedronis: updated again [09:45] thanks [09:50] pedronis: nitpick, can we call $TESTTOOLS $TESTTOOLS instead? [09:50] er [09:50] TESTSTOOLS -> TESTTOOLS [09:50] the env vra [09:50] *var [09:53] we have TESTSLIB not TESTLIB [09:53] the var mimicks path [09:53] but the name sounds bad if you think about it [09:53] both plural [09:54] I'm worried about TESTSLIB TESTTOOLS [09:54] that they differ? [09:54] test libs would be better for english var [09:54] the path can stay as is [09:54] TESTLIBS [09:54] TESTTOOLS [09:55] plural in programming are always trouble [09:55] as I said I'm more worried about consistency than english here [09:55] we can do both ;) [09:55] I mean we can rename both while we are at it [09:56] $TESTTOOLS/foo reads better than $TESTSTOOLS/foo [09:56] I don't know, not a fan of TESTLIBS [10:01] zyga: I fear TESTSLIB and TESTSTOOLS, anyway my nature we shouldn't need to use the latter that much [10:01] s/my nature/by nature/ [10:01] not sure what you mean by feal [10:01] fear * [10:01] I mean, that we go with those [10:02] mvo: I re-reviewed #8762 [10:02] PR #8762: snap-bootstrap: remove sealed key file on reinstall

[10:04] anyway on unrelated note the user service stuff seems still fragile, I think we could tweak the tests more, but probably the fragility is real and the code need some changes [10:05] pedronis: thanks! so your suggestion is to run the same tests as for the install case in the reinstall case? [10:05] mvo: either that because it seems cheap (at least code wise), or to at least run a small subset related to the encrypted partition [10:06] pedronis: yeah, it makes sense [10:06] pedronis: do you prefer duplication of the code or should I move it into a helper like with "prepare.sh" ? [10:07] mvo: sorry, you don't need to duplicate code [10:08] you can just reduce the scope of the if, no? [10:08] pedronis: it's a test bug, I fixed it just now [10:08] pedronis: (user service stuff) [10:08] pedronis: aha, nice [10:08] pedronis: yeah, I think this will work [10:08] zyga: I doubt is just a test bug [10:08] apart from the known TODO, I think it is [10:09] the TODO being that removing snapd doesn't properly clean up user session services [10:10] you'll see in a moment [10:10] zyga: there's a deeper issue, the existence of the snapd.session-agent.socket used by the code, doesn't really tell much about the state of the session [10:11] what kind of state are you referring to? [10:13] we are going to ask to do it things with services in the session but the session, might be just starting up [10:14] we don't know if the session overall in in a steady state [10:15] I think there's no such thing as a stady state, there are concurrently running processes that may stop or start as they please; I think it depends on what kind of ops we'd like the agent to perform [10:15] I understand that [10:16] but the existence of the socket is not here nor there [10:16] mvo: can we land https://github.com/snapcore/snapd/pull/8520 ? the failure on 20.04 is the user service again [10:16] in terms of a barrier related to the state of other services [10:16] PR #8520: data: fix shellcheck warnings in snapd.sh.in [10:17] pedronis: the problem we are seeing in tests right now is related to the particular handling of socket units by systemd; I think that's a separate issue from what you are describing [10:17] our tests continously remove and reinstall snapd, making the socket definition in memory wrong, that was fixed (for tests) recently; what we are seeing now is that when a socket is removed from disk you must perform additional actions, more than just daemon-reload, for it to go away [10:18] and oddly enough installing a new definition of the socket is not sufficient [10:21] PR snapd#7869 closed: travis-ci: split integration tests into parts (>4MB log limit) [10:23] I also wonder if this is not the same problem we had with pulseaudio, without realizing it [10:23] it's just there are so few tests that touch it it may not be triggered often [10:39] zyga: btw, related question, does the refresh-awareness tracking for services also works for user services? [10:39] yes [10:39] fortunately :) [10:40] anyone tried the liferea snap? the font cache from the host is not mounted inside the snap, but i still get boxes [10:42] all i get is https://i.imgur.com/rNFMezE.png [10:42] hmmm [10:42] mborzecki: nsenter and poke around [10:42] wonder if there's something more going on [10:42] at some point fontconfig will figure out how to IPC the boxes around [10:47] hm i mounted a tmpfs over /usr/share/fonts in the mount ns of a snap and there's no more boxes [10:47] what's in /usr/share/fonts? [10:47] maybe a read only cache? [10:47] mborzecki, pedronis: please check this out https://github.com/snapcore/snapd/pull/8768 [10:47] PR #8768: tests: fix broken snapd.session agent.socket [10:48] what if the fontconfig from gnome runtime is jsut buggy? [10:48] is there any configuration in /usr/share/fonts that is read by fontconfig? [10:49] zyga: there shouldn't be, i have a tmpfs on /etc/fonts and /usr/share/fonts [10:49] I mean [10:49] before you do the tmpfs [10:50] before you hide stuff there [10:50] what do you see if you find /usr/share/fonts [10:50] mborzecki: alternatively strace and explore [10:50] just fonts in directories [10:51] or ltrace [10:51] so are you saying those fonts cannot be rendered? [10:51] zyga: yeah, either boxes or segfaults again [10:51] hmmm [10:51] what kind of fonts do you have there [10:51] PR snapd#8768 opened: tests: fix broken snapd.session agent.socket [10:51] maybe bisect [10:52] and see which file triggers it [10:52] I think there must be something more than that [10:52] nothign special, looks like i need 18.04 desktop vm [10:52] maybe some font has a special hint bytecode that is disabled in arch [10:52] and then without a build option that crashes [10:53] zyga: but fontconfig is from the snap, if it can't handle arbitrary fonts the all hope is lost ;P [10:53] ahhh [10:53] interesting [10:53] yeah [10:53] but I think bisecting is the way to go [10:53] how many files do you have there? [10:54] zyga: is tests cleanup meant to be on the PATH? [10:54] mborzecki: or maybe another idea: are there any duplicates? [10:54] zyga: sorry, tests.cleanup [10:54] pedronis: I don't think so, it's only going to be used in prepare-restore in two places [10:54] zyga: then it shouldn't have the tests. prefix [10:54] it's not something I expect to see in any task.yaml [10:54] oh? I misread the readme then [10:54] yeah, I see the fragment now [10:55] "commands that expect to be on PATH should follow more speicfic naming conventions" [10:55] I'll correct that, it's just a draft :) [10:55] so just $TESTSTOOLS/clenaup? [10:55] *cleanup [10:55] or state-cleanup [10:55] ok [10:55] we should think about session services more [10:56] I think without handling the remove story we should be very careful [10:58] mborzecki: what do you think about the agent socket [10:58] I think this is actually a property of any unit [10:59] it's just sockets seem worse as "hey, it's there, let me talk to it" [10:59] and for non-session stuff we do stop services and sockets so it's not something we've seen before [10:59] pedronis: I reworked 8762, test is much nicer now and shares a ton. silly me for not thinking about this earlier :) [11:05] zyga: hmm making /usr/share/fonts/adobe-source-code-pro available inside the snap causes boxes and segfaults :P [11:06] awesome [11:06] now put that font on a 18.04 vm [11:06] what's the base of the snap you tried? [11:06] maybe strace/gdb the app [11:06] and get fontconfig-dbg [11:06] zyga: funny, i don't even use any of that fonts [11:06] PR snapd#8765 closed: gadget: make ext4 filesystems with or without metadata checksum (master) [11:07] zyga: the backtrace is the same as before https://forum.snapcraft.io/t/gimp-and-glimpse-editor-snaps-segfault-on-arch/17100 [11:08] where's the backtrace? [11:08] that's just some log [11:12] time to take Bit out for a walk [11:12] back in a bit [11:12] zyga: SourceCodeVariable-Roman.otf this font is causing a problem [11:12] wodner what's so special about it [11:12] mborzecki: developers and their fancy pantsy fonts ;) [11:12] mborzecki: yeah, it's curious [11:12] if you can reporduce the crash outside of snaps [11:12] we should report those [11:12] zyga: but i'm not even using this font [11:13] and even debug and fix them, if possible [11:13] mborzecki: so what, it's there [11:13] it gets scanned [11:13] glimpse-editor and gimp segfault, probably because the load up all info about fonts [11:13] yeah [11:13] try inkscape [11:13] apps that aren't so interested get boxes? [11:13] from the edge channel [11:13] mborzecki: maybe boxes are for another reason [11:13] the cache is corrupted [11:13] it has this new fancy font handling hack [11:13] ogra: oh, right [11:13] try generating the cache [11:13] and see if that crashes [11:14] try inkscape first to see if the hack works ! [11:14] mvo: please check this out [11:14] https://github.com/snapcore/snapd/pull/8768 [11:14] PR #8768: tests: fix broken snapd.session agent.socket [11:14] now really afk [11:14] yeah, inkscape edge works [11:14] awesome [11:15] it throws some font.conf errors on startup though ... (unknown options) ... but i see it working in zoom as well [11:25] Bug #1881276 opened: shutdown fails to unmount some loops because of udevd, mark them LazyUnmount [11:26] pedronis: i've updated #8567, let me know if this is what you had in mind. image_linux became slightly ugly [11:26] PR #8567: o/devicestate: core20 early config from gadget defaults [11:29] mvo: can you take a look at https://github.com/snapcore/snapd/pull/8766 ? [11:29] PR #8766: daemon: fix filtering of service-control changes for snap.app [11:31] Bug #1881276 changed: shutdown fails to unmount some loops because of udevd, mark them LazyUnmount [11:34] morning folks [11:37] Bug #1881276 opened: shutdown fails to unmount some loops because of udevd, mark them LazyUnmount [11:43] Bug #1881276 changed: shutdown fails to unmount some loops because of udevd, mark them LazyUnmount [11:46] Bug #1881276 opened: shutdown fails to unmount some loops because of udevd, mark them LazyUnmount [11:49] zyga: so the same font works in ubuntu, even in the same snap on ubuntu [11:49] i've mounted tmpfs over the fonts cache and /etc/fonts [11:49] and it still works, wtf?? [11:58] Hmmmmm [11:58] Drop font cache [11:58] Does it crash? [12:04] no [12:04] it's dropped already [12:04] no /var/cache/fontconfig, no /etc/fonts [12:05] and ~/.local/fonts ... etc ? [12:06] (iirc the desktop launchers pick these up too) [12:06] pstolowski: I reviewed #8567 [12:06] PR #8567: o/devicestate: core20 early config from gadget defaults [12:07] pedronis: thanks [12:07] thank you [12:10] heh, wiped ~/.cache/fontconfig/ and it's working now [12:10] damn, i don't see how we can make it work if there's even a slightest bit of fontconfig cache being shared [12:12] mborzecki: re 8766, if you do `snap restart lxd.daemon` now we don't pass in inst.Names though, we always pass in namesToSnapNames(...) so how could the names on the task get set to the inst.Names [12:12] PR snapd#8639 closed: o/cmdstate: handle ignore flag on exec-command tasks (1/N) [12:14] ijohnson: i clarified in a comment there, what meant by `names` in the comment is inst.Names [12:14] ijohnson: anyways, i rephrased the comment, please check if it makes sense now [12:14] mborzecki: ahhhhh sorry I'm a bit slow this morning it seems [12:14] * ijohnson looks now [12:14] thank you makes sense now [12:15] cool [12:17] mborzecki: so which part of our code makes your ~/.cache/fontconfig available to the snap? [12:18]