[00:58] PR snapd#7449 closed: overlord/configstate: special-case "null" in transaction Changes() [02:36] PR snapd#7454 opened: interfaces: extend the fwupd slot to be implicit on classic [03:07] PR snapd#7455 opened: tests: moving ubuntu-core-transition tests to nightly test suite [04:57] morning [05:37] gorename has gone crazy since 1.13 update [06:03] Morning school run [06:31] zyga: hey [06:34] hey [06:34] in the office now [06:34] mborzecki: can you give me +1 on https://github.com/snapcore/snapd/pull/7446 [06:34] PR #7446: tests/mountinfo-tool: proper formatting of opt_fields [06:34] this will unblock the whole avalanche of related PRs [06:35] zyga: started looking at it yday, but didn't finish bc of kids :/ [06:35] mborzecki: it's super easy, just fixes formatting of the optional fields list to contain "-" [06:36] thanks, I'll look at other stuff as well [06:36] while at it i just pushed some updates to #7451 [06:36] PR #7451: sandbox/cgroup: introduce cgroup wrappers package [06:37] let's start with that [06:38] good morning mvo [06:38] mvo: hey [06:39] zyga: tbh wondering what the distros still shipping py2 will do next year [06:39] mborzecki: macos will remove python from the OS [06:39] it's not used by the os itself, unlike on linux [06:39] zyga: and perl i think? [06:39] I think others will port / remove [06:39] mborzecki: likewise [06:39] ruby, perl and python are removed [06:39] as is bash [06:40] you can install them from the upstream distributions but the OS won't have them [06:40] zyga: well, they were outdated anyway, so nobody used them? [06:40] mborzecki: I think they were used [06:40] even though outdated [06:40] hey zyga and mborzecki ! [06:40] but nowadays it's docker / vms instead [06:41] PR snapd#7446 closed: tests/mountinfo-tool: proper formatting of opt_fields [06:41] tests more green today? [06:41] mvo: not sure [06:42] mvo: last night I restarted a single test like every 30 minutes [06:42] all day [06:42] so if something is green it's because blood, tears and mud [06:42] it'd be nice if we could move google:ubuntu-18.04-64:tests/main/desktop-portal-filechooser to nitghtly too :/ [06:42] mvo: store had some issues frequently, there's lots of chatter in snapstore [06:42] mborzecki: I think we need a different suite [06:43] nightly is not a dumpster for stuff that is broken [06:43] we need a fragile suite [06:43] nightly should never fail [06:43] zyga: deskop-dumpster suite [06:43] and if it fails, it's all hands on deck to fix [06:43] otherwise nightly will degrade to that thing that is always red so we don't care [06:43] yeah [06:43] zyga: you're saying nightly is where the tests go to die? :) [06:43] and then a real issue explodes on us and in a post-mortem someone will say, "this test failed in nightly for 3 weeks" [06:45] 72 prs [06:45] mvo: I commented on a related issue here https://github.com/snapcore/snapd/pull/7453 [06:46] mvo: I think we should be very careful with changes like that [06:46] PR #7453: tests/classic-custom-device-reg: disable on opensuse-15 and fedora-30 [06:48] a simple review https://github.com/snapcore/snapd/pull/7448 [06:48] PR #7448: tests: remove mount_id and parent_id from mount-ns test data [06:48] +2,505, -2,504 [06:48] it's a one line diff really :) [06:49] zyga: yeah, we should blindly disable tests [06:49] zyga: hm, master is red for the last ~5 runs or so - thats not good [06:49] mvo: let's look at why [06:49] * zyga checks [06:50] I see LXD failure [06:50] snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks [06:50] grep error: pattern not found, got: [06:50] this is something I reported a few days ago [06:50] that it really happens and indicates we're racing somewhere [06:50] or perhaps that one of the mirrors has a corrupt LXD image? [06:51] who knows but it's real [06:51] zyga: oh, I saw this too in a branch of mine [06:51] I'll save the log and look at past tests [06:51] zyga: this was the lxd PR to move to the faster mirrors [06:51] perhaps there's some clue there [06:51] snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks still happening oocassionaly inside lxd? [06:51] yes [06:52] error: cannot fetch snap signatures/assertions: cannot add assertion account-key (BWDEoaqyr25nF5SNCvEv2v7QnM9QsfCc0PBMYD_i2NGSQ32EF2d4D0hqUel3m8ul): failed signature verification: openpgp: invalid signature: hash tag doesn't match [06:52] this is another, cannot prepare suite [06:52] ubuntu-core assertion fetch failed because the signature was invalid [06:53] another error [06:54] error: Get https://api.snapcraft.io/api/v1/snaps/download/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Let's Encrypt Authority X3") [06:54] is the store really using a let's encrypt certificate?! [06:54] that last error happened at least three times [06:55] sanity check test failed, expected log message was not found in the journal [06:55] mvo: I was thinking about that [06:55] I think our journal handling is still bad [06:56] and I was thinking that for the sake of robustness we could log to a file as well [06:56] perhaps only under testing debug mode [06:56] it would also help us on 14.04 where journal has weaker APIs and just doesn't do what we need many times [06:57] - Consider re-refresh of "test-snapd-classic-confinement" (cannot query the store for updates: got unexpected HTTP status code 503 via POST to "https://api.snapcraft.io/v2/snaps/refresh") [06:57] but this is when store was down [06:57] error: cannot install "core16": cannot get device session from store: store [06:57] server returned status 503 and body "

503 Service [06:57] Unavailable

\nNo server is available to handle this [06:57] request.\n\n" [06:57] we apparently print the body of responses sometimes too? [07:03] woah [07:04] zyga: also "error: cannot fetch snap signatures/assertions: cannot add assertion account-key (BWDEoaqyr25nF5SNCvEv2v7QnM9QsfCc0PBMYD_i2NGSQ32EF2d4D0hqUel3m8ul): failed signature verification: openpgp: invalid signature: hash tag doesn't match" === pstolowski|afk is now known as pstolowski [07:05] morning [07:06] hey pstolowski [07:06] zyga: I mean, *sigh* [07:06] zyga: fwiw, the lxd error I also got in 7286 [07:07] zyga: and from looking at the logs I was wondering if its real so that the apparomor profile is not loaded by the postinst under certain circumstance [07:07] Good morning Paweł [07:07] zyga: *but* it all does not make sense and is hard to debug so I got a bit despaired [07:07] pstolowski: last night Catalina update fixed VMware [07:08] zyga: but yeah, it looks like the store was mightly unhappy last night [07:08] mvo: in cases like this I would look at all the errors to see if something stands out more than others but then proceed to focus on a single case [07:09] It does look grim when things are failing but we are not alone :) [07:09] as I said it would be good to have stasts on error frequency but it's work [07:09] pedronis: you mean we should have stats on our side? [07:10] pedronis: and good morning :) [07:10] which tests/hpw fail the most [07:10] *how [07:10] * mvo nods [07:19] zyga: uh - I see "Sep 11 20:27:08 sep112008-564525 snapd[16506]: Sep 11 20:27:05 sep112008-564525 systemd[1]: [/etc/systemd/system/snap-test\x2dsnapd\x2drsync-12.mount:10] Unknown lvalue 'LazyUnmount' in section 'Mount'" in the logs. a agzillion of those [07:20] zyga: looks like this was added in 232 but 16.04 has 229 [07:21] ah [07:21] fun [07:23] mvo: should be harmless though [07:23] Oh man [07:23] How come nothing failed on that? [07:24] we don't test for the behavior no? [07:24] it's just spamming the logs [07:24] I think [07:24] not failing to use the unit at all [07:24] Should we revert it or extend it so that we perform the unmount ourselves, without involving systemd? [07:26] looking into desktop-portal-filechooser, for some reasons this keeps failing quite often on 18.04 but not on later releases [07:28] mvo: I have 3 more seedwriter PRs ready (+1 very close), but they probabbly would not be helpful right now, I also the accumulated size would be scary [07:29] a simple pr - https://github.com/snapcore/snapd/pull/7390 only adds a new test and is green but needs reviews [07:29] pedronis: yeah, let me start looking at those [07:29] PR #7390: tests: unit test for a refresh failing on configure hook [07:29] pedronis: I'm almost ready scheduling for paris [07:52] mborzecki: the cgroup spread failure on i386 looks real (fwiw) [07:56] mvo: do you have log? [07:57] pstolowski: hi, #7196 can be simply closed now, no? we have #7092 waiting on snapcraft [07:57] PR #7196: packaging: use passthrough for type:snapd <⛔ Blocked> [07:57] PR #7092: packaging: use snapd type and snapcraft 3.x <⛔ Blocked> [07:57] zyga: fwiw. looking at some things on amzn2, systemd complains about LazyUnmount, StartLimitInterval and FileDescriptorName, so it's a reasonable fallback behavior by systemd [07:58] pedronis: yes, done [07:58] PR snapd#7196 closed: packaging: use passthrough for type:snapd <⛔ Blocked> [07:59] mborzecki: zyga: do we still need #7198 ? [07:59] PR #7198: tests: reboot the node when restoring after a test involving lxd <⛔ Blocked> [08:00] pedronis: nope, we can close it [08:00] let me do it [08:00] moin moin [08:01] Chipaca: hi [08:01] PR snapd#7198 closed: tests: reboot the node when restoring after a test involving lxd <⛔ Blocked> [08:02] Chipaca: hey [08:03] obviously desktop-portal-fielchooser does not fail when you try to debug what's happening :/ [08:03] mborzecki: than you [08:05] sergio has a PR about it, not sure what it does [08:05] mborzecki: what about #7193 ? [08:05] PR #7193: [WIP] cgroupsv2 spread run [08:06] PR snapd#7452 closed: tests: move classic-ubuntu-core-transition* to nightly [08:06] PR snapd#7455 closed: tests: moving ubuntu-core-transition tests to nightly test suite [08:08] pedronis: i'd liek to keep it around to track the failing tests (also needs an update) [08:11] mborzecki: ok, that's fine, just wondered [08:16] Chipaca: are you aware of #7447 ? [08:16] PR #7447: snapstate: do not allow removal of the snapd snap [08:16] the jumbo jet pr [08:17] yeah let's not do it this way :-) [08:17] I'll push a replacement for it and close this one [08:18] … as soon as i can get the current one green? or should i push it stacked? or ...? [08:21] pedronis: question about canRemove: both baseInUse and coreInUse call snapstate.All, but handle failure differently: one returns false, the other returns err != ErrNoState. Which is the correct approach? [08:22] Chipaca: they probably both wrong [08:22] pedronis: … go on [08:22] let me look [08:22] :) ok [08:22] but eating errors is never a good idea [08:23] canremove would have to start bubbling the error out, in that case [08:25] Chipaca: is used in one place no? [08:25] pedronis: right now yes [08:25] anyway let me look [08:25] pedronis: i don't know why disable doesn't also check this though [08:25] Chipaca: disable is a use at your own peril op [08:25] disable, the oft-forgotten ugly sibling [08:25] also because it's our turn it off/on again remedy tbh [08:26] or has been at least [08:30] pstolowski: something for you [08:30] https://www.irccloud.com/pastebin/yrr42wg7/ [08:31] zyga: uuuh [08:31] zyga: is it on your box? state.json? [08:32] pstolowski: https://bugs.launchpad.net/snapd/+bug/1843683 [08:32] nope, just spread on master [08:32] Bug #1843683: cannot unmarshal state entry for "timings" [08:32] since the 4.39e+08 is there, you can just write a test to see what happens [08:32] zyga: ok, thanks, will investigate [08:33] thank you [08:33] zyga: is this now blocking master / happening all the time? [08:33] no [08:34] but feels dangerous since it may break state handling [08:34] 6404 and 6839 are green and need a second review I think [08:34] zyga: only on `snap debug timings ..` [08:35] ah, that's good [08:35] though I thought that was always on and collected? [08:35] but perhaps I don't know well [08:36] zyga: yes always collected but we dont' unmarshall them, only append raw messages [08:37] regardless.. i'll investiagte today [08:41] pstolowski: thank you! [09:01] hi, is this the right place for multipass questions? [09:03] PR snapd#7456 opened: usersession/client: add a client library for the user session agent [09:11] ackk: ask away, Saviq is here [09:13] https://github.com/snapcore/snapd/pull/7448 is green and would unblock me [09:13] PR #7448: tests: remove mount_id and parent_id from mount-ns test data [09:15] Saviq, hi, I'm getting failures when trying to build a snap in multipass at the "apt update" phase, it seems it's trying to use ipv6 but failing contact the archive over ipv6 [09:15] ackk: what version of Multipass? [09:15] Saviq, I can ping that address from the machine where multipass runs. is there something to configure in multipass itself? [09:15] `multipass version` [09:15] Saviq, installed: 0.9.0-dev.199+g69dad0f (1135) 169MB classic [09:16] and: [09:16] multipass 0.9.0-dev.199+g69dad0f [09:16] multipassd 0.9.0-dev.199+g69dad0f [09:16] ackk: go back to beta, we have a networking issue on edge [09:16] oh, I'm on edge [09:16] yay for channels [09:16] Saviq, should I use stable or beta? [09:16] :) [09:16] I think this is so fantastic [09:16] zyga, +1 [09:16] ackk: stable not there [09:16] Coming soon [09:16] oh right :) [09:16] sorry, haven't tried in in a while [09:17] zyga: even better, bisecting snap revisions ;) [09:17] * ackk tries beta [09:18] i need to pop round to the bank, will bbiab [09:18] off for a bit [09:36] sarnold, beta worked, thanks! [09:36] err, Saviq ^ [09:36] :) [09:37] I'm sure sarnold will be happy, too !) [09:47] mvo: I gave my +1 to 6404 and 6839 , I left a question in the latter though. They still need 2nd reviews [09:52] pedronis: thank you! I will check if there is a missing test in 6839 [09:55] PR snapd#7448 closed: tests: remove mount_id and parent_id from mount-ns test data [10:03] re [10:03] https://github.com/snapcore/snapd/pull/7442/files shrank from 5K diff to 500 diff [10:03] :) [10:03] PR #7442: tests: extend mount-ns test to handle mimics [10:17] pstolowski: I did another pass on #7277 [10:17] PR #7277: overlord/snapstate: fix undo on firstboot seeding [10:18] pedronis: ty [10:57] brb [11:06] #7390 needs 2nd review, it's just a new test and green :} [11:06] PR #7390: tests: unit test for a refresh failing on configure hook [11:08] nooo, koza is leaving [11:12] was jonathan riddel on irc? anybody know their nick? [11:13] Chipaca: the kde guy? jriddel afair, or maybe just riddel [11:13] hm [11:15] Chipaca: maybe you could take a look at #7390? [11:15] PR #7390: tests: unit test for a refresh failing on configure hook [11:16] re [11:16] sorry, some woes with daugther [11:16] 10-year-olds and all that stuff [11:17] pstolowski: in a bit yes [11:17] zyga: pffff desktop portals https://paste.ubuntu.com/p/VttStX82xQ/ [11:18] desktop-mortal [11:18] jamesh: hello [11:18] zyga: hi [11:18] are you still around by any chance? I know it's probably super late for you [11:18] hey! [11:18] mborzecki was looking at some of the failures in the xdg-desktop-portal test, specifically on 18.04 [11:19] and concluded that it is likely that this specific version 0.11 of the portal is just buggy [11:19] how likely is to backport the portal stack to 18.04 [11:19] zyga: jamesh: actually 1.0.3 was pushed to 18.04, but the issue persists [11:19] and if unlikely, should we formally support it at the current version? [11:19] mborzecki: 1.0.3 is in bionic-updates [11:20] oh. [11:20] We'll definitely want to upgrade for the snap support improvement work I haven't started [11:23] * pstolowski bbiab [11:24] Chipaca: Riddell, two ls [11:24] ah [11:24] (that was their nick as well as their surname. Dunno if they're still on IRC) [11:24] * Chipaca bad with names [11:30] mborzecki, zyga: the only patch to the document portal FUSE code missing from the Bionic packages is this one: https://github.com/flatpak/xdg-desktop-portal/commit/c9159903e151d5aba3067dcaeb977ceb9a1a9e8b#diff-113fd392ffd1391fc2e8547af3dd1789 [11:31] jamesh: so 2 problems we have now, 1. when the test code in desktop-portal-filechooser closes the fd it writes to, the content cannot be immediately read from the actual file, but after waiting for a bit it does appear there (some race with closing/flushing data in xdp in the portal)? [11:32] jamesh: then 2nd. occasionally we get OSError with -ENOSYS when the file did not exist pefore it was picked for saving-to (the same test) [11:32] mborzecki: I wonder if the link count could affect that first one? [11:33] jamesh: i've started a portal separately with G_MESSAGES_DEBUG=all, replaced the one running in test user's session, but cannot reproduce the problem this way, so maybe a race again [11:33] perhaps all versions are buggy then? [11:33] mborzecki: does that test run on anything more recent than 18.04? [11:34] zyga: yes, 19.04 and 19.10, haven't really seen issues on 19.10 [11:34] mmm [11:34] jamesh: maybe that's a data point then [11:34] so it's a real thing then [11:35] mborzecki: It's quite possible that there is a problem outside of the xdg-desktop-portal code too [11:35] mborzecki: this is the other change to document-portal-fuse.c between 1.0.3 and master: https://github.com/flatpak/xdg-desktop-portal/pull/310 [11:35] PR flatpak/xdg-desktop-portal#310: document-portal: make xdp_inode_kernel_unref check that kernel_ref_count > 0 [11:36] which we included in the Bionic package [11:37] I don't think we ever worked out the root cause for that [11:40] it is possible that it's a bug in the bionic kernel [11:41] jamesh: another data point, i can't really reproduce any of the problems by stating the portal manually, but happens regularly when it's started by user systemd [11:53] jamesh: https://paste.ubuntu.com/p/P7Jj5BVsgz/ captured when rename happens after fuse_release, so the content is not really available yet [12:00] mborzecki: the kernel unref PR I mentioned earlier is something that shouldn't be necessary: it essentially means that the fuse filesystem and kernel disagree about the kernel-side reference count of an inode [12:00] mborzecki: it's something I haven't been able to reproduce on any other distro release [12:01] if there is some bug in the kernel-side fuse code for that release, it could very well lead to this kind of problem [12:09] PR snapd#7390 closed: tests: unit test for a refresh failing on configure hook [12:11] does snapcraft need to use the lxd snap or can it use an already set up lxd? [12:20] whoops, just spotted a bug in the model pr [12:20] * Chipaca pokes at it [12:31] zyga: left a coment https://github.com/snapcore/snapd/pull/7442#discussion_r323712211 [12:31] PR #7442: tests: extend mount-ns test to handle mimics [12:32] zyga: fwiw, core18 mounts seem ok, though with /usr/share/gdb you'd have just 2 entries [12:32] yeah, I will change that [12:33] I made progress on https://github.com/snapcore/snapd/pull/7421 [12:33] PR #7421: cmd/snap-confine: unmount /writable from snap view [12:33] check out the pretty red diff there :) === ricab is now known as ricab|lunch [13:50] re [13:50] mvo: sorry, daughter had existential problems related to lunch [13:50] mvo: needed to have a walk and talk to her [13:50] uh :) [13:51] zyga: who doesnt have the occasional mid-lunch crisis :P [13:54] ijohnson: ah just saw your last question, and you're right i hadn't replied [13:54] np, thanks for clarifying [13:54] ijohnson: let me clarify more just in case [13:55] * ijohnson needs to get a cup of coffee then will read through [13:55] ijohnson: _if_ the output is yamlish, and _if_ the value needs to start with "- ", then yes use esc.dash [13:56] it might be our case there [13:57] ijohnson: e.g. in 'snap info', we use esc.dash for "this channel is closed" (e.g. stable in 'snap info multipass'), but not for the - at the end of a channel line === ricab|lunch is now known as ricab [14:31] okay, thanks Chipaca for that explanation [14:32] ijohnson_: short answer: copy the output without the dash into http://yaml-online-parser.appspot.com/ and if it complains, change it to dash [14:32] yes for the `snap model --verbose` we will hit that case where the output is yamlish and the value starts with "- (..." [14:46] PR snapcraft#2708 closed: tools: make environment-setup container name and image configurable [15:05] oh well, fun with json, i feel that https://bugs.launchpad.net/snapd/+bug/1843683 is a tip of an iceberg [15:05] Bug #1843683: cannot unmarshal state entry for "timings" [15:08] fun fact: the only way to unmarshall something like 4.39e+08 is to have float in struct; using DecodeWithNumber doesn't help [15:11] niemeyer: hey, we talked yesterday about pagination support for the spread google backend and you suggested to look into something more generic. I did that and pushed https://github.com/snapcore/spread/pull/87 - it would be great if you could have a look and tell me if that is what you had in mind. no need for a in-depth review yet, just a check. if you think this should be the path forward I will polish it a bit and add more tests. if you prefer [15:11] the simpler approach in pr69 thats also fine of course :) [15:11] PR spread#87: google: add generic pagination support on GCE results [15:11] while i can change durations in debug timings to float64 both in timing definition and in the client to make everything happy end-to-end (and it's backward compatible change), we may have similiar problem with doing/undoing time on tasks. but what made json marshaller use 4.39e+08 out of sudden to encode int64 duration in the state is still mistery to me [15:16] * cachio bank [15:16]