=== setuid is now known as zZZZzzetuid [05:14] morning [06:56] mborzecki: great job on 7166 - thanks that this is landed now [06:56] mvo: yw, it was fun :P [06:56] mvo: btw. got a report from fedora user about snapd dying due to ABRT, got the logs eventually [06:57] mvo: what happens is that the system goes to sleep, once it's woken up systemd kills snapd due to watchdog timeout :/ [06:57] mborzecki: oh no! [06:58] mborzecki: we can't be the only ones affected by this, this sounds like a very generic issue - or are we using the watchdog wrong in some way? [06:58] (I guess its a bit early to ask these questions :) [07:00] good morning [07:00] zyga: hey [07:02] mvo: thank you for the review on https://github.com/snapcore/snapd/pull/7549 [07:02] PR #7549: cmd/libsnap: use cgroup.procs instead of tasks [07:02] I'm sorry for being absent on Friday [07:02] I was a full-time mom for the last three days [07:04] Good morning [07:04] hey zyga - good morning! [07:04] zyga: Speaking from experiance imho that is both wonderful and sucks. I hope you get decently paid while home with sick kids at least [07:06] pokk: paid with hugs and diapers :) [07:06] zyga: that's great :) [07:19] mborzecki: can you do a 2nd review on https://github.com/snapcore/snapd/pull/7549 please [07:19] PR #7549: cmd/libsnap: use cgroup.procs instead of tasks [07:27] sil2100: good morning! could you please have a look at the pending snapd SRU in the unapproved queue? and who should I poke about universe SRUs? I did a drive-by rxtx sru to bionic on sunday to fix the arduino IDE in bionic, pretty trivial and was wondering if I need to poke someone else? [07:35] mvo: hey! Let me take a look at those then [07:35] sil2100: thank you [07:36] hm, weird, someone accepted snapd to disco but didn't for bionic and xenial [07:36] Anyway, looking into that [07:36] Ah [07:36] mvo: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1840740/comments/3 [07:36] Bug #1840740: [SRU] 2.41

[07:37] mvo: can you address that by any chance? [07:37] sil2100: uh, sorry, I must have missed that [07:37] sil2100: will look into a fix, please reject [07:38] mvo: ok! Give me a poke once you re-upload, I can review those then - in the meantime, I'll look at rxtx [07:39] sil2100: thanks, will probably take a bit until I will reupload, need to medidate over this a bit :/ [07:39] (mostly the test-case part) [07:40] mvo: I suppose xenial also has the same change, so I'll reject that as well [07:47] PR snapd#7544 closed: tests: fix snapd-failover test for core18 tests on boards [07:50] PR snapd#7549 closed: cmd/libsnap: use cgroup.procs instead of tasks [07:51] mvo: based on https://github.com/golang/go/issues/19810 i think we could try to replace the time.Ticker in cmd/snapd/main.go with time.After(), but that's about the only idea i have atm [07:52] and we ping systemd at twice the required frequency, while systemd should be smart enough to use monotonic timers === pstolowski|afk is now known as pstolowski [07:57] morning [08:00] pstolowski: hey [08:00] mborzecki: ta [08:12] mvo: updated https://github.com/snapcore/snapd/pull/7543 [08:13] PR #7543: release: make forced dev mode look at cgroupv2 support [08:26] mborzecki: thank you, much appreciated. sorry for the nitpicking there but it took me a couple of minutes(!) before my first cup of tea to understand the defer. which probably says more about me than the code :/ [08:27] mvo: no worries :) [08:52] PR snapd#7430 closed: overlord/snapstate: add SetTaskSnapSetup helper + unit tests [08:52] zyga: some comments in #7547 [08:52] PR #7547: many: use a dedicated named cgroup hierarchy for tracking [08:54] zyga: iirc we'd stil want to degrade gracefully, but maybe my memory is playing tricks [08:54] PR snapd#7437 closed: wrappers/services.go: add disabled svc list arg to AddSnapServices [08:55] mborzecki: I'm not sure, we'll see [08:55] mborzecki: thank you for the review, I should have posted the more fresh version last week but nothing is lost :) You'll see it soon enough [08:55] mvo: pedronis: more and more i wonder why we did /v2/download instead of just having a download action on /v2/snaps [08:56] Chipaca: because of different response ? [08:56] i guess it makes it easier to change our mind / fine-tune the rules for access, but eh [08:57] pedronis: pr'aps. Or maybe because then we can use it to download things that aren't snaps? [08:57] that might happen [08:57] pedronis: mvo: have you talked about the assertions side of 'snap download' when going via snapd? [08:58] pedronis: mvo: also 'snap known --remote' [08:58] Chipaca: mvo added a remote option to the assertion end point [08:58] Chipaca: yeah [08:58] it has landed [08:58] afaik [08:58] neat [08:58] Chipaca: we are ready pretty much, hold on a sec [08:59] Chipaca: I just chunked the PR up into small bits [08:59] zyga: so far it's looking good, all things considered the change should be rather simple as we discussed before :) [08:59] mborzecki: yeah, I was just thinking about how to stage it [08:59] mborzecki: perhaps the next chunk to land for real would be the cgroup alone, without the agent [08:59] mborzecki: and after that just the agent [08:59] mborzecki: does that sound okay? [09:00] mborzecki: I worry that together it will be less approachable [09:00] zyga: yeah, just the cgroup change would be functionally identical to what we have now [09:00] Chipaca: https://github.com/snapcore/snapd/compare/master...mvo5:snap-download-via-snapd?expand=1 this is the full pr, it needs a bit of polish still around the edges that are not proposed yet but once my other two prs are merged the rest is hopefully eay [09:00] Chipaca: fwiw, 7533 and 7510 [09:01] mvo: pedronis: completely unrelated and so I either write it down as a TODO, or forget about it: do we want to change the cache (as in store/cache) to use the same format for sha3 as 'snap info'? right now it's different which makes it harder to use the two together [09:01] Chipaca: I think that makes sense [09:01] Chipaca: in which sense together? [09:02] the cache is an implementation detail [09:02] pedronis: 'snap info --verbose some-snap' tells you the sha, but you can't then look in the cache to know which file it is [09:02] is this for debugging? [09:03] yes [09:04] Chipaca: it means we need to look at both formats for a while? [09:04] pedronis: you can always 'sudo find /var/lib/snapd/cache | sudo xargs snap info --verbose ' but that is really expensive, particularly on slow machines [09:04] pedronis: yes [09:07] i've added a card [09:07] Chipaca: we could also have snap debug list-cache or something [09:07] anyway it's kind of bottom of the queue given all the things [09:07] yeah :) [09:08] pedronis: ooh, snap debug cache might be useful [09:08] yeah, thats a nice idea [09:08] not just for the snap cache [09:11] Chipaca: but given is low, it should really go to the forum as backlog, not as a card, we already have cards that need to move back to the forum [09:12] ah [09:12] sorry [09:12] Chipaca: 19.10 TODO is really for stuff we must do [09:12] it can go to upcoming if we were sure it would be done soon [09:12] but we aren't sure of that :) [09:50] coffee [10:05] pedronis: #7541 can land [10:05] PR #7541: seed/seedwriter: support for extra snaps [10:08] mborzecki: mvo: thanks [10:08] PR snapd#7541 closed: seed/seedwriter: support for extra snaps [10:13] mvo: mborzecki: pstolowski: #7556 is ready for review, is the actual switch to use seedwriter in image.go, there is quite a bit of red in it [10:14] PR #7556: image,seed/seedwriter: switch image to use seedwriter.Writer [10:14] mvo: #7462 needs a re-review [10:14] PR #7462: asserts: introduce explicit support for grade for Core 20 models [10:15] pedronis: thanks [10:16] pedronis: I have a look [10:16] pedronis: just sent you a mail about something unrelated, would be great if you could have a look [10:16] pedronis: (but not high priority) [10:18] mvo: #7533 now suffering from the context conflict that mborzecki called out [10:18] PR #7533: client: add support to use the new "download" API [10:19] pedronis: ty [10:20] Chipaca: let me fix it [10:20] I'll swithc to doing some reviewing after lunch [10:21] *switch [10:30] pedronis: thanks [10:31] * Chipaca takes a break [10:47] - Download snap "test-snapd-dbus-consumer" (6) from channel "beta" (stream error: stream ID 1; PROTOCOL_ERROR) [10:47] meh, again [10:48] how did we fix it the last time? [10:48] mborzecki: maybe its failing 3 times? [10:49] mborzecki: we only retry for protocol errror [10:49] mborzecki: so if it happens multiple times in the same stream eventually it fails iirc [10:49] mborzecki: what pr is that? [10:49] mvo: https://github.com/snapcore/snapd/pull/7543 [10:49] PR #7543: release: make forced dev mode look at cgroupv2 support [10:49] mvo: https://travis-ci.org/snapcore/snapd/jobs/594466695 [10:52] mvo: from the logs it does not look like snapd retried due to PROTOCOL_ERROR [10:52] mborzecki: I see some retries here [10:53] mborzecki: let me look further [10:53] mvo: i must be looking at other tests, I see a retry due to unexpected EOF and then there's protocol err but no retry [10:55] mborzecki: oh, interessting, I have one tests here with "Retry because of: stream error: stream ID 1; PROTOCOL ERROR [10:56] mborzecki: but I have no looked at all of them yet [10:56] mvo: one failure also kept getting &errors.errorString{s:"unexpected EOF"} [10:57] mborzecki: yeah [10:57] mborzecki: I think I see what you see, one test seems to not retry [10:58] mborzecki: I wonder if different bits of the code in http2 produce slightly different protocol errors or what is going on there, its very strange [10:58] * mvo should look closer before making strong assertions like "very strange" [10:58] mvo: perhaps, but we look for 'PROTOCOL_ERROR' which appears in the non retry case too [10:59] mvo: and it's the same ubuntu 16.04-32 system, so cannot be go version change at play [11:03] mborzecki: right, in the log it seems like the total time of the attemp is ~3m30 so things looks pretty bad [11:03] mborzecki: to download yq [11:03] mborzecki: so maybe just a fastly issue? [11:07] mborzecki: I updated https://github.com/snapcore/snapd/pull/7547 [11:07] PR #7547: many: use a dedicated named cgroup hierarchy for tracking [11:07] mborzecki: I think it has a chance to pass and get merged now [11:08] mborzecki: I will put the agent on hold for a sec and do a branch from this to fix setting up cgroup for classic snaps as well (the omission I mentioned earlier) [11:08] mborzecki: then come back to integrate the agent [11:10] rogpeppe: hello [11:10] zyga: hiya [11:10] rogpeppe: do you have any updates for us? is there a chance to get the SD image? [11:11] zyga: unfortunately my dad thought he'd dropboxed the image before going away, but only did a zip file with just the script instead, sadly [11:11] ah, I see [11:11] zyga: so i'm waiting for him to get back from his travels (soon, I think) [11:11] that's okay, just keep us posted if you can get the image [11:11] thank you :) [11:12] we are moving some stuff behind the scenes to make the pi more reliable in conjunction with other developers from beyond the snapd team [11:12] it's not something that will be done overnight but I think that at least one annoying thing can be fixed in the next one or two releases (the reboot getting stuck there) [11:19] mborzecki: did you save the protocol error log? [11:20] Chipaca: i still ahve the log opened [11:20] mborzecki: can you get at the 'raw' log? [11:20] pstolowski: I did pass over 7355, just nitpicks of various importance [11:20] *a pass [11:20] Chipaca: sure, just a sec [11:21] pedronis: ty [11:22] Chipaca: https://paste.ubuntu.com/p/bFgCsXWvzX/ [11:23] hah! [11:24] pstolowski: a question could we/should we consider using SetupMany in setupSecurityByBackend ? [11:29] mborzecki: mvo: AFAICT the problem is that we got protocol error not on doing the request but on downloading [11:29] mborzecki: mvo: i.e., we might need to add the same check to ShouldRetryHttpResponse? [11:29] Chipaca: aha, I missed that [11:29] not sure tho [11:30] ah, it's called directly in the store package [11:31] Chipaca: duh, but we call SHouldRetryError() there too [11:31] mborzecki: yeah i was wrong (surprise!) [11:32] the RetryHttpResponse will say yes because it's 206 [11:32] wait, no, it'll say no for that same reason [11:32] huh [11:41] pedronis: indeed! good idea, we could save a little there as well. i can do in a followup [11:50] PR snapd#7557 opened: interfaces/mount: account for cgroup version when reporting supported features [11:51] zyga: ^^ [11:51] ack [11:51] looking [11:52] +1 [11:52] mborzecki: we don't have 31 tests yet, right? [11:52] zyga: we don't, there was an image cachio prepared a while ago, but that was before 31 branched iirc [11:52] I see [11:52] ok, [11:52] it would be good to discuss this today [11:52] I could use 31 image [11:53] * Chipaca takes a deep breath and reminds himself he's not here to remove the stupid from random people [11:53] * Chipaca goes to check on lunch [11:54] * mvo hugs juliank for his command-nof-found upload [11:56] will that finally fix core18 ? [11:56] :) [12:12] off to pick up the kids [12:14] mborzecki: did a pass with some questions on the timeutil bits of #7443 [12:15] PR #7443: timeutil: fix schedules with ambiguous nth weekday spans [12:40] Chipaca: do you remember why given it takes at most one snap we support taking multiple in the json to /download? [12:50] PR snapd#7558 opened: boot,dirs,image: various refinements in the prepare-image code switched to seedwriter === ricab is now known as ricab|lunch [13:13] PR snapd#7543 closed: release: make forced dev mode look at cgroupv2 support [13:21] Chipaca: o/ - to follow up on refresh issues from last week - when do the systemd files get regenerated in the refresh action? It looks like at some point we're getting services started that are pointing at old snaps working directory [13:25] bloodearnest: if you have a recent 'refresh' change [13:25] bloodearnest: you can 'snap tasks --last=refresh' [13:26] Chipaca: thanks [13:27] bloodearnest: but it is: run pre-refresh hook, stop services, remove aliases, remove current symlink, copy data, … , post-refresh hook, start serviecs [13:32] Chipaca: does start services include generating the systemd files? [13:32] Because the services seem to be starting up with the old working directory [13:38] mborzecki: hey [13:39] mborzecki: can you quickly decide if I should open the named cgroup PR (away from draft mode) [13:39] mborzecki: or finish the change to include classic snaps as well first [13:40] zyga: i think it's ok to open it now while to code is small, then classic can be done later as part of #7302 (or an intro to that PR) [13:40] PR #7302: cmd/snap-confine: add support for parallel instances of classic snaps, global mount ns initialization [13:40] okay, I'll do so now [13:42] mborzecki: https://github.com/snapcore/snapd/pull/7547 is open now [13:42] PR #7547: many: use a dedicated named cgroup hierarchy for tracking [13:42] mborzecki: do you want to add that snap mount unit to 7302? [13:43] zyga, hi, it seems I managed to reproduce the issue with "snap try" and rsyncing files, leading to snapd not finding the executable in the snap [13:43] hey, cool, do you have it scripted by any chance? [13:43] zyga: i'll try a separate PR, just to see the whole spread suite run [13:43] something we can play with and debug? [13:43] mborzecki: Ok [13:44] zyga, no, I don't have a way to reproduce it, it just happens [13:44] zyga, but I could debug, if you have commands I can run [13:44] ackk: what is the error message you see that is the immediate problem? [13:45] $ maas [13:45] execv failed: No such file or directory [13:45] zyga, ^ [13:45] are you able to execute more commands now to explore this? [13:45] zyga, and of course, $SNAP/command-maas.wrapper and $SNAP/bin/maas (which is called by that one) are there [13:45] zyga, yeah [13:45] ackk: can you ls -ld /snap/maas/current [13:46] zyga, lrwxrwxrwx 1 root root 2 Oct 7 10:23 /snap/maas/current -> x1 [13:46] ackk: snap run --strace=raw maas [13:46] ackk: is x1 the expected correct revision? [13:46] ackk: cat /proc/self/mountinfo | grep -F '(deleted') [13:46] er [13:46] zyga, yes [13:46] fix the quote please [13:46] zyga, nothing there [13:47] also: [13:47] $ snap run --strace=raw maas [13:47] error: exit status 1 [13:47] ackk: cat /proc/self/mountinfo | grep maas [13:47] zyga, FTR this is in a bionic container [13:47] right, I recall those are used [13:47] and are most likely a factor [13:47] I'd love to understand what's the key bit that is broken [13:47] zyga, https://paste.ubuntu.com/p/8VcMZhmtrS/ [13:48] ackk: dmesg | grep DENIED [13:49] ackk: also, sudo nsenter -m/run/snapd/ns/maas.mnt -- in the resulting shell cat /proc/self/mountinfo | grep -F deleted [13:49] zyga, on the host or in the container? [13:49] (the dmesg) [13:49] ackk: hmmm, should be the same AFAIK [13:50] so either [13:50] zyga, I used to run snappy-debug on the host, or I would miss some denials [13:50] not sure if it's relevant [13:50] hmm, perhaps there's something I don't understand about lxd [13:50] try on the host please [13:50] zyga, https://paste.ubuntu.com/p/JSXhbrR4PV/ [13:51] zyga, empty result for deleted mountinfo [13:51] hum [13:51] [92186.616879] audit: type=1400 audit(1570431426.676:2750): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-maas_" profile="/snap/snapd/4605/usr/lib/snapd/snap-confine" pid=4605 comm="snap-confine" family="netlink" sock_type="raw" protocol=15 requested_mask="send receive" denied_mask="send receive" [13:51] <- this is interesting, separately from the issue that affects you [13:52] ackk: can you navigate around in that nsenter shell, to see if the executables are where you expect [13:52] zyga, this would be basically $SNAP right? [13:52] yeah, but SNAP is not set, so manually [13:52] sure [13:52] ackk: one more idea, SNAP_CONFINE_DEBUG=yes snap run ... (as you run maas otherwise) [13:53] $ SNAP_CONFINE_DBUG=yes snap run -- maas --help [13:53] execv failed: No such file or directory [13:53] zyga, ^ [13:54] zyga, afaict executables are there in the right place [13:54] SNAP_CONFINE_DEBUG=yes [13:54] DBUG vs DEBUG [13:54] interesting, perhaps it is failing to exec something other than the app [13:54] ugh [13:55] zyga, http://paste.ubuntu.com/p/jdKzRHbFsH/ [13:55] holly molly [13:55] I suspect I know what this is [13:55] from the nsenter shell [13:55] can you ls -ld /usr/lib/snapd/snap-exec [13:56] bloodearnest: sorry, missed your question. Actually writing the service files is part of the "link-snap" step, i.e. "make snap available to the system" [13:56] ackk: this is a core18 system? [13:56] ackk: as in [13:56] ackk: a classic system with core18 and snapd snap? [13:56] zyga, correct [13:56] zyga, snap-exect not found [13:57] zyga, bingo :) [13:57] bloodearnest: i'm very curious to know what the cause of the weirdness you're seeing is, fwiw [13:57] and you happen to follow which snapd channel? [13:57] ackk: ls -ld /usr/lib/snapd [13:57] bloodearnest: if you're looking at 'snap tasks --last=refresh' note you might also find interesting info in 'snap debug timings --last=refresh' [13:57] zyga, http://paste.ubuntu.com/p/wTsT8yPBHF/ tl;dr all stable [13:58] zyga, /usr/lib/snapd is there [13:58] is it a directory? [13:58] what's in it? [13:58] zyga, yes: drwxr-xr-x 2 root root 0 Oct 1 05:44 /usr/lib/snapd [13:58] zyga, but, empty [13:58] uhuhhh [13:58] okay [13:58] I strongly suspect I know what this is [13:59] mvo: ^ [13:59] ackk: the good news, it's high up our stack [13:59] ackk: the bad news, it's not your system, it's a real general bug [13:59] ackk: can you do snap changes please? [13:59] zyga, well, makes me feel less bad :) [13:59] zyga, in the nsenter or outside? [13:59] zyga: oh no [13:59] ackk: outside please [13:59] ackk: is this reported anywhere? [14:00] ackk: I think you might have already [14:00] mvo: I think the tools problem is more severe [14:00] zyga, http://paste.ubuntu.com/p/2MdPHqdyfd/ [14:00] mvo: with enough rotations I suspect we unmount /usr/lib/snapd from actively used namespaces [14:00] zyga, I don't remember if I did report it, unfortunately I never found a reliable way to make it fail. since last time we talked about it, it hasn't happened anymore and I tried a loop of rsync's [14:00] ackk: can you attempt to pinpoint when the issue occurred for you this time? [14:00] ackk: in that log, at which time was it [14:01] ackk: was it _before_ all of this? [14:01] zyga, no, I mean I've "snap try"'d and removed a bunch of times [14:01] zyga, but basically snap try , bunch of rsync -> broken [14:01] zyga, I didn't snap try from an already installed snap, if that helps [14:01] hmm [14:01] hmm [14:02] right [14:02] I wonder if snapd refreshed in the middle of all the "snap try" commands anywhere [14:02] mvo: pedronis: wrt downloads, if we want top drop action and change snaps to snap, we should do that before or together with 7510 [14:03] zyga, the snapd snap says refreshed 33d ago [14:03] hmm [14:03] in that case I know less, it's not my initial assumption [14:03] Chipaca: for clarity (hopefully) I wrote this, related to channel stuff, https://trello.com/c/GTSFCPG5/292-do-not-switch-tracks-with-risk-only-channel-specification-snap-refresh-switch#comment-5d9b456d31aa8d1b410bcdc2 [14:03] ackk: the maas snap has a configure hook? [14:04] zyga, yes [14:04] right? [14:04] ok [14:04] which does a lot [14:04] so each snap try runs stuff [14:04] zyga, correct [14:04] ackk: can you please share the exact rsync command you are using [14:04] I'll try to reproduce this with a demo snap with a configure hook [14:05] PR snapd#7559 opened: tests: change regex to validate access to cdn during snap download