[05:14] <mborzecki> morning
[06:56] <mvo> mborzecki: great job on 7166 - thanks that this is landed now
[06:56] <mborzecki> mvo: yw, it was fun :P
[06:56] <mborzecki> mvo: btw. got a report from fedora user about snapd dying due to ABRT, got the logs eventually
[06:57] <mborzecki> mvo: what happens is that the system goes to sleep, once it's woken up systemd kills snapd due to watchdog timeout :/
[06:57] <mvo> mborzecki: oh no!
[06:58] <mvo> mborzecki: we can't be the only ones affected by this, this sounds like a very generic issue - or are we using the watchdog wrong in some way?
[06:58] <mvo> (I guess its a bit early to ask these questions :)
[07:00] <zyga> good morning
[07:00] <mborzecki> zyga:  hey
[07:02] <zyga> mvo: thank you for the review on https://github.com/snapcore/snapd/pull/7549
[07:02] <mup> PR #7549: cmd/libsnap: use cgroup.procs instead of tasks <Created by zyga> <https://github.com/snapcore/snapd/pull/7549>
[07:02] <zyga> I'm sorry for being absent on Friday
[07:02] <zyga> I was a full-time mom for the last three days
[07:04] <pokk> Good morning
[07:04] <mvo> hey zyga - good morning!
[07:04] <pokk> zyga: Speaking from experiance imho that is both wonderful and sucks. I hope you get decently paid while home with sick kids at least
[07:06] <zyga> pokk: paid with hugs and diapers :)
[07:06] <pokk> zyga: that's great :)
[07:19] <zyga> mborzecki: can you do a 2nd review on https://github.com/snapcore/snapd/pull/7549 please
[07:19] <mup> PR #7549: cmd/libsnap: use cgroup.procs instead of tasks <Created by zyga> <https://github.com/snapcore/snapd/pull/7549>
[07:27] <mvo> sil2100: good morning! could you please have a look at the pending snapd SRU in the unapproved queue? and who should I poke about universe SRUs? I did a drive-by rxtx sru to bionic on sunday to fix the arduino IDE in bionic, pretty trivial and was wondering if I need to poke someone else?
[07:35] <sil2100> mvo: hey! Let me take a look at those then
[07:35] <mvo> sil2100: thank you
[07:36] <sil2100> hm, weird, someone accepted snapd to disco but didn't for bionic and xenial
[07:36] <sil2100> Anyway, looking into that
[07:36] <sil2100> Ah
[07:36] <sil2100> mvo: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1840740/comments/3
[07:36] <mup> Bug #1840740: [SRU] 2.41 <verification-needed> <verification-needed-disco> <snapd (Ubuntu):Fix Released> <snapd (Ubuntu Xenial):New> <snapd (Ubuntu Bionic):Incomplete> <snapd (Ubuntu Disco):Fix Committed> <https://launchpad.net/bugs/1840740>
[07:37] <sil2100> mvo: can you address that by any chance?
[07:37] <mvo> sil2100: uh, sorry, I must have missed that
[07:37] <mvo> sil2100: will look into a fix, please reject
[07:38] <sil2100> mvo: ok! Give me a poke once you re-upload, I can review those then - in the meantime, I'll look at rxtx
[07:39] <mvo> sil2100: thanks, will probably take a bit until I will reupload, need to medidate over this a bit :/
[07:39] <mvo> (mostly the test-case part)
[07:40] <sil2100> mvo: I suppose xenial also has the same change, so I'll reject that as well
[07:47] <mup> PR snapd#7544 closed: tests: fix snapd-failover test for core18 tests on boards <Created by sergiocazzolato> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7544>
[07:50] <mup> PR snapd#7549 closed: cmd/libsnap: use cgroup.procs instead of tasks <Created by zyga> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7549>
[07:51] <mborzecki> mvo: based on https://github.com/golang/go/issues/19810 i think we could try to replace the time.Ticker in cmd/snapd/main.go with time.After(), but that's about the only idea i have atm
[07:52] <mborzecki> and we ping systemd at twice the required frequency, while systemd should be smart enough to use monotonic timers
[07:57] <pstolowski> morning
[08:00] <mborzecki> pstolowski: hey
[08:00] <mvo> mborzecki: ta
[08:12] <mborzecki> mvo:  updated https://github.com/snapcore/snapd/pull/7543
[08:13] <mup> PR #7543: release: make forced dev mode look at cgroupv2 support <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7543>
[08:26] <mvo> mborzecki: thank you, much appreciated. sorry for the nitpicking there but it took me a couple of minutes(!) before my first cup of tea to understand the defer. which probably says more about me than the code :/
[08:27] <mborzecki> mvo:  no worries :)
[08:52] <mup> PR snapd#7430 closed: overlord/snapstate: add SetTaskSnapSetup helper + unit tests <Created by anonymouse64> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/7430>
[08:52] <mborzecki> zyga:  some comments in #7547
[08:52] <mup> PR #7547: many: use a dedicated named cgroup hierarchy for tracking <Created by zyga> <https://github.com/snapcore/snapd/pull/7547>
[08:54] <mborzecki> zyga: iirc we'd stil want to degrade gracefully, but maybe my memory is playing tricks
[08:54] <mup> PR snapd#7437 closed:  wrappers/services.go: add disabled svc list arg to AddSnapServices <Created by anonymouse64> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/7437>
[08:55] <zyga> mborzecki: I'm not sure, we'll see
[08:55] <zyga> mborzecki: thank you for the review, I should have posted the more fresh version last week but nothing is lost :) You'll see it soon enough
[08:55] <Chipaca> mvo: pedronis: more and more i wonder why we did /v2/download instead of just having a download action on /v2/snaps
[08:56] <pedronis> Chipaca: because of different response ?
[08:56] <Chipaca> i guess it makes it easier to change our mind / fine-tune the rules for access, but eh
[08:57] <Chipaca> pedronis: pr'aps. Or maybe because then we can use it to download things that aren't snaps?
[08:57] <pedronis> that might happen
[08:57] <Chipaca> pedronis: mvo: have you talked about the assertions side of 'snap download' when going via snapd?
[08:58] <Chipaca> pedronis: mvo: also 'snap known --remote'
[08:58] <pedronis> Chipaca: mvo added a remote option to the assertion end point
[08:58] <mvo> Chipaca: yeah
[08:58] <pedronis> it has landed
[08:58] <pedronis> afaik
[08:58] <Chipaca> neat
[08:58] <mvo> Chipaca: we are ready pretty much, hold on a sec
[08:59] <mvo> Chipaca: I just chunked the PR up into small bits
[08:59] <mborzecki> zyga: so far it's looking good, all things considered the change should be rather simple as we discussed before :)
[08:59] <zyga> mborzecki: yeah, I was just thinking about how to stage it
[08:59] <zyga> mborzecki: perhaps the next chunk to land for real would be the cgroup alone, without the agent
[08:59] <zyga> mborzecki: and after that just the agent
[08:59] <zyga> mborzecki: does that sound okay?
[09:00] <zyga> mborzecki: I worry that together it will be less approachable
[09:00] <mborzecki> zyga: yeah, just the cgroup change would be functionally identical to what we have now
[09:00] <mvo> Chipaca: https://github.com/snapcore/snapd/compare/master...mvo5:snap-download-via-snapd?expand=1 this is the full pr, it needs a bit of polish still around the edges that are not proposed yet but once my other two prs are  merged the rest is hopefully eay
[09:00] <mvo> Chipaca: fwiw, 7533 and 7510
[09:01] <Chipaca> mvo: pedronis: completely unrelated and so I either write it down as a TODO, or forget about it: do we want to change the cache (as in store/cache) to use the same format for sha3 as 'snap info'? right now it's different which makes it harder to use the two together
[09:01] <mvo> Chipaca: I think that makes sense
[09:01] <pedronis> Chipaca: in which sense together?
[09:02] <pedronis> the cache is an implementation detail
[09:02] <Chipaca> pedronis: 'snap info --verbose some-snap' tells you the sha, but you can't then look in the cache to know which file it is
[09:02] <pedronis> is this for debugging?
[09:03] <Chipaca> yes
[09:04] <pedronis> Chipaca: it means we need to look at both formats for a while?
[09:04] <Chipaca> pedronis: you can always 'sudo find /var/lib/snapd/cache | sudo xargs snap info --verbose ' but that is really expensive, particularly on slow machines
[09:04] <Chipaca> pedronis: yes
[09:07] <Chipaca> i've added a card
[09:07] <pedronis> Chipaca: we could also have snap debug list-cache or something
[09:07] <pedronis> anyway it's kind of bottom of the queue given all the things
[09:07] <Chipaca> yeah :)
[09:08] <Chipaca> pedronis: ooh, snap debug cache might be useful
[09:08] <mvo> yeah, thats a nice idea
[09:08] <Chipaca> not just for the snap cache
[09:11] <pedronis> Chipaca: but given is low, it should really go to the forum as backlog, not as a card, we already have cards that need to move back to the forum
[09:12] <Chipaca> ah
[09:12] <Chipaca> sorry
[09:12] <pedronis> Chipaca: 19.10 TODO is really for stuff we must do
[09:12] <pedronis> it can go to upcoming if we were sure it would be done soon
[09:12] <pedronis> but we aren't sure of that :)
[09:50] <zyga> coffee
[10:05] <mborzecki> pedronis: #7541 can land
[10:05] <mup> PR #7541: seed/seedwriter: support for extra snaps <Created by pedronis> <https://github.com/snapcore/snapd/pull/7541>
[10:08] <pedronis> mborzecki: mvo: thanks
[10:08] <mup> PR snapd#7541 closed: seed/seedwriter: support for extra snaps <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7541>
[10:13] <pedronis> mvo: mborzecki: pstolowski:  #7556 is ready for review, is the actual switch to use seedwriter in image.go, there is quite a bit of red in it
[10:14] <mup> PR #7556: image,seed/seedwriter: switch image to use seedwriter.Writer <Created by pedronis> <https://github.com/snapcore/snapd/pull/7556>
[10:14] <pedronis> mvo: #7462 needs a re-review
[10:14] <mup> PR #7462: asserts: introduce explicit support for grade for Core 20 models <Created by pedronis> <https://github.com/snapcore/snapd/pull/7462>
[10:15] <mvo> pedronis: thanks
[10:16] <mvo> pedronis: I have a look
[10:16] <mvo> pedronis: just sent you a mail about something unrelated, would be great if you could have a look
[10:16] <mvo> pedronis: (but not high priority)
[10:18] <Chipaca> mvo: #7533 now suffering from the context conflict that mborzecki called out
[10:18] <mup> PR #7533: client: add support to use the new "download" API <Needs Samuele review> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7533>
[10:19] <pstolowski> pedronis: ty
[10:20] <mvo> Chipaca: let me fix it
[10:20] <pedronis> I'll swithc to doing some reviewing after lunch
[10:21] <pedronis> *switch
[10:30] <mvo> pedronis: thanks
[10:31]  * Chipaca takes a break
[10:47] <mborzecki> - Download snap "test-snapd-dbus-consumer" (6) from channel "beta" (stream error: stream ID 1; PROTOCOL_ERROR)
[10:47] <mborzecki> meh, again
[10:48] <mborzecki> how did we fix it the last time?
[10:48] <mvo> mborzecki: maybe its failing 3 times?
[10:49] <mvo> mborzecki: we only retry for protocol errror
[10:49] <mvo> mborzecki: so if it happens multiple times in the same stream eventually it fails iirc
[10:49] <mvo> mborzecki: what pr is that?
[10:49] <mborzecki> mvo:  https://github.com/snapcore/snapd/pull/7543
[10:49] <mup> PR #7543: release: make forced dev mode look at cgroupv2 support <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7543>
[10:49] <mborzecki> mvo: https://travis-ci.org/snapcore/snapd/jobs/594466695
[10:52] <mborzecki> mvo: from the logs it does not look like snapd retried due to PROTOCOL_ERROR
[10:52] <mvo> mborzecki: I see some retries here
[10:53] <mvo> mborzecki: let me look further
[10:53] <mborzecki> mvo: i must be looking at other tests, I see a retry due to unexpected EOF and then there's protocol err but no retry
[10:55] <mvo> mborzecki: oh, interessting, I have one tests here with "Retry because of: stream error: stream ID 1; PROTOCOL ERROR
[10:56] <mvo> mborzecki: but I have no looked at all of them yet
[10:56] <mborzecki> mvo:  one failure also kept getting &errors.errorString{s:"unexpected EOF"}
[10:57] <mvo> mborzecki: yeah
[10:57] <mvo> mborzecki: I think I see what you see, one test seems to not retry
[10:58] <mvo> mborzecki: I wonder if different bits of the code in http2 produce slightly different protocol errors or what is going on there, its very strange
[10:58]  * mvo should look closer before making strong assertions like "very strange"
[10:58] <mborzecki> mvo: perhaps, but we look for 'PROTOCOL_ERROR' which appears in the non retry case too
[10:59] <mborzecki> mvo:  and it's the same ubuntu 16.04-32 system, so cannot be go version change at play
[11:03] <mvo> mborzecki: right, in the log it seems like the total time of the attemp is ~3m30 so things looks pretty bad
[11:03] <mvo> mborzecki: to download yq
[11:03] <mvo> mborzecki: so maybe just a fastly issue?
[11:07] <zyga> mborzecki: I updated https://github.com/snapcore/snapd/pull/7547
[11:07] <mup> PR #7547: many: use a dedicated named cgroup hierarchy for tracking <Created by zyga> <https://github.com/snapcore/snapd/pull/7547>
[11:07] <zyga> mborzecki: I think it has a chance to pass and get merged now
[11:08] <zyga> mborzecki: I will put the agent on hold for a sec and do a branch from this to fix setting up cgroup for classic snaps as well (the omission I mentioned earlier)
[11:08] <zyga> mborzecki: then come back to integrate the agent
[11:10] <zyga> rogpeppe: hello
[11:10] <rogpeppe> zyga: hiya
[11:10] <zyga> rogpeppe: do you have any updates for us? is there a chance to get the SD image?
[11:11] <rogpeppe> zyga: unfortunately my dad thought he'd dropboxed the image before going away, but only did a zip file with just the script instead, sadly
[11:11] <zyga> ah, I see
[11:11] <rogpeppe> zyga: so i'm waiting for him to get back from his travels (soon, I think)
[11:11] <zyga> that's okay, just keep us posted if you can get the image
[11:11] <zyga> thank you :)
[11:12] <zyga> we are moving some stuff behind the scenes to make the pi more reliable in conjunction with other developers from beyond the snapd team
[11:12] <zyga> it's not something that will be done overnight but I think that at least one annoying thing can be fixed in the next one or two releases (the reboot getting stuck there)
[11:19] <Chipaca> mborzecki: did you save the protocol error log?
[11:20] <mborzecki> Chipaca: i still ahve the log opened
[11:20] <Chipaca> mborzecki: can you get at the 'raw' log?
[11:20] <pedronis> pstolowski: I did pass over 7355, just nitpicks of various importance
[11:20] <pedronis> *a pass
[11:20] <mborzecki> Chipaca: sure, just a sec
[11:21] <pstolowski> pedronis: ty
[11:22] <mborzecki> Chipaca: https://paste.ubuntu.com/p/bFgCsXWvzX/
[11:23] <Chipaca> hah!
[11:24] <pedronis> pstolowski: a question could we/should we consider using SetupMany in setupSecurityByBackend ?
[11:29] <Chipaca> mborzecki: mvo: AFAICT the problem is that we got protocol error not on doing the request but on downloading
[11:29] <Chipaca> mborzecki: mvo: i.e., we might need to add the same check to ShouldRetryHttpResponse?
[11:29] <mvo> Chipaca: aha, I missed that
[11:29] <Chipaca> not sure tho
[11:30] <mborzecki> ah, it's called directly in the store package
[11:31] <mborzecki> Chipaca: duh, but we call SHouldRetryError() there too
[11:31] <Chipaca> mborzecki: yeah i was wrong (surprise!)
[11:32] <Chipaca> the RetryHttpResponse will say yes because it's 206
[11:32] <Chipaca> wait, no, it'll say no for that same reason
[11:32] <Chipaca> huh
[11:41] <pstolowski> pedronis: indeed! good idea, we could save a little there as well. i can do in a followup
[11:50] <mup> PR snapd#7557 opened: interfaces/mount: account for cgroup version when reporting supported features <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7557>
[11:51] <mborzecki> zyga: ^^
[11:51] <zyga> ack
[11:51] <zyga> looking
[11:52] <zyga> +1
[11:52] <zyga> mborzecki: we don't have 31 tests yet, right?
[11:52] <mborzecki> zyga: we don't, there was an image cachio prepared a while ago, but that was before 31 branched iirc
[11:52] <zyga> I see
[11:52] <zyga> ok,
[11:52] <zyga> it would be good to discuss this today
[11:52] <zyga> I could use 31 image
[11:53]  * Chipaca takes a deep breath and reminds himself he's not here to remove the stupid from random people
[11:53]  * Chipaca goes to check on lunch
[11:54]  * mvo hugs juliank for his command-nof-found upload
[11:56] <ogra> will that finally fix core18 ?
[11:56] <juliank> :)
[12:12] <mborzecki> off to pick up the kids
[12:14] <pedronis> mborzecki: did a pass with some questions on the timeutil bits of #7443
[12:15] <mup> PR #7443: timeutil: fix schedules with ambiguous nth weekday spans <Bug> <Needs Samuele review> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7443>
[12:40] <pedronis> Chipaca: do you remember why given it takes at most one snap we support taking multiple in the json to /download?
[12:50] <mup> PR snapd#7558 opened: boot,dirs,image: various refinements in the prepare-image code switched to seedwriter <Created by pedronis> <https://github.com/snapcore/snapd/pull/7558>
[13:13] <mup> PR snapd#7543 closed: release: make forced dev mode look at cgroupv2 support <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7543>
[13:21] <bloodearnest> Chipaca: o/ - to follow up on refresh issues from last week - when do the systemd files get regenerated in the refresh action? It looks like at some point we're getting services started that are pointing at old snaps working directory
[13:25] <Chipaca> bloodearnest: if you have a recent 'refresh' change
[13:25] <Chipaca> bloodearnest: you can 'snap tasks --last=refresh'
[13:26] <bloodearnest> Chipaca: thanks
[13:27] <Chipaca> bloodearnest: but it is: run pre-refresh hook, stop services, remove aliases, remove current symlink, copy data, … , post-refresh hook, start serviecs
[13:32] <bloodearnest> Chipaca: does start services include generating the systemd files?
[13:32] <bloodearnest> Because the services seem to be starting up with the old working directory
[13:38] <zyga> mborzecki: hey
[13:39] <zyga> mborzecki: can you quickly decide if I should open the named cgroup PR (away from draft mode)
[13:39] <zyga> mborzecki: or finish the change to include classic snaps as well first
[13:40] <mborzecki> zyga: i think it's ok to open it now while to code is small, then classic can be done later as part of #7302 (or an intro to that PR)
[13:40] <mup> PR #7302: cmd/snap-confine: add support for parallel instances of classic snaps, global mount ns initialization <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7302>
[13:40] <zyga> okay, I'll do so now
[13:42] <zyga> mborzecki: https://github.com/snapcore/snapd/pull/7547 is open now
[13:42] <mup> PR #7547: many: use a dedicated named cgroup hierarchy for tracking <Created by zyga> <https://github.com/snapcore/snapd/pull/7547>
[13:42] <zyga> mborzecki: do you want to add that snap mount unit to 7302?
[13:43] <ackk> zyga, hi, it seems I managed to reproduce the issue with "snap try" and rsyncing files, leading to snapd not finding the executable in the snap
[13:43] <zyga> hey, cool, do you have it scripted by any chance?
[13:43] <mborzecki> zyga: i'll try a separate PR, just to see the whole spread suite run
[13:43] <zyga> something we can play with and debug?
[13:43] <zyga> mborzecki: Ok
[13:44] <ackk> zyga, no, I don't have a way to reproduce it, it just happens
[13:44] <ackk> zyga, but I could debug, if you have commands I can run
[13:44] <zyga> ackk: what is the error message you see that is the immediate problem?
[13:45] <ackk> $ maas
[13:45] <ackk> execv failed: No such file or directory
[13:45] <ackk> zyga, ^
[13:45] <zyga> are you able to execute more commands now to explore this?
[13:45] <ackk> zyga, and of course, $SNAP/command-maas.wrapper and $SNAP/bin/maas (which is called by that one) are there
[13:45] <ackk> zyga, yeah
[13:45] <zyga> ackk: can you ls -ld /snap/maas/current
[13:46] <ackk> zyga, lrwxrwxrwx 1 root root 2 Oct  7 10:23 /snap/maas/current -> x1
[13:46] <zyga> ackk: snap run --strace=raw maas
[13:46] <zyga> ackk: is x1 the expected correct revision?
[13:46] <zyga> ackk: cat /proc/self/mountinfo | grep -F '(deleted')
[13:46] <zyga> er
[13:46] <ackk> zyga, yes
[13:46] <zyga> fix the quote please
[13:46] <ackk> zyga, nothing there
[13:47] <ackk> also:
[13:47] <ackk> $ snap run --strace=raw maas
[13:47] <ackk> error: exit status 1
[13:47] <zyga> ackk: cat /proc/self/mountinfo | grep maas
[13:47] <ackk> zyga, FTR this is in a bionic container
[13:47] <zyga> right, I recall those are used
[13:47] <zyga> and are most likely a factor
[13:47] <zyga> I'd love to understand what's the key bit that is broken
[13:47] <ackk> zyga, https://paste.ubuntu.com/p/8VcMZhmtrS/
[13:48] <zyga> ackk: dmesg | grep DENIED
[13:49] <zyga> ackk: also, sudo nsenter -m/run/snapd/ns/maas.mnt -- in the resulting shell cat /proc/self/mountinfo | grep -F deleted
[13:49] <ackk> zyga, on the host or in the container?
[13:49] <ackk> (the dmesg)
[13:49] <zyga> ackk: hmmm, should be the same AFAIK
[13:50] <zyga> so either
[13:50] <ackk> zyga, I used to run snappy-debug on the host, or I would miss some denials
[13:50] <ackk> not sure if it's relevant
[13:50] <zyga> hmm, perhaps there's something I don't understand about lxd
[13:50] <zyga> try on the host please
[13:50] <ackk> zyga, https://paste.ubuntu.com/p/JSXhbrR4PV/
[13:51] <ackk> zyga, empty result for deleted mountinfo
[13:51] <zyga> hum
[13:51] <zyga> [92186.616879] audit: type=1400 audit(1570431426.676:2750): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-maas_<var-snap-lxd-common-lxd>" profile="/snap/snapd/4605/usr/lib/snapd/snap-confine" pid=4605 comm="snap-confine" family="netlink" sock_type="raw" protocol=15 requested_mask="send receive" denied_mask="send receive"
[13:51] <zyga> <- this is interesting, separately from the issue that affects you
[13:52] <zyga> ackk: can you navigate around in that nsenter shell, to see if the executables are where you expect
[13:52] <ackk> zyga, this would be basically $SNAP right?
[13:52] <zyga> yeah, but SNAP is not set, so manually
[13:52] <ackk> sure
[13:52] <zyga> ackk: one more idea, SNAP_CONFINE_DEBUG=yes snap run ... (as you run maas otherwise)
[13:53] <ackk> $ SNAP_CONFINE_DBUG=yes snap run -- maas --help
[13:53] <ackk> execv failed: No such file or directory
[13:53] <ackk> zyga, ^
[13:54] <ackk> zyga, afaict executables are there in the right place
[13:54] <zyga> SNAP_CONFINE_DEBUG=yes
[13:54] <zyga> DBUG vs DEBUG
[13:54] <zyga> interesting, perhaps it is failing to exec something other than the app
[13:54] <ackk> ugh
[13:55] <ackk> zyga, http://paste.ubuntu.com/p/jdKzRHbFsH/
[13:55] <zyga> holly molly
[13:55] <zyga> I suspect I know what this is
[13:55] <zyga> from the nsenter shell
[13:55] <zyga> can you ls -ld /usr/lib/snapd/snap-exec
[13:56] <Chipaca> bloodearnest: sorry, missed your question. Actually writing the service files is part of the "link-snap" step, i.e. "make snap available to the system"
[13:56] <zyga> ackk: this is a core18 system?
[13:56] <zyga> ackk: as in
[13:56] <zyga> ackk: a classic system with core18 and snapd snap?
[13:56] <ackk> zyga, correct
[13:56] <ackk> zyga, snap-exect not found
[13:57] <ackk> zyga, bingo :)
[13:57] <Chipaca> bloodearnest: i'm very curious to know what the cause of the weirdness you're seeing is, fwiw
[13:57] <zyga> and you happen to follow which snapd channel?
[13:57] <zyga> ackk: ls -ld /usr/lib/snapd
[13:57] <Chipaca> bloodearnest: if you're looking at 'snap tasks --last=refresh' note you might also find interesting info in 'snap debug timings --last=refresh'
[13:57] <ackk> zyga, http://paste.ubuntu.com/p/wTsT8yPBHF/ tl;dr all stable
[13:58] <ackk> zyga, /usr/lib/snapd is there
[13:58] <zyga> is it a directory?
[13:58] <zyga> what's in it?
[13:58] <ackk> zyga, yes: drwxr-xr-x 2 root root 0 Oct  1 05:44 /usr/lib/snapd
[13:58] <ackk> zyga, but, empty
[13:58] <zyga> uhuhhh
[13:58] <zyga> okay
[13:58] <zyga> I strongly suspect I know what this is
[13:59] <zyga> mvo: ^
[13:59] <zyga> ackk: the good news, it's high up our stack
[13:59] <zyga> ackk: the bad news, it's not your system, it's a real general bug
[13:59] <zyga> ackk: can you do snap changes please?
[13:59] <ackk> zyga, well, makes me feel less bad :)
[13:59] <ackk> zyga, in the nsenter or outside?
[13:59] <mvo> zyga: oh no
[13:59] <zyga> ackk: outside please
[13:59] <zyga> ackk: is this reported anywhere?
[14:00] <zyga> ackk: I think  you might have already
[14:00] <zyga> mvo: I think the tools problem is more severe
[14:00] <ackk> zyga, http://paste.ubuntu.com/p/2MdPHqdyfd/
[14:00] <zyga> mvo: with enough rotations I suspect we unmount /usr/lib/snapd from actively used namespaces
[14:00] <ackk> zyga, I don't remember if I did report it, unfortunately I never found a reliable way to make it fail. since last time we talked about it, it hasn't happened anymore and I tried a loop of rsync's
[14:00] <zyga> ackk: can you attempt to pinpoint when the issue occurred for you this time?
[14:00] <zyga> ackk: in that log, at which time was it
[14:01] <zyga> ackk: was it _before_ all of this?
[14:01] <ackk> zyga, no, I mean I've "snap try"'d and removed a bunch of times
[14:01] <ackk> zyga, but basically snap try , bunch of rsync -> broken
[14:01] <ackk> zyga, I didn't snap try from an already installed snap, if that helps
[14:01] <zyga> hmm
[14:01] <zyga> hmm
[14:02] <zyga> right
[14:02] <zyga> I wonder if snapd refreshed in the middle of all the "snap try" commands anywhere
[14:02] <Chipaca> mvo: pedronis: wrt downloads, if we want top drop action and change snaps to snap, we should do that before or together with 7510
[14:03] <ackk> zyga, the snapd snap says refreshed 33d ago
[14:03] <zyga> hmm
[14:03] <zyga> in that case I know less, it's not my initial assumption
[14:03] <pedronis> Chipaca: for clarity (hopefully) I wrote this, related to channel stuff, https://trello.com/c/GTSFCPG5/292-do-not-switch-tracks-with-risk-only-channel-specification-snap-refresh-switch#comment-5d9b456d31aa8d1b410bcdc2
[14:03] <zyga> ackk: the maas snap has a configure hook?
[14:04] <ackk> zyga, yes
[14:04] <zyga> right?
[14:04] <zyga> ok
[14:04] <ackk> which does a lot
[14:04] <zyga> so each snap try runs stuff
[14:04] <ackk> zyga, correct
[14:04] <zyga> ackk: can you please share the exact rsync command you are using
[14:04] <zyga> I'll try to reproduce this with a demo snap with a configure hook
[14:05] <mup> PR snapd#7559 opened: tests: change regex to validate access to cdn during snap download <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7559>
[14:05] <ackk> zyga, https://paste.ubuntu.com/p/B69smBJ73N/
[14:06] <ackk> zyga, where build/dev-snap/prime is the prime dir that gets snap try'd
[14:06] <mvo> Chipaca: in a meeting right now
[14:06] <zyga> and on the container you "snap try build/dev-snap/prime?
[14:06] <ackk> zyga, correct
[14:07] <zyga> ok
[14:07] <zyga> hmm, for now I have no more ideas
[14:07] <zyga> thank you for reporting this again
[14:07] <zyga> I'll write a test tomorrow and see if I can hit something
[14:07] <zyga> your host is 16.04 or something newer?
[14:07] <zyga> the container was 18.04 IIRC
[14:08] <bloodearnest> Chipaca: thanks, testing a new release now, will report back
[14:10] <ackk> zyga, my host is bionic
[14:10] <ackk> zyga, sorry, it's dingo
[14:10] <zyga> ah, perfect.
[14:10] <zyga> so I have the same setup readty
[14:10] <zyga> ok, I'll keep you posted, thank you again
[14:10] <ackk> zyga, thank you for the help
[14:15] <pedronis> Chipaca: mvo: I would do that I think (I find the current state confusing with all the extra checks)
[14:40] <pedronis> mvo: what's the status of this https://github.com/snapcore/core/pull/83 ?
[14:40] <mup> PR core#83: move most of the ubuntu-core config deb into the snap snap build <Created by mvo5> <https://github.com/snapcore/core/pull/83>
[14:41] <mvo> pedronis: I think it never got reviews :/
[14:41] <mvo> pedronis: I can fix the conflict and double check that its up-to-date
[14:42] <pedronis> mvo: reviews from whom? foundations?
[14:42] <pedronis> there's a review by ogra fwiw
[14:42] <mvo> sil2100: silly question, should the split of snapcore/pi3-gadget be on snapcore/pi-gadget (the "unified" one we use in core18)
[14:43] <mvo> sil2100: actually I guess I should ask in the PR
[14:44] <ogra> pedronis, well, more a yoga lesson :) but yeah ... +1 for that one
[14:46]  * cachio lunch
[14:46] <pedronis> zyga: is this being or was fixed: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1785410 ?
[14:46] <mup> Bug #1785410: Can not open Gnome snaps as they are not connected to «gnome platform snap» <amd64> <apport-bug> <bionic> <snapd:Triaged by zyga> <snapd (Ubuntu):Confirmed> <https://launchpad.net/bugs/1785410>
[14:47] <mvo> Chipaca: I missed the earlier bits, what should happen with 7510? we remove "action=download"?
[14:48] <Chipaca> mvo: either in 7510 or before it, we drop action and change snaps to snap
[14:49] <mvo> Chipaca: ok, will do
[14:57] <pedronis> mvo: probably best to call it snap-name,  we might have download by id
[14:58] <mvo> pedronis: great advise
[14:59] <zyga> pedronis: not sure if this instance, many similar issues were
[14:59] <zyga> many issues having similar observable result
[15:00] <zyga> I'll inspect that bug and respond
[15:00] <pedronis> zyga: you made a card specifically about this one, that's why I'm asking: https://trello.com/c/CFDRWAg5/206-content-connection-issues-https-launchpadnet-bugs-1785410
[15:08]  * zyga breaks to rest 
[15:18] <mup> PR snapd#7560 opened: daemon: change /v2/download API to take "snap-name" as input <Created by mvo5> <https://github.com/snapcore/snapd/pull/7560>
[15:20] <mvo> Chipaca: -^
[15:32] <mvo> pedronis: a quick review on 7560 would be great, should be quick and simple
[15:33] <pedronis> mvo: I was already looking at it
[15:35]  * mvo hugs pe
[15:35]  * mvo hugs pedronis 
[16:19] <pedronis> pstolowski: this is done now, right?  https://trello.com/c/2lsJLlXB/316-revisit-fix-per-revision-snap-configuration-handling
[16:22] <pstolowski> pedronis: yes
[16:27] <pedronis> pstolowski: is it in 2.42 or will it be in 2.43 ?
[16:45] <pedronis> Chipaca: this is done now:  https://forum.snapcraft.io/t/expose-a-more-consistent-subset-of-systemds-service-directives/2268 ?
[16:48] <mup> PR snapd#7560 closed: daemon: change /v2/download API to take "snap-name" as input <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7560>
[16:50] <mup> PR snapd#7561 opened: tests: update system-usernames test now that opensuse-15.1 works <Simple 😃> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7561>
[16:50] <Chipaca> pedronis: not yet, not in full
[16:51] <cachio> zyga, hey
[16:52] <cachio> zyga, I am trygin to fix an issue in opensuse
[16:52] <cachio> https://travis-ci.org/snapcore/spread-cron/builds/594484689#L1747
[16:52] <pedronis> Chipaca: when you have a moment could you update it to reflect what is missing? or maybe create a new post if you think it's too old to reopen, in which case remove the tagging on the first, thank you
[16:52] <cachio> it is just happening with the last update
[17:06] <pstolowski> pedronis: 2.42. however it was only a small cleanup (not a fix)
[17:06] <pstolowski> pedronis: i mean, nothing for release notes imho
[17:11] <zyga> cachio: snapd is built without apparmor on opensuse leap
[17:11] <zyga> that test should not run there
[17:11] <zyga> cachio: check exclusion rules please
[17:14] <cachio> zyga, systems: [-ubuntu-core-*, -debian-*]
[17:14] <cachio> it is currently running on leap 15.1
[17:32] <pedronis> pstolowski: thx, just wanted to know which done lane to use
[18:25] <mup> PR snapcraft#2709 closed: incorporate content provider snaps in dependency resolution <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2709>
[19:09] <joedborg> hey zyga, have you got 5 minutes?
[19:28] <mup> PR snapd#7562 opened: tests: chech the apparmor_parser when the file exists on snap-confine test <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7562>
[20:52] <mup> PR snapcraft#2743 opened: meta: warn about command mangling <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2743>
[22:19] <mup> PR snapcraft#2744 opened: project_loader: load build-environment after snapcraft environment <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2744>