[01:15] <mup> PR snapcraft#3314 closed: schema: add missing name item in package-repositories <Created by jc-lab> <Closed by jc-lab> <https://github.com/snapcore/snapcraft/pull/3314>
[05:40] <mborzecki> morning
[06:03] <mborzecki> mvo: hey
[06:08] <mvo> good morning mborzecki
[06:22] <mborzecki> mvo: do you recall vfat corruption when we got uboot.env and UBOOT.ENV in the same fs?
[06:23] <mborzecki> mvo: do you have any tooling to corrupt the vfat from that investigation? i was able to find https://gist.github.com/bboozzoo/e68254507eef4673dd8c6f9b82f65d92 with the hope it can used in #9499
[06:23] <mup> PR #9499: tests: add tests for fsck <Created by zyga> <https://github.com/snapcore/snapd/pull/9499>
[06:31] <mvo> mborzecki: I found some go libraries to directly access fat that can probably be used for this
[06:31] <mvo> mborzecki: let me look at your stuff
[06:31] <mborzecki> mvo: fatcat?
[06:32] <mborzecki> mvo: i have a vague recollection we tried to do something with https://github.com/Gregwar/fatcat/
[06:32] <mvo> mborzecki: I don't remember exactly, let me try to look
[06:35] <mvo> mborzecki: https://github.com/mvo5/go-fs
[06:36] <mborzecki> mvo: interesting, thanks!
[06:38] <mvo> mborzecki: if you want fat32 and lfn support you will need my version but it does not do much, maybe it's actually not that great
[06:39] <zyga> good morning
[06:39] <zyga> mvo: hey!
[06:39] <mvo> good morning zyga
[06:39] <mborzecki> mvo: i was thinking that maybe we could break one of the entries, or a superblock
[06:39] <mvo> mborzecki: +1
[06:39] <mborzecki> zyga: morning, left some comments in 9499
[06:40] <zyga> thanks
[06:40] <zyga> I'm looking at those now
[06:40] <zyga> it was late so maybe a typo crept in?
[06:40] <mvo> mborzecki: we could experiment with corrupting a cluster too, up to you, I follow with popcorn from the sideline :)
[06:41] <zyga> I'll check the offsets first
[06:41] <zyga> mborzecki: I replied
[06:42] <zyga> I really think it's a separate test
[06:42] <zyga> and I think the core16 / core18 situation is unclear, for the reason I outlined in my own comments
[06:42] <zyga> so I'll start by expanding that test not to rely on early-boot kernel messages
[06:42] <zyga> mborzecki: still, we don't fsck on core20 so ... bummer
[06:42] <zyga> I suspect we have to explicitly at the level we are at now
[06:43] <mborzecki> zyga: yeah, i recall ian mentioning something about fsck when he worked on snap-bootstrap
[06:43] <zyga> mborzecki: the good thing is - we know :)
[06:43] <zyga> the bad thing is, it's not good :)
[06:45] <zyga> mborzecki: separately I think we should consider re-mouting vfat as read only
[06:45] <zyga> ubuntu-seed is writable and we mount snaps from there
[06:45] <zyga> it's just waiting to corrupt and die
[06:45] <zyga> we could really keep it read only for most of the runtime of a device
[06:45] <zyga> and only remount to writable when we need to perform a specific operation
[06:45] <zyga> read-only fat doesn't corrupt on power loss
[06:47] <mborzecki> zyga: yeah, it was raised for uc20
[06:56] <zyga> mmm, good
[06:56] <zyga> today is a weird day, no school for kids, wife sick and back from work
[06:56] <zyga> feels like back to quarantine
[07:00] <pstolowski> morning
[07:00] <pstolowski> zyga: hey, i think we forgot to mount boot back after fsck yesterday, that's why core18 failed to refresh :)
[07:01] <zyga> pstolowski: hmmm, could be but I don't recall
[07:01] <zyga> pstolowski: we can look at that again
[07:01] <pstolowski> yes i just checked
[07:01] <zyga> pstolowski: maybe at ... 9:30?
[07:01] <pstolowski> ok
[07:01] <zyga> I need to move back to the office and grab coffee (still upstairs in the kitchen)
[07:01] <pstolowski> that's also an interesting material for spread test
[07:01] <zyga> pstolowski: unmounted boot?
[07:01] <amurray> zyga: hey so I tried your approach but I am not having much success (I think I am not using the right path for the core snap or perhaps am missing something else) - is this about what you had in mind? https://pastebin.ubuntu.com/p/2FXrMhN2Ck/
[07:01] <pstolowski> zyga: yes, but just to provoke failure of refresh
[07:02] <zyga> pstolowski: indeed
[07:02] <zyga> amurray: hey!
[07:02] <zyga> amurray: looking
[07:03] <zyga> amurray: it looks semi okay, I think the current symlink may be a problem, we code defensively and we avoid mounting over symlinks
[07:03] <zyga> what happens in your case?
[07:03] <amurray> zyga: this results in the following: https://pastebin.ubuntu.com/p/mf8fGRW6dr/ so I think it is on the right track
[07:03] <zyga> yeah
[07:03] <zyga> that definitely looks good
[07:03] <amurray> ahh current symlink... do you know how I could know what the actual path would be then to the core snap?
[07:04] <amurray> *or should I say whatever the base snap would be?
[07:04] <zyga> amurray: this _may_ be a little more complex, I came to realize, because we cannot easily capture the base revision
[07:04] <zyga> amurray: so without checking I suspect this is good on paper but breaks because snap-update-ns doesn't want to mount with a source /snap/core/current
[07:05] <zyga> but it also shows a deeper problem, that now the snap has an explicit relationship to the base snap
[07:05] <zyga> amurray: so when the base revision changes, ... what then?
[07:05] <zyga> amurray: normally we have some logic to re-create the mount namespace in that case
[07:05] <amurray> zyga: also do you know how I can debug what snap-update-ns is doing...?
[07:05] <amurray> ah right.. so on base refresh...
[07:05] <zyga> amurray: yes, export SNAP_DEBUG or SNAPD_DEBUG and run it by hand
[07:05] <zyga> you can actually do something simpler
[07:06] <zyga> just set SNAP_CONFINE_DEBUG=1 and snap run --shell docker
[07:06] <zyga> this will give you a lot of details
[07:06] <zyga> it's a bit complex
[07:06] <zyga> because those snaps have services
[07:06] <zyga> it may be better to start with a fake-docker snap that just has the interface
[07:06] <zyga> connect it
[07:06] <zyga> and then see what happens
[07:06] <zyga> otherwise the mount namespace is created by the service app they contain
[07:06] <zyga> and you don't see everything
[07:07] <zyga> by using a new fake snap you can run it by hand with more exact control
[07:07] <zyga> amurray: my advice is to see exactly what happens in snap-update-ns first
[07:07] <zyga> amurray: and then consider two options:
[07:08] <zyga> amurray: special-case /snap/$name/current as source and de-reference that in snap-update-ns
[07:08] <zyga> amurray: explicitly teach snap-update-ns about $BASE_SNAP_NAME $BASE_SNAP_REVISION
[07:08] <zyga> (those variables are made up)
[07:08] <zyga> snap-update-ns could expand those variables and we could use those variables in the mount profile
[07:08] <zyga> then we don't need to special case a path
[07:09] <zyga> and the feature may be useful for other things
[07:09] <zyga> amurray: how does that sound?
[07:11] <amurray> it sounds a lot harder than the snap-confine approach... but if it is the better solution then I'll keep plugging away at it - you said 'see exactly what happens in snap-update-ns' - I am still not sure how to see that.. .I tried running snapd manually with SNAP_DEBUG etc but it didn't seem to show anything relevant
[07:12] <zyga> amurray: don't run snapd with debug
[07:12] <zyga> amurray: run snap run with debug
[07:12] <zyga> snap run invokes snap-confine which invokes snap-update-ns
[07:12] <zyga> if you enable debug there you will see that
[07:13] <zyga> one sec
[07:14] <amurray> hmm https://pastebin.ubuntu.com/p/mDGTJp7GCH/ - I think the 'DEBUG: joined preserved mount namespace docker' means it is not showing what is actually happening
[07:14] <zyga> amurray: zyga@x240:~$ SNAPD_DEBUG=yes snap run gimp
[07:14] <zyga> amurray: exactly, because the service has constructed the mount namespace and you are not seeing that anymore
[07:14] <zyga> amurray: you can work around that without a new snap though
[07:14] <zyga> amurray: just stop all docker services
[07:14] <zyga> amurray: discard the mount namespace with sudo /usr/lib/snapd/snap-discard-ns docker
[07:15] <zyga> and then run
[07:15] <zyga> SNAPD_DEBUG=yes snap run --shell docker
[07:15] <zyga> that will give you the full output
[07:15] <zyga> amurray: snap-confine-based approach is possible too but it would be coarse and unable to handle interfaces
[07:15] <zyga> I'd prefer to avoid that
[07:16] <zyga> and it's a step in the wrong direction in a way, as we want to avoid sharing /etc over time, and have less rigid set of mounts there
[07:17] <amurray> yep that did it (snap-discard-ns) - https://pastebin.ubuntu.com/p/Nx3C4VgCPt/ - ok I gotta run to pick up my daughter but will check back with you later tonight - thanks again for your help :)
[07:19] <zyga> amurray: ack
[07:23]  * zyga improved the fsck test and is running it to see if core16 really fscks or not
[07:23] <zyga> mborzecki: I need a review for https://github.com/snapcore/snapd/pull/9446
[07:23] <mup> PR #9446: overlord,usersession: initial notifications of pending refreshes <Created by zyga> <https://github.com/snapcore/snapd/pull/9446>
[07:23] <zyga> mvo: ^ and perhaps you since ian is off this week?
[07:24] <zyga> mborzecki:     - google:fedora-31-64:tests/main/sudo-env is that due to sudo changes in fedora?
[07:25] <zyga>     - google:arch-linux-64:tests/main/snapshot-cross-revno failed on unrelated pr
[07:26] <zyga> pstolowski: ok, I'll set what I'm doing aside and prep for our session
[07:26] <mvo> zyga: yes, it's on my list but I suck and have not managed to get to it :/ need to do the 360 and an interview prep and then will hpoefully get to it
[07:26] <mvo> zyga: I really want this to land asap
[07:29] <mborzecki> zyga: yes, most likely due to sudo changes
[07:29] <mborzecki> zyga: i'll open a PR with the fix
[07:29] <zyga> mborzecki: superb
[07:29] <zyga> ok
[07:29] <zyga> coffee
[07:29] <zyga> pstolowski: give ma 3 minutes
[07:33] <zyga> pstolowski: just warming stuff up
[07:33] <pstolowski> ok
[07:33] <zyga> pstolowski: do you have the session?
[07:34] <pstolowski> zyga: i've now
[07:45] <zyga> mvo: drat, we don't fsck on core16 or core18
[07:45] <zyga> the test is now improved and fails everywhere
[07:45] <zyga> so .. well :)
[07:46] <mborzecki> zyga: w8, what?
[07:47] <mborzecki> aah, core16 and core18 don't use systemd in initrd, so fsck is not automagic
[07:47] <zyga> mborzecki: yep
[07:47] <zyga> mborzecki: I have a test
[07:49] <pstolowski> rogpeppe: hi
[07:49] <zyga> mborzecki: the previous test was flawed because core16 and core18 did not have systemd forwarding messages from early boot
[07:50] <zyga> anyway, I'll push the test
[07:50] <rogpeppe> pstolowski: hello!
[07:51] <pstolowski> rogpeppe: hey, we have mounted /boot back and managed to refresh core18 and everything looked ok, but unfortunately after reboot it didn't come back, cannot ssh anymore :(
[07:51] <rogpeppe> pstolowski: ok, i'll ask it to be power-cycled
[07:51] <pstolowski> rogpeppe: looks like we need a manual restart of your pi3
[07:52] <rogpeppe> pstolowski: that's what always happens, unfortunately
[07:52] <zyga> mborzecki: pushede
[07:52] <rogpeppe> pstolowski: which is a bit of a shame, because it makes the whole thing somewhat unreliable
[07:52] <pstolowski> rogpeppe: yeah, no denying, hopefully we will get to the bottom of this
[07:53] <zyga> rogpeppe: please let us know when you can power cycle it, we will try to understand what's going on
[07:53] <rogpeppe> pstolowski: i suspect that snappy will sometimes try to reboot itself to update system s/w, and this causes the machine to go down
[07:53] <zyga> mborzecki: please look at my logic in https://github.com/snapcore/snapd/pull/9499
[07:53] <mup> PR #9499: tests: add tests for fsck <Created by zyga> <https://github.com/snapcore/snapd/pull/9499>
[07:54] <zyga> mborzecki: I really would like to know if we fsck
[07:54] <zyga> and the test is wrong
[07:54] <zyga> or if we just don't fsck
[07:54] <zyga> and the test is sadly right
[07:57] <zyga> rogpeppe: pstolowski and me will resume debugging once you give us a note that the power cycle occurred
[08:00] <rogpeppe> zyga: should happen in the next 5 minutes or so; i'll let you know
[08:00] <zyga> rogpeppe: great, thank you
[08:03] <mup> PR snapd#9500 opened: tests/main/sudo-env: snap bin is avaialble on Fedora <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9500>
[08:03] <mborzecki> zyga: ^^
[08:03] <zyga> superb
[08:04] <zyga> +1
[08:08] <rogpeppe> zyga, pstolowski: ok, it's back up now. it took two power-cycles as usual.
[08:08] <zyga> rogpeppe: thank you
[08:08] <zyga> two? as in yank cable twice?
[08:09] <zyga> interesting
[08:09] <zyga> pstolowski: can you connect?
[08:10] <pstolowski> checking
[08:10] <pstolowski> zyga: yes, i can
[08:10] <zyga> pstolowski: ok, let's examine the boot partition again
[08:10] <zyga> and maybe snap changes
[08:10] <zyga> and journal?
[08:10] <zyga> shall we get back to the HO?
[08:11] <pstolowski> zyga: yes, give me 5
[08:11] <zyga> sure
[08:15] <pstolowski> ok rdy
[08:18] <zyga> we were just talking with pawel that maybe the second reboot is because we fail to do the initial reboot at all
[08:19] <mborzecki> mvo: per monday's discussion i've updated https://github.com/snapcore/snapd/pull/9420 to use 384MB for UC20
[08:19] <mup> PR #9420: tests/nested/manual/minimal-smoke: use 384MB of RAM for nested UC20 <Run nested> <UC20> <⛔ Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9420>
[08:31] <mvo> mborzecki: thanks! (in a meeting)
[08:33] <zyga> rogpeppe: does the device reboot correctly if you just "sudo reboot"?
[08:33] <zyga> rogpeppe: or is it also going into this weird state that requires more than one reboot to complete?
[08:34] <rogpeppe> zyga: i don't know... i'd presume not, otherwise it wouldn't have gone down last night
[08:34] <rogpeppe> zyga: shall we try?
[08:34] <zyga> mmm
[08:34] <zyga> we're still coming up with options, we may decide to try that to learn more
[08:34] <zyga> we'll let you know for sure
[08:34] <rogpeppe> zyga: ta
[08:35] <zyga> rogpeppe: no chance of a serial log? :D
[08:36] <rogpeppe> zyga: nope, i'm afraid not. i've never worked out how to get serial output from the pi.
[08:37] <rogpeppe> zyga: don't you have to solder something onto the board or something like that?
[08:40] <zyga> rogpeppe: no, just a pair of wires and a computer with usb port to capture the log
[08:41] <rogpeppe> zyga: presumably you need some kind of connector for the wires, and a serial port adaptor
[08:42] <rogpeppe> zyga: not sure my dad has either of those things around currently (actually, he _may_ have a serial port adaptor, but i'm not sure i want to get him to do all this remotely :) )
[08:42] <zyga> rogpeppe: yeah, those are super common on ebay and amazon and go for a few pounds
[08:42] <zyga> I was more of a joking, it'd be ideal for debugging
[08:42] <zyga> we're eliminating possibilities
[08:43] <rogpeppe> zyga: yeah, i've thought that before, but thought it would be too much hassle
[08:51] <zyga> rogpeppe: ok, we'd like to reboot the device without any changes
[08:51] <zyga> we have some ideas but we'd be much faster if we can do some reboots
[08:51] <zyga> without going through refresh
[08:51] <zyga> rogpeppe: is it okay if we reboot the device now?
[08:51] <rogpeppe> zyga: please do
[08:52] <zyga> ok
[08:52] <rogpeppe> zyga: let me know if you need it power-cycling again
[08:52] <rogpeppe> zyga: although i'm in a meeting in 8 minutes
[08:53] <zyga> ok
[08:53] <zyga> we are back
[08:53] <zyga> thank you!
[09:00] <zyga> rogpeppe: it rebooted successfully
[09:00] <rogpeppe> zyga: yay!
[09:01] <mborzecki> quick errand, brb
[09:21] <mborzecki> re
[09:42] <mvo> zyga: \o/ you rock!
[09:42] <zyga> mvo: we're not done, it's not over yet
[09:42] <zyga> and it's pawel and maciek :)
[09:57]  * mvo hugs mborzecki and pstolowski 
[09:57] <pstolowski> mvo: yeah it just survived manual reboot, but not reboot after refresh; we're seeing 'rollback across reboot'
[10:17] <mvo> quick question: do we have an example gadget with cloud init configuration?
[10:18]  * zyga doesn't remember one
[10:18] <mvo> that's fine, I will just create one I think
[10:21] <zyga-mbp> not a good store day
[11:13] <zyga> rogpeppe: we did lots of analysis but ultimately reboot just failed
[11:14] <zyga> rogpeppe: if you can power cycle (once for now), that would be great
[11:14] <rogpeppe> zyga: ok :\
[11:14] <zyga> let us know when ready
[11:18] <zyga-mbp> rogpeppe: so, it would be amazing if you could eventually get this
[11:18] <zyga-mbp> https://www.amazon.co.uk/UART-CP2102-Module-Serial-Converter/dp/B00AFRXKFU/ref=sr_1_7?dchild=1&keywords=USB+TTL+serial&qid=1602674287&sr=8-7
[11:19] <zyga-mbp> rogpeppe: let us know when the 1st power-cycle had occurred
[11:19] <zyga-mbp> if you already did that you can go ahead with the second one (2nd power cycle)
[11:19] <zyga-mbp> 1st power cycle did not (as expected, though unclear why) recover the device
[11:22] <zyga> ogra: do you have the permission to download pi-kernel revno 44?
[11:45] <zyga-mbp> mvo: do you have access to core18 snap?
[11:45] <zyga-mbp> I could use revision 1076 with assertions
[11:45] <zyga-mbp> similarly to pi-kernel at revision 44
[11:45] <zyga-mbp> and pi gadget revision 17
[11:59] <ogra> zyga-mbp, let me try ...
[11:59] <zyga-mbp> thank you!
[12:00] <ogra> $ snap download --revision=44 pi-kernel
[12:00] <ogra> Fetching snap "pi-kernel"
[12:00] <ogra> error: cannot download snap "pi-kernel": Access by specifying a revision is not allowed for this Snap.
[12:00] <ogra> nope
[12:00] <zyga-mbp> oh well
[12:00] <ogra> i fear only the kernel team can ...
[12:01] <ogra> (i can download the current revision for each track/channel but not specify --revision)
[12:01] <zyga-mbp> yeah, same here
[12:02] <ogra> also 44 is pretty old .. we're at 200 currently
[12:02] <zyga-mbp> I know, we are trying to reproduce a peculiar failure
[12:02] <ogra> (for armhf that is)
[12:27] <mborzecki> cmatsuoka: hi, can you take a look at https://github.com/snapcore/snapd/pull/9474 ? it's been +1'ed by pedronis
[12:27] <mup> PR #9474: boot, overlord/devicestate: list trusted and managed assets upfront <Run nested> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9474>
[12:28] <cmatsuoka> mborzecki: already looking, was about to +1 it
[12:29] <mborzecki> cmatsuoka: cool thanks!
[12:29] <mborzecki> cmatsuoka: fwiw https://github.com/snapcore/snapd/pull/9443 could use your review too :)
[12:29] <mup> PR #9443: gadget, gadget/install: support for ubuntu-save, create one during install if needed <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9443>
[12:36] <cmatsuoka> mborzecki: ah yes, I started reviewing it a few days ago and didn't finish it, resuming now
[12:44] <mborzecki> cmatsuoka: thank you
[12:53] <cmatsuoka> mborzecki: done
[12:54] <mborzecki> cmatsuoka: yay, now if only testing was green
[12:55] <jibel> Hi, I cannot launch any snap on groovy. When I launch a command from the terminal, it waits for a moment then I get this error:
[12:55] <jibel> $ matterhorn
[12:55] <jibel> cannot stat /var/lib/snapd/seccomp/bpf/snap.matterhorn.matterhorn.bin: No such file or directory
[12:56] <jibel> is there a knows issue?
[12:56] <jibel> $ snap version
[12:56] <jibel> snap    2.47.1+20.10
[12:56] <jibel> snapd   2.47.1+20.10
[12:56] <jibel> series  16
[12:56] <jibel> ubuntu  20.10
[12:56] <jibel> kernel  5.8.0-22-generic
[12:58] <zyga-mbp> jibel hi
[12:58] <zyga-mbp> hmm
[12:58] <mborzecki> interesting
[12:58] <zyga-mbp> I assume the referenced file does not exist
[12:58] <zyga-mbp> can you run snap list without failure?
[12:59] <jibel> yes
[12:59] <jibel> no failure
[12:59] <mborzecki> jibel: yeah, can you also do `ls -l /var/lib/snap/seccomp/bpf` ?
[12:59] <zyga-mbp> and if you run "snap changes" do you see any errors?
[12:59] <jibel> and the snaps I cannot launc hare listed
[13:00] <jibel> ah
[13:00] <jibel> 65   Error   today at 13:04 CEST  today at 13:06 CEST  Auto-refresh snaps "snapd"
[13:00] <jibel> and the error
[13:00] <jibel> 2020-10-14T13:04:37+02:00 ERROR cannot compile /var/lib/snapd/seccomp/bpf/snap.dl-ubuntu-test-iso.dl-ubuntu-test-iso.src: fork/exec /snap/snapd/8790/usr/lib/snapd/snap-seccomp: no such file or directory
[13:01] <jibel> log of change 65 https://paste.ubuntu.com/p/MKWRHhZpX4/
[13:01] <jibel> upgrade of snapd didnt go well apparently
[13:02] <mvo> zyga-mbp: I can give you core18 r1076 - give me 1min
[13:02] <zyga-mbp> mvo thank you
[13:02] <zyga-mbp> jibel we have a standup call now
[13:02] <zyga-mbp> I'll get back to you after that
[13:02] <jibel> np
[13:05] <jibel> zyga-mbp, for some reason auto-refresh of snapd failed, I re-refreshed snapd manually and it fixed $world
[13:05] <jibel> weird
[13:08] <zyga-mbp> jibel could you look for system logs from that moment
[13:08] <zyga-mbp> perhaps there are clues there?
[13:19] <jibel> here it is https://paste.ubuntu.com/p/ZSRcqbsVCM/
[13:28] <zyga-mbp> jibel thank you
[13:28] <zyga-mbp> jibel very interesting
[13:38] <zyga-mbp> jibel let me read the log quickly
[13:38] <zyga-mbp> jibel can you ls -ld /snap/snapd/8790/usr/lib/snapd/snap-seccomp
[13:38] <zyga-mbp> and report a bug with this log, snap changes and snap tasks NNN where NNN is the change that failed
[13:38] <zyga-mbp> I need to get my dog to the vet in a moment and I don't want to lose a chance to see the state your system was in right now
[13:40] <zyga-mbp> jibel: ^
[13:40] <jibel> zyga-mbp, version 8790 has been removed from the system, I've only 9279 and 9607
[13:40] <zyga-mbp> that explains a lot
[13:40] <zyga-mbp> thank you!
[13:40] <jibel> $ ls -ld /snap/snapd/*/usr/lib/snapd/snap-seccomp
[13:40] <zyga-mbp> so snap changes
[13:41] <jibel> -rwxr-xr-x 1 root root 2306928 sept.  4 18:33 /snap/snapd/9279/usr/lib/snapd/snap-seccomp
[13:41] <jibel> -rwxr-xr-x 1 root root 2306928 sept. 30 06:59 /snap/snapd/9607/usr/lib/snapd/snap-seccomp
[13:41] <jibel> -rwxr-xr-x 1 root root 2306928 sept. 30 06:59 /snap/snapd/current/usr/lib/snapd/snap-seccomp
[13:41] <zyga-mbp> and snap tasks for the last few changes that happened
[13:41] <zyga-mbp> I really need to go now
[13:41] <jibel> I'll report a bug
[13:41] <zyga-mbp> mborzecki, pstolowski: ^ perhaps something you can pick up to the extent that the relevant data is collected in a bug
[13:43]  * zyga-mbp goes 
[13:46] <mborzecki> jibel: can you find the pid of `snapd` and then run `sudo ls -l /proc/<pid>/exe` ?
[13:47] <cachio> mvo, could you find the error for sbuild?
[13:51] <mborzecki> cachio: got log of that failure?
[13:51] <cachio> mborzecki, which failure?
[13:52] <cachio> the sbuild on?
[13:52] <mborzecki> cachio: yes, the sbuild one
[13:53] <cachio> mborzecki, https://paste.ubuntu.com/p/f9MJNgc6r9/
[13:54] <jibel> mborzecki, lrwxrwxrwx 1 root root 0 oct.  14 15:02 /proc/70906/exe -> /usr/lib/snapd/snapd
[13:54] <jibel> mborzecki, zyga-mbp bug 1899794
[13:54] <mup> Bug #1899794: snapd error during refresh <snapd (Ubuntu):New> <https://launchpad.net/bugs/1899794>
[13:58] <mborzecki> jibel: can you attach the output of journalctl -u snapd.failure.service too?
[14:00] <jibel> done
[14:03] <mborzecki> jibel: thanks, this looks very interesting, can you also grab the output of `journalctl -u snapd.service` from before oct 12?
[14:05] <mborzecki> jibel: looks like something happend on oct 12 ~7:03 and the failure handler got started, so maybe a complete log of the snapd.service would be even better
[14:09] <jibel> mborzecki, attached
[14:09] <mborzecki> jibel: thanks
[14:09] <jibel> I'm wondering if it could be related to bug 1871538 where systemd brings the entire session down
[14:09] <mup> Bug #1871538: dbus timeout-ed during an upgrade, taking services down including gdm <amd64> <apport-bug> <champagne> <focal> <id-5e988df7fb344884f67bc04f> <rls-ff-incoming> <systemd:New> <accountsservice (Ubuntu):Invalid> <dbus (Ubuntu):Incomplete> <gnome-shell (Ubuntu):Invalid> <accountsservice
[14:09] <mup> (Ubuntu Focal):Invalid> <dbus (Ubuntu Focal):Incomplete> <gnome-shell (Ubuntu Focal):Invalid> <https://launchpad.net/bugs/1871538>
[14:17] <mborzecki> jibel: not sure, looks like there were 2 instances of snapd service running one point
[14:30] <mborzecki> jibel: was there a package update sometime ~7:00 on oct 12?
[14:32] <jibel> mborzecki, yes among other there was snapd:amd64 (2.46.1+20.10, 2.47.1+20.10)
[14:32] <mborzecki> jibel: around that time?
[14:33] <jibel> from 07:01:25 to 07:03:53
[14:33] <mborzecki> jibel: cool, thanks
[14:44] <mborzecki> jibel: ok, added more comments to the bug, a nice one
[14:45] <mborzecki> mvo: pedronis: i've added some comments to https://bugs.launchpad.net/snapd/+bug/1899794 looks like a real problem
[14:45] <mup> Bug #1899794: Error during refresh of snapd leads to unusable system <snapd:Triaged> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1899794>
[14:46] <mborzecki> errands
[14:49] <jibel> mborzecki, ta
[15:23] <pedronis> mvo: I did a pass in #9469
[15:23] <mup> PR #9469: snapshots: import of a snapshot set <Created by mvo5> <https://github.com/snapcore/snapd/pull/9469>
[15:25] <pedronis> mborzecki: thx, snap-failure /revert logic needs improvements
[15:35] <mvo> pedronis: awesome, thank you!
[15:36] <cachio>  /me lunch
[15:36]  * cachio lunch
[16:30] <mup> PR snapd#9501 opened: [RFC] wrappers: do not error out on read-only /etc/dbus-1/session.d filesystem on core18 <Created by stolowski> <https://github.com/snapcore/snapd/pull/9501>
[17:31] <mvo> cachio: I did not even look at the sbiuld error, so sorry! had meeting, an interview and 360
[17:32] <cachio> mvo, np, I'll take a look
[17:33] <cachio> mvo, is it failing to build https://paste.ubuntu.com/p/hdxvkmrRVG/
[17:33] <cachio> mvo, is it any way to show more info about what failed?
[17:34] <mvo> cachio: yeah, the error as is is really not helpful
[17:34] <mvo> cachio: there should be maybe a sbuild.log with more info?
[17:34] <cachio> mvo, nice, thanks
[17:41] <zyga> re
[17:50] <mvo> cachio: if you don't find anything, please remind me tomorrow, my day should be a bit less crazy tomorrow :)
[17:50] <cachio> mvo, sure, thanks!
[17:53] <zyga> sorry, this was a long visit
[17:53] <zyga> I'll grab some food and do reviews for cachio
[17:53] <zyga> and then try to install the kernel and core mvo shared
[18:03] <mvo> zyga \o/
[18:04] <zyga> :)
[18:13] <mvo> pedronis: I addressed the snapshot import feedback, will ask pawel tomorrow
[18:13] <mvo> pedronis: for a second review
[18:13] <mvo> pedronis: will call it a day now
[18:23] <mup> PR snapcraft#3317 opened: plugin handler: properly handle snapcraftctl errors <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3317>
[18:46]  * cachio -> kinesiologist
[19:38]  * zyga picks up zyga:tweak/ignore-running aka https://github.com/snapcore/snapd/pull/9406
[19:38] <mup> PR #9406: many: allow ignoring running apps for specific request <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9406>
[19:53] <mup> PR snapcraft#3318 opened: plugin handler: set -x for scriptlets <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3318>
[20:36] <mup> PR snapd#9502 opened: tests: more output for sbuild test <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9502>
[20:39]  * zyga made some progress and calls it a day
[20:49] <zyga> ok one more patch
[20:49] <zyga> pushed some tests to https://github.com/snapcore/snapd/pull/9406
[20:49] <mup> PR #9406: many: allow ignoring running apps for specific request <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9406>
[21:03] <mup> PR snapcraft#3319 opened: [RFC] plugin handler: use bash with additional error checking for core20 scripts <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3319>
[21:31] <mup> PR snapd#9503 opened: tests: use tests.backup tool <Run nested> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9503>