mup | PR snapcraft#3314 closed: schema: add missing name item in package-repositories <Created by jc-lab> <Closed by jc-lab> <https://github.com/snapcore/snapcraft/pull/3314> | 01:15 |
---|---|---|
mborzecki | morning | 05:40 |
mborzecki | mvo: hey | 06:03 |
mvo | good morning mborzecki | 06:08 |
mborzecki | mvo: do you recall vfat corruption when we got uboot.env and UBOOT.ENV in the same fs? | 06:22 |
mborzecki | mvo: do you have any tooling to corrupt the vfat from that investigation? i was able to find https://gist.github.com/bboozzoo/e68254507eef4673dd8c6f9b82f65d92 with the hope it can used in #9499 | 06:23 |
mup | PR #9499: tests: add tests for fsck <Created by zyga> <https://github.com/snapcore/snapd/pull/9499> | 06:23 |
mvo | mborzecki: I found some go libraries to directly access fat that can probably be used for this | 06:31 |
mvo | mborzecki: let me look at your stuff | 06:31 |
mborzecki | mvo: fatcat? | 06:31 |
mborzecki | mvo: i have a vague recollection we tried to do something with https://github.com/Gregwar/fatcat/ | 06:32 |
mvo | mborzecki: I don't remember exactly, let me try to look | 06:32 |
mvo | mborzecki: https://github.com/mvo5/go-fs | 06:35 |
mborzecki | mvo: interesting, thanks! | 06:36 |
mvo | mborzecki: if you want fat32 and lfn support you will need my version but it does not do much, maybe it's actually not that great | 06:38 |
zyga | good morning | 06:39 |
zyga | mvo: hey! | 06:39 |
mvo | good morning zyga | 06:39 |
mborzecki | mvo: i was thinking that maybe we could break one of the entries, or a superblock | 06:39 |
mvo | mborzecki: +1 | 06:39 |
mborzecki | zyga: morning, left some comments in 9499 | 06:39 |
zyga | thanks | 06:40 |
zyga | I'm looking at those now | 06:40 |
zyga | it was late so maybe a typo crept in? | 06:40 |
mvo | mborzecki: we could experiment with corrupting a cluster too, up to you, I follow with popcorn from the sideline :) | 06:40 |
zyga | I'll check the offsets first | 06:41 |
zyga | mborzecki: I replied | 06:41 |
zyga | I really think it's a separate test | 06:42 |
zyga | and I think the core16 / core18 situation is unclear, for the reason I outlined in my own comments | 06:42 |
zyga | so I'll start by expanding that test not to rely on early-boot kernel messages | 06:42 |
zyga | mborzecki: still, we don't fsck on core20 so ... bummer | 06:42 |
zyga | I suspect we have to explicitly at the level we are at now | 06:42 |
mborzecki | zyga: yeah, i recall ian mentioning something about fsck when he worked on snap-bootstrap | 06:43 |
zyga | mborzecki: the good thing is - we know :) | 06:43 |
zyga | the bad thing is, it's not good :) | 06:43 |
zyga | mborzecki: separately I think we should consider re-mouting vfat as read only | 06:45 |
zyga | ubuntu-seed is writable and we mount snaps from there | 06:45 |
zyga | it's just waiting to corrupt and die | 06:45 |
zyga | we could really keep it read only for most of the runtime of a device | 06:45 |
zyga | and only remount to writable when we need to perform a specific operation | 06:45 |
zyga | read-only fat doesn't corrupt on power loss | 06:45 |
mborzecki | zyga: yeah, it was raised for uc20 | 06:47 |
zyga | mmm, good | 06:56 |
zyga | today is a weird day, no school for kids, wife sick and back from work | 06:56 |
zyga | feels like back to quarantine | 06:56 |
pstolowski | morning | 07:00 |
pstolowski | zyga: hey, i think we forgot to mount boot back after fsck yesterday, that's why core18 failed to refresh :) | 07:00 |
zyga | pstolowski: hmmm, could be but I don't recall | 07:01 |
zyga | pstolowski: we can look at that again | 07:01 |
pstolowski | yes i just checked | 07:01 |
zyga | pstolowski: maybe at ... 9:30? | 07:01 |
pstolowski | ok | 07:01 |
zyga | I need to move back to the office and grab coffee (still upstairs in the kitchen) | 07:01 |
pstolowski | that's also an interesting material for spread test | 07:01 |
zyga | pstolowski: unmounted boot? | 07:01 |
amurray | zyga: hey so I tried your approach but I am not having much success (I think I am not using the right path for the core snap or perhaps am missing something else) - is this about what you had in mind? https://pastebin.ubuntu.com/p/2FXrMhN2Ck/ | 07:01 |
pstolowski | zyga: yes, but just to provoke failure of refresh | 07:01 |
zyga | pstolowski: indeed | 07:02 |
zyga | amurray: hey! | 07:02 |
zyga | amurray: looking | 07:02 |
zyga | amurray: it looks semi okay, I think the current symlink may be a problem, we code defensively and we avoid mounting over symlinks | 07:03 |
zyga | what happens in your case? | 07:03 |
amurray | zyga: this results in the following: https://pastebin.ubuntu.com/p/mf8fGRW6dr/ so I think it is on the right track | 07:03 |
zyga | yeah | 07:03 |
zyga | that definitely looks good | 07:03 |
amurray | ahh current symlink... do you know how I could know what the actual path would be then to the core snap? | 07:03 |
amurray | *or should I say whatever the base snap would be? | 07:04 |
zyga | amurray: this _may_ be a little more complex, I came to realize, because we cannot easily capture the base revision | 07:04 |
zyga | amurray: so without checking I suspect this is good on paper but breaks because snap-update-ns doesn't want to mount with a source /snap/core/current | 07:04 |
zyga | but it also shows a deeper problem, that now the snap has an explicit relationship to the base snap | 07:05 |
zyga | amurray: so when the base revision changes, ... what then? | 07:05 |
zyga | amurray: normally we have some logic to re-create the mount namespace in that case | 07:05 |
amurray | zyga: also do you know how I can debug what snap-update-ns is doing...? | 07:05 |
amurray | ah right.. so on base refresh... | 07:05 |
zyga | amurray: yes, export SNAP_DEBUG or SNAPD_DEBUG and run it by hand | 07:05 |
zyga | you can actually do something simpler | 07:05 |
zyga | just set SNAP_CONFINE_DEBUG=1 and snap run --shell docker | 07:06 |
zyga | this will give you a lot of details | 07:06 |
zyga | it's a bit complex | 07:06 |
zyga | because those snaps have services | 07:06 |
zyga | it may be better to start with a fake-docker snap that just has the interface | 07:06 |
zyga | connect it | 07:06 |
zyga | and then see what happens | 07:06 |
zyga | otherwise the mount namespace is created by the service app they contain | 07:06 |
zyga | and you don't see everything | 07:06 |
zyga | by using a new fake snap you can run it by hand with more exact control | 07:07 |
zyga | amurray: my advice is to see exactly what happens in snap-update-ns first | 07:07 |
zyga | amurray: and then consider two options: | 07:07 |
zyga | amurray: special-case /snap/$name/current as source and de-reference that in snap-update-ns | 07:08 |
zyga | amurray: explicitly teach snap-update-ns about $BASE_SNAP_NAME $BASE_SNAP_REVISION | 07:08 |
zyga | (those variables are made up) | 07:08 |
zyga | snap-update-ns could expand those variables and we could use those variables in the mount profile | 07:08 |
zyga | then we don't need to special case a path | 07:08 |
zyga | and the feature may be useful for other things | 07:09 |
zyga | amurray: how does that sound? | 07:09 |
amurray | it sounds a lot harder than the snap-confine approach... but if it is the better solution then I'll keep plugging away at it - you said 'see exactly what happens in snap-update-ns' - I am still not sure how to see that.. .I tried running snapd manually with SNAP_DEBUG etc but it didn't seem to show anything relevant | 07:11 |
zyga | amurray: don't run snapd with debug | 07:12 |
zyga | amurray: run snap run with debug | 07:12 |
zyga | snap run invokes snap-confine which invokes snap-update-ns | 07:12 |
zyga | if you enable debug there you will see that | 07:12 |
zyga | one sec | 07:13 |
amurray | hmm https://pastebin.ubuntu.com/p/mDGTJp7GCH/ - I think the 'DEBUG: joined preserved mount namespace docker' means it is not showing what is actually happening | 07:14 |
zyga | amurray: zyga@x240:~$ SNAPD_DEBUG=yes snap run gimp | 07:14 |
zyga | amurray: exactly, because the service has constructed the mount namespace and you are not seeing that anymore | 07:14 |
zyga | amurray: you can work around that without a new snap though | 07:14 |
zyga | amurray: just stop all docker services | 07:14 |
zyga | amurray: discard the mount namespace with sudo /usr/lib/snapd/snap-discard-ns docker | 07:14 |
zyga | and then run | 07:15 |
zyga | SNAPD_DEBUG=yes snap run --shell docker | 07:15 |
zyga | that will give you the full output | 07:15 |
zyga | amurray: snap-confine-based approach is possible too but it would be coarse and unable to handle interfaces | 07:15 |
zyga | I'd prefer to avoid that | 07:15 |
zyga | and it's a step in the wrong direction in a way, as we want to avoid sharing /etc over time, and have less rigid set of mounts there | 07:16 |
amurray | yep that did it (snap-discard-ns) - https://pastebin.ubuntu.com/p/Nx3C4VgCPt/ - ok I gotta run to pick up my daughter but will check back with you later tonight - thanks again for your help :) | 07:17 |
zyga | amurray: ack | 07:19 |
* zyga improved the fsck test and is running it to see if core16 really fscks or not | 07:23 | |
zyga | mborzecki: I need a review for https://github.com/snapcore/snapd/pull/9446 | 07:23 |
mup | PR #9446: overlord,usersession: initial notifications of pending refreshes <Created by zyga> <https://github.com/snapcore/snapd/pull/9446> | 07:23 |
zyga | mvo: ^ and perhaps you since ian is off this week? | 07:23 |
zyga | mborzecki: - google:fedora-31-64:tests/main/sudo-env is that due to sudo changes in fedora? | 07:24 |
zyga | - google:arch-linux-64:tests/main/snapshot-cross-revno failed on unrelated pr | 07:25 |
zyga | pstolowski: ok, I'll set what I'm doing aside and prep for our session | 07:26 |
mvo | zyga: yes, it's on my list but I suck and have not managed to get to it :/ need to do the 360 and an interview prep and then will hpoefully get to it | 07:26 |
mvo | zyga: I really want this to land asap | 07:26 |
mborzecki | zyga: yes, most likely due to sudo changes | 07:29 |
mborzecki | zyga: i'll open a PR with the fix | 07:29 |
zyga | mborzecki: superb | 07:29 |
zyga | ok | 07:29 |
zyga | coffee | 07:29 |
zyga | pstolowski: give ma 3 minutes | 07:29 |
zyga | pstolowski: just warming stuff up | 07:33 |
pstolowski | ok | 07:33 |
zyga | pstolowski: do you have the session? | 07:33 |
pstolowski | zyga: i've now | 07:34 |
zyga | mvo: drat, we don't fsck on core16 or core18 | 07:45 |
zyga | the test is now improved and fails everywhere | 07:45 |
zyga | so .. well :) | 07:45 |
mborzecki | zyga: w8, what? | 07:46 |
mborzecki | aah, core16 and core18 don't use systemd in initrd, so fsck is not automagic | 07:47 |
zyga | mborzecki: yep | 07:47 |
zyga | mborzecki: I have a test | 07:47 |
pstolowski | rogpeppe: hi | 07:49 |
zyga | mborzecki: the previous test was flawed because core16 and core18 did not have systemd forwarding messages from early boot | 07:49 |
zyga | anyway, I'll push the test | 07:50 |
rogpeppe | pstolowski: hello! | 07:50 |
pstolowski | rogpeppe: hey, we have mounted /boot back and managed to refresh core18 and everything looked ok, but unfortunately after reboot it didn't come back, cannot ssh anymore :( | 07:51 |
rogpeppe | pstolowski: ok, i'll ask it to be power-cycled | 07:51 |
pstolowski | rogpeppe: looks like we need a manual restart of your pi3 | 07:51 |
rogpeppe | pstolowski: that's what always happens, unfortunately | 07:52 |
zyga | mborzecki: pushede | 07:52 |
rogpeppe | pstolowski: which is a bit of a shame, because it makes the whole thing somewhat unreliable | 07:52 |
pstolowski | rogpeppe: yeah, no denying, hopefully we will get to the bottom of this | 07:52 |
zyga | rogpeppe: please let us know when you can power cycle it, we will try to understand what's going on | 07:53 |
rogpeppe | pstolowski: i suspect that snappy will sometimes try to reboot itself to update system s/w, and this causes the machine to go down | 07:53 |
zyga | mborzecki: please look at my logic in https://github.com/snapcore/snapd/pull/9499 | 07:53 |
mup | PR #9499: tests: add tests for fsck <Created by zyga> <https://github.com/snapcore/snapd/pull/9499> | 07:53 |
zyga | mborzecki: I really would like to know if we fsck | 07:54 |
zyga | and the test is wrong | 07:54 |
zyga | or if we just don't fsck | 07:54 |
zyga | and the test is sadly right | 07:54 |
zyga | rogpeppe: pstolowski and me will resume debugging once you give us a note that the power cycle occurred | 07:57 |
rogpeppe | zyga: should happen in the next 5 minutes or so; i'll let you know | 08:00 |
zyga | rogpeppe: great, thank you | 08:00 |
mup | PR snapd#9500 opened: tests/main/sudo-env: snap bin is avaialble on Fedora <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9500> | 08:03 |
mborzecki | zyga: ^^ | 08:03 |
zyga | superb | 08:03 |
zyga | +1 | 08:04 |
rogpeppe | zyga, pstolowski: ok, it's back up now. it took two power-cycles as usual. | 08:08 |
zyga | rogpeppe: thank you | 08:08 |
zyga | two? as in yank cable twice? | 08:08 |
zyga | interesting | 08:09 |
zyga | pstolowski: can you connect? | 08:09 |
pstolowski | checking | 08:10 |
pstolowski | zyga: yes, i can | 08:10 |
zyga | pstolowski: ok, let's examine the boot partition again | 08:10 |
zyga | and maybe snap changes | 08:10 |
zyga | and journal? | 08:10 |
zyga | shall we get back to the HO? | 08:10 |
pstolowski | zyga: yes, give me 5 | 08:11 |
zyga | sure | 08:11 |
pstolowski | ok rdy | 08:15 |
zyga | we were just talking with pawel that maybe the second reboot is because we fail to do the initial reboot at all | 08:18 |
mborzecki | mvo: per monday's discussion i've updated https://github.com/snapcore/snapd/pull/9420 to use 384MB for UC20 | 08:19 |
mup | PR #9420: tests/nested/manual/minimal-smoke: use 384MB of RAM for nested UC20 <Run nested> <UC20> <â›” Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9420> | 08:19 |
mvo | mborzecki: thanks! (in a meeting) | 08:31 |
zyga | rogpeppe: does the device reboot correctly if you just "sudo reboot"? | 08:33 |
zyga | rogpeppe: or is it also going into this weird state that requires more than one reboot to complete? | 08:33 |
rogpeppe | zyga: i don't know... i'd presume not, otherwise it wouldn't have gone down last night | 08:34 |
rogpeppe | zyga: shall we try? | 08:34 |
zyga | mmm | 08:34 |
zyga | we're still coming up with options, we may decide to try that to learn more | 08:34 |
zyga | we'll let you know for sure | 08:34 |
rogpeppe | zyga: ta | 08:34 |
zyga | rogpeppe: no chance of a serial log? :D | 08:35 |
rogpeppe | zyga: nope, i'm afraid not. i've never worked out how to get serial output from the pi. | 08:36 |
rogpeppe | zyga: don't you have to solder something onto the board or something like that? | 08:37 |
zyga | rogpeppe: no, just a pair of wires and a computer with usb port to capture the log | 08:40 |
rogpeppe | zyga: presumably you need some kind of connector for the wires, and a serial port adaptor | 08:41 |
rogpeppe | zyga: not sure my dad has either of those things around currently (actually, he _may_ have a serial port adaptor, but i'm not sure i want to get him to do all this remotely :) ) | 08:42 |
zyga | rogpeppe: yeah, those are super common on ebay and amazon and go for a few pounds | 08:42 |
zyga | I was more of a joking, it'd be ideal for debugging | 08:42 |
zyga | we're eliminating possibilities | 08:42 |
rogpeppe | zyga: yeah, i've thought that before, but thought it would be too much hassle | 08:43 |
zyga | rogpeppe: ok, we'd like to reboot the device without any changes | 08:51 |
zyga | we have some ideas but we'd be much faster if we can do some reboots | 08:51 |
zyga | without going through refresh | 08:51 |
zyga | rogpeppe: is it okay if we reboot the device now? | 08:51 |
rogpeppe | zyga: please do | 08:51 |
zyga | ok | 08:52 |
rogpeppe | zyga: let me know if you need it power-cycling again | 08:52 |
rogpeppe | zyga: although i'm in a meeting in 8 minutes | 08:52 |
zyga | ok | 08:53 |
zyga | we are back | 08:53 |
zyga | thank you! | 08:53 |
zyga | rogpeppe: it rebooted successfully | 09:00 |
rogpeppe | zyga: yay! | 09:00 |
mborzecki | quick errand, brb | 09:01 |
mborzecki | re | 09:21 |
mvo | zyga: \o/ you rock! | 09:42 |
zyga | mvo: we're not done, it's not over yet | 09:42 |
zyga | and it's pawel and maciek :) | 09:42 |
* mvo hugs mborzecki and pstolowski | 09:57 | |
pstolowski | mvo: yeah it just survived manual reboot, but not reboot after refresh; we're seeing 'rollback across reboot' | 09:57 |
mvo | quick question: do we have an example gadget with cloud init configuration? | 10:17 |
* zyga doesn't remember one | 10:18 | |
mvo | that's fine, I will just create one I think | 10:18 |
zyga-mbp | not a good store day | 10:21 |
zyga | rogpeppe: we did lots of analysis but ultimately reboot just failed | 11:13 |
zyga | rogpeppe: if you can power cycle (once for now), that would be great | 11:14 |
rogpeppe | zyga: ok :\ | 11:14 |
zyga | let us know when ready | 11:14 |
zyga-mbp | rogpeppe: so, it would be amazing if you could eventually get this | 11:18 |
zyga-mbp | https://www.amazon.co.uk/UART-CP2102-Module-Serial-Converter/dp/B00AFRXKFU/ref=sr_1_7?dchild=1&keywords=USB+TTL+serial&qid=1602674287&sr=8-7 | 11:18 |
zyga-mbp | rogpeppe: let us know when the 1st power-cycle had occurred | 11:19 |
zyga-mbp | if you already did that you can go ahead with the second one (2nd power cycle) | 11:19 |
zyga-mbp | 1st power cycle did not (as expected, though unclear why) recover the device | 11:19 |
zyga | ogra: do you have the permission to download pi-kernel revno 44? | 11:22 |
zyga-mbp | mvo: do you have access to core18 snap? | 11:45 |
zyga-mbp | I could use revision 1076 with assertions | 11:45 |
zyga-mbp | similarly to pi-kernel at revision 44 | 11:45 |
zyga-mbp | and pi gadget revision 17 | 11:45 |
ogra | zyga-mbp, let me try ... | 11:59 |
zyga-mbp | thank you! | 11:59 |
ogra | $ snap download --revision=44 pi-kernel | 12:00 |
ogra | Fetching snap "pi-kernel" | 12:00 |
ogra | error: cannot download snap "pi-kernel": Access by specifying a revision is not allowed for this Snap. | 12:00 |
ogra | nope | 12:00 |
zyga-mbp | oh well | 12:00 |
ogra | i fear only the kernel team can ... | 12:00 |
ogra | (i can download the current revision for each track/channel but not specify --revision) | 12:01 |
zyga-mbp | yeah, same here | 12:01 |
ogra | also 44 is pretty old .. we're at 200 currently | 12:02 |
zyga-mbp | I know, we are trying to reproduce a peculiar failure | 12:02 |
ogra | (for armhf that is) | 12:02 |
mborzecki | cmatsuoka: hi, can you take a look at https://github.com/snapcore/snapd/pull/9474 ? it's been +1'ed by pedronis | 12:27 |
mup | PR #9474: boot, overlord/devicestate: list trusted and managed assets upfront <Run nested> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9474> | 12:27 |
cmatsuoka | mborzecki: already looking, was about to +1 it | 12:28 |
mborzecki | cmatsuoka: cool thanks! | 12:29 |
mborzecki | cmatsuoka: fwiw https://github.com/snapcore/snapd/pull/9443 could use your review too :) | 12:29 |
mup | PR #9443: gadget, gadget/install: support for ubuntu-save, create one during install if needed <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9443> | 12:29 |
cmatsuoka | mborzecki: ah yes, I started reviewing it a few days ago and didn't finish it, resuming now | 12:36 |
mborzecki | cmatsuoka: thank you | 12:44 |
cmatsuoka | mborzecki: done | 12:53 |
mborzecki | cmatsuoka: yay, now if only testing was green | 12:54 |
jibel | Hi, I cannot launch any snap on groovy. When I launch a command from the terminal, it waits for a moment then I get this error: | 12:55 |
jibel | $ matterhorn | 12:55 |
jibel | cannot stat /var/lib/snapd/seccomp/bpf/snap.matterhorn.matterhorn.bin: No such file or directory | 12:55 |
jibel | is there a knows issue? | 12:56 |
jibel | $ snap version | 12:56 |
jibel | snap 2.47.1+20.10 | 12:56 |
jibel | snapd 2.47.1+20.10 | 12:56 |
jibel | series 16 | 12:56 |
jibel | ubuntu 20.10 | 12:56 |
jibel | kernel 5.8.0-22-generic | 12:56 |
zyga-mbp | jibel hi | 12:58 |
zyga-mbp | hmm | 12:58 |
mborzecki | interesting | 12:58 |
zyga-mbp | I assume the referenced file does not exist | 12:58 |
zyga-mbp | can you run snap list without failure? | 12:58 |
jibel | yes | 12:59 |
jibel | no failure | 12:59 |
mborzecki | jibel: yeah, can you also do `ls -l /var/lib/snap/seccomp/bpf` ? | 12:59 |
zyga-mbp | and if you run "snap changes" do you see any errors? | 12:59 |
jibel | and the snaps I cannot launc hare listed | 12:59 |
jibel | ah | 13:00 |
jibel | 65 Error today at 13:04 CEST today at 13:06 CEST Auto-refresh snaps "snapd" | 13:00 |
jibel | and the error | 13:00 |
jibel | 2020-10-14T13:04:37+02:00 ERROR cannot compile /var/lib/snapd/seccomp/bpf/snap.dl-ubuntu-test-iso.dl-ubuntu-test-iso.src: fork/exec /snap/snapd/8790/usr/lib/snapd/snap-seccomp: no such file or directory | 13:00 |
jibel | log of change 65 https://paste.ubuntu.com/p/MKWRHhZpX4/ | 13:01 |
jibel | upgrade of snapd didnt go well apparently | 13:01 |
mvo | zyga-mbp: I can give you core18 r1076 - give me 1min | 13:02 |
zyga-mbp | mvo thank you | 13:02 |
zyga-mbp | jibel we have a standup call now | 13:02 |
zyga-mbp | I'll get back to you after that | 13:02 |
jibel | np | 13:02 |
jibel | zyga-mbp, for some reason auto-refresh of snapd failed, I re-refreshed snapd manually and it fixed $world | 13:05 |
jibel | weird | 13:05 |
zyga-mbp | jibel could you look for system logs from that moment | 13:08 |
zyga-mbp | perhaps there are clues there? | 13:08 |
jibel | here it is https://paste.ubuntu.com/p/ZSRcqbsVCM/ | 13:19 |
zyga-mbp | jibel thank you | 13:28 |
zyga-mbp | jibel very interesting | 13:28 |
zyga-mbp | jibel let me read the log quickly | 13:38 |
zyga-mbp | jibel can you ls -ld /snap/snapd/8790/usr/lib/snapd/snap-seccomp | 13:38 |
zyga-mbp | and report a bug with this log, snap changes and snap tasks NNN where NNN is the change that failed | 13:38 |
zyga-mbp | I need to get my dog to the vet in a moment and I don't want to lose a chance to see the state your system was in right now | 13:38 |
zyga-mbp | jibel: ^ | 13:40 |
jibel | zyga-mbp, version 8790 has been removed from the system, I've only 9279 and 9607 | 13:40 |
zyga-mbp | that explains a lot | 13:40 |
zyga-mbp | thank you! | 13:40 |
jibel | $ ls -ld /snap/snapd/*/usr/lib/snapd/snap-seccomp | 13:40 |
zyga-mbp | so snap changes | 13:40 |
jibel | -rwxr-xr-x 1 root root 2306928 sept. 4 18:33 /snap/snapd/9279/usr/lib/snapd/snap-seccomp | 13:41 |
jibel | -rwxr-xr-x 1 root root 2306928 sept. 30 06:59 /snap/snapd/9607/usr/lib/snapd/snap-seccomp | 13:41 |
jibel | -rwxr-xr-x 1 root root 2306928 sept. 30 06:59 /snap/snapd/current/usr/lib/snapd/snap-seccomp | 13:41 |
zyga-mbp | and snap tasks for the last few changes that happened | 13:41 |
zyga-mbp | I really need to go now | 13:41 |
jibel | I'll report a bug | 13:41 |
zyga-mbp | mborzecki, pstolowski: ^ perhaps something you can pick up to the extent that the relevant data is collected in a bug | 13:41 |
* zyga-mbp goes | 13:43 | |
mborzecki | jibel: can you find the pid of `snapd` and then run `sudo ls -l /proc/<pid>/exe` ? | 13:46 |
cachio | mvo, could you find the error for sbuild? | 13:47 |
mborzecki | cachio: got log of that failure? | 13:51 |
cachio | mborzecki, which failure? | 13:51 |
cachio | the sbuild on? | 13:52 |
mborzecki | cachio: yes, the sbuild one | 13:52 |
cachio | mborzecki, https://paste.ubuntu.com/p/f9MJNgc6r9/ | 13:53 |
jibel | mborzecki, lrwxrwxrwx 1 root root 0 oct. 14 15:02 /proc/70906/exe -> /usr/lib/snapd/snapd | 13:54 |
jibel | mborzecki, zyga-mbp bug 1899794 | 13:54 |
mup | Bug #1899794: snapd error during refresh <snapd (Ubuntu):New> <https://launchpad.net/bugs/1899794> | 13:54 |
mborzecki | jibel: can you attach the output of journalctl -u snapd.failure.service too? | 13:58 |
jibel | done | 14:00 |
mborzecki | jibel: thanks, this looks very interesting, can you also grab the output of `journalctl -u snapd.service` from before oct 12? | 14:03 |
mborzecki | jibel: looks like something happend on oct 12 ~7:03 and the failure handler got started, so maybe a complete log of the snapd.service would be even better | 14:05 |
jibel | mborzecki, attached | 14:09 |
mborzecki | jibel: thanks | 14:09 |
jibel | I'm wondering if it could be related to bug 1871538 where systemd brings the entire session down | 14:09 |
mup | Bug #1871538: dbus timeout-ed during an upgrade, taking services down including gdm <amd64> <apport-bug> <champagne> <focal> <id-5e988df7fb344884f67bc04f> <rls-ff-incoming> <systemd:New> <accountsservice (Ubuntu):Invalid> <dbus (Ubuntu):Incomplete> <gnome-shell (Ubuntu):Invalid> <accountsservice | 14:09 |
mup | (Ubuntu Focal):Invalid> <dbus (Ubuntu Focal):Incomplete> <gnome-shell (Ubuntu Focal):Invalid> <https://launchpad.net/bugs/1871538> | 14:09 |
mborzecki | jibel: not sure, looks like there were 2 instances of snapd service running one point | 14:17 |
mborzecki | jibel: was there a package update sometime ~7:00 on oct 12? | 14:30 |
jibel | mborzecki, yes among other there was snapd:amd64 (2.46.1+20.10, 2.47.1+20.10) | 14:32 |
mborzecki | jibel: around that time? | 14:32 |
jibel | from 07:01:25 to 07:03:53 | 14:33 |
mborzecki | jibel: cool, thanks | 14:33 |
mborzecki | jibel: ok, added more comments to the bug, a nice one | 14:44 |
mborzecki | mvo: pedronis: i've added some comments to https://bugs.launchpad.net/snapd/+bug/1899794 looks like a real problem | 14:45 |
mup | Bug #1899794: Error during refresh of snapd leads to unusable system <snapd:Triaged> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1899794> | 14:45 |
mborzecki | errands | 14:46 |
jibel | mborzecki, ta | 14:49 |
pedronis | mvo: I did a pass in #9469 | 15:23 |
mup | PR #9469: snapshots: import of a snapshot set <Created by mvo5> <https://github.com/snapcore/snapd/pull/9469> | 15:23 |
pedronis | mborzecki: thx, snap-failure /revert logic needs improvements | 15:25 |
mvo | pedronis: awesome, thank you! | 15:35 |
cachio | /me lunch | 15:36 |
* cachio lunch | 15:36 | |
mup | PR snapd#9501 opened: [RFC] wrappers: do not error out on read-only /etc/dbus-1/session.d filesystem on core18 <Created by stolowski> <https://github.com/snapcore/snapd/pull/9501> | 16:30 |
mvo | cachio: I did not even look at the sbiuld error, so sorry! had meeting, an interview and 360 | 17:31 |
cachio | mvo, np, I'll take a look | 17:32 |
cachio | mvo, is it failing to build https://paste.ubuntu.com/p/hdxvkmrRVG/ | 17:33 |
cachio | mvo, is it any way to show more info about what failed? | 17:33 |
mvo | cachio: yeah, the error as is is really not helpful | 17:34 |
mvo | cachio: there should be maybe a sbuild.log with more info? | 17:34 |
cachio | mvo, nice, thanks | 17:34 |
zyga | re | 17:41 |
mvo | cachio: if you don't find anything, please remind me tomorrow, my day should be a bit less crazy tomorrow :) | 17:50 |
cachio | mvo, sure, thanks! | 17:50 |
zyga | sorry, this was a long visit | 17:53 |
zyga | I'll grab some food and do reviews for cachio | 17:53 |
zyga | and then try to install the kernel and core mvo shared | 17:53 |
mvo | zyga \o/ | 18:03 |
zyga | :) | 18:04 |
mvo | pedronis: I addressed the snapshot import feedback, will ask pawel tomorrow | 18:13 |
mvo | pedronis: for a second review | 18:13 |
mvo | pedronis: will call it a day now | 18:13 |
mup | PR snapcraft#3317 opened: plugin handler: properly handle snapcraftctl errors <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3317> | 18:23 |
* cachio -> kinesiologist | 18:46 | |
* zyga picks up zyga:tweak/ignore-running aka https://github.com/snapcore/snapd/pull/9406 | 19:38 | |
mup | PR #9406: many: allow ignoring running apps for specific request <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9406> | 19:38 |
mup | PR snapcraft#3318 opened: plugin handler: set -x for scriptlets <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3318> | 19:53 |
mup | PR snapd#9502 opened: tests: more output for sbuild test <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9502> | 20:36 |
* zyga made some progress and calls it a day | 20:39 | |
zyga | ok one more patch | 20:49 |
zyga | pushed some tests to https://github.com/snapcore/snapd/pull/9406 | 20:49 |
mup | PR #9406: many: allow ignoring running apps for specific request <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9406> | 20:49 |
mup | PR snapcraft#3319 opened: [RFC] plugin handler: use bash with additional error checking for core20 scripts <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3319> | 21:03 |
mup | PR snapd#9503 opened: tests: use tests.backup tool <Run nested> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9503> | 21:31 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!