[05:22] PR snapd#9145 closed: boot: track trusted assets during initial install, assets cache [05:59] morning [06:20] zyga: hey [06:20] hey :) [06:25] I both wake up at around 6:30 [06:25] and am sleepy because I cannot get to bed earlier than 1AM [06:46] mvo: hey [06:51] good morning mborzecki [06:53] PR snapd#9139 closed: interfaces/many: miscellaneous updates for strict microk8s [06:53] PR snapd#9143 closed: snap: fix repeated "cannot list recovery system" and add test [06:53] mborzecki: is 9137 something you are looking at or shall I do that? [06:58] PR snapd#9122 closed: mkversion.sh: disallow changelog versions that have git in it, if we also have git version [06:58] mvo: i'm looking at it [06:59] ta [06:59] mborzecki: I had it open so just wanted to make sure we don't duplicate work :) [06:59] * mvo looks at fuse instead [07:03] morning [07:10] now all under control [07:12] back to bugfixing [07:12] good morning pstolowski and zyga [07:13] good morning :) [07:15] o/ [07:15] hey Pawel :) [07:15] pstolowski: hey [07:15] have you seen https://news.ycombinator.com/item?id=24129208 ? [07:16] and https://www.reddit.com/r/linux/comments/i7yk1i/why_is_there_only_one_snap_store/ ? [07:26] mborzecki the amount of crap there really makes me wonder if having a walk is an order of magnitude more productive way to spend time [07:26] no amount of facts will help with the infinite pool of opinion [07:27] zyga: haha, i appraise galgalesh's effort to explain the state of things, but at the same time if feels it's all futile [07:30] hm there's no tool in core to calculate sha3-384? [07:31] mborzecki: we should probably have a snap debug for this :/ [07:31] pstolowski: sorry that the generator PR is such a bikeseed :/ [07:32] mvo: i'll try with python's hashlib first, don't recall if sha3 is supported there or not [07:32] ok [07:34] mvo: don't think there is bikeshed, it's fine [07:34] thanks for review and suggestions [07:42] mvo: the /run/systemd/container tip is excellent [07:48] yeah, that's really cool [07:53] error: cannot install snap file: snap "test-snapd-app" has running apps (pause) [07:53] heh, I guess that's dogfooding allright [07:57] zyga: re #9152 and system("which.."), do you suggest splitting+looping over PATH and "manual" checks? [07:57] PR #9152: cmd/snapd-generator: generate drop-in to use fuse in container [07:57] yes [07:57] strtok_r or something like [07:57] stat or access(X_OK) [08:02] yep [08:02] zyga: do you think it should be a helper in libprivate..? [08:02] only if you want to - we don't need that in snap-confine today [08:08] is there a way to stop 'snap watch '? [08:08] ctrl-C, q, aren't working [08:08] neither is esc [08:10] hmm? [08:10] that's weird [08:10] what's the state of the process? [08:10] what process? [08:10] there is something weird [08:11] Télécharger un paquet Snap "ubuntu-bug-triage" (205) à partir du canal "l… 56% 3.75MB/s 731ms [08:11] the snap is 19M [08:11] latest/stable: 2020.08.13+git2e8c31d 2020-08-13 (205) 19MB - [08:11] and it's doing that 56% for some minutes [08:11] so I think something is stuck [08:13] also according to gnome-system-monitor there is no activity, so the snap watch info is lying [08:20] zyga, any idea? can I get any more debug info before I kill snapd? [08:20] seb128 snapd or snap wait? [08:20] I'm going to kill snapd since it seems stuck [08:20] hmm [08:21] can you abort the change [08:21] and try again [08:21] I've killed snap wait a bunch of times already [08:21] I should have to do that though [08:21] yeah [08:21] but I want to see if snapd is responsive [08:21] what's the state of snapd.service? [08:22] zyga, https://people.canonical.com/~seb128/snapd.png [08:24] zyga, https://people.canonical.com/~seb128/snapdstatus.txt [08:24] août 14 09:00:03 sebxps snapd[1336]: 2020/08/14 09:00:03 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n

408 Request Time-out

\nYour browser didn't send a complete request in time.\n\n"; err= [08:24] ha [08:24] that's interesting [08:24] I guess that's what failed [08:24] if you "snap abort" the refresh change it will fix itself [08:24] and we should fix our side to handle this error better [08:25] should I open a bug on launchpad? [08:25] do you need more info before I abort? [08:25] yeah, I think it will help, I'm not super familiar with out store stack [08:25] pstolowski ^ [08:25] which I guess is going to loose the state [08:25] can you have a look please and advise seb128 [08:25] zyga, thanks for the replies! [08:25] always :) [08:32] seb128: yes, please open a bug. i don't have immediate answer to that, seems error handling is missing somewhere, needs investigating [08:33] pstolowski, I still have the state, I didn't abort yet, do you need any more info before I press the trigger? [08:38] seb128: just in case - 'snap changes' and 'snap change ' if snapd is responding; if not, then 'snap debug state /var/lib/snapd/state.json --changes' and 'snap debug state /var/lib/snapd/state.json --change=..' [08:38] 408 are kind of a known issue, a combination of store and snapd [08:39] pstolowski, snapd is responding [08:39] 315 Doing today at 08:59 CEST - Actualiser automatiquement les paquets Snap "openstackclients", "ubuntu-bug-triage", "snapcraft" [08:41] seb128: ok, also snap change 315, please attach to bug report (@mvo: do we already have a bug for that?) [08:41] pstolowski, https://people.canonical.com/~seb128/snapchange.txt [08:41] k, I will open a bug with those info [08:41] thanks [08:44] seb128: thank you [08:49] re [08:50] I'm making progress in my recovery [08:50] but man is that annoying [08:50] my ass hurts from sitting [08:50] just not used to it [08:50] I need to take breaks every 15 minutes or so [09:03] pstolowski, zyga, mvo, k, reported as https://bugs.launchpad.net/snapd/+bug/1891618 now [09:03] Bug #1891618: Snapd stucked after a request timeout error [09:05] seb128: ty [09:07] bah [09:07] so I did snap abort 315 [09:07] PR core18#43 closed: Remove manpages and other documentation, cleanup empty doc dirs [09:07] PR core18#72 closed: writable-path: enable persistent journal [09:07] $ snap changes [09:07] ... [09:07] seb128: could you please attach full https://people.canonical.com/~seb128/snapchange.txt there? [09:07] 315 Abort today at 08:59 CEST - Actualiser automatiquement les paquets Snap "openstackclients", "ubuntu-bug-triage", "snapcraft" [09:07] but [09:07] $ sudo snap refresh ubuntu-bug-triage [09:07] erreur : snap "ubuntu-bug-triage" has "auto-refresh" change in progress [09:07] [09:07] I still can't refresh my snap? [09:08] pstolowski, I'm trying but launchpad timeouts when I try to add a file :/ [09:08] mhm [09:08] ah, worked now [09:08] what worked? refresh or adding attachment? [09:08] addind attachment [09:09] I guess I need to kill snapd [09:09] or to reboot but I've work open and I don't want to loose the context [09:10] seb128: give it a few moments, i think it needs to abort and undo things [09:11] pstolowski, it has been 5 minutes on a modern config and I see no sign of new activity in snap changes [09:11] I think it's not going to recover or unstuck itself [09:12] I can wait a bit longer though [09:12] PR core18#90 closed: Placeholder files [09:12] PR core18#126 closed: hooks: reduce snapd skeleton directories [09:12] maybe it's stuck inside this task, give it a minute then yes, snapd restart [09:12] PR core18#134 closed: hooks: add apt/apt-get/apt-cache warning (based on mvo's branch) [09:12] we will look into it [09:17] PR core18#166 closed: hooks/900-cleanup-etc-var.chroot: rm cloud-init config file which we don't want [09:18] pstolowski, it has been 10 minutes, I don't think it makes sense to keep waiting, I restart snapd [09:18] seb128: yes [09:19] what, systemctl restart snapd is also blocking [09:19] PR core20#11 closed: Add arm64 architecture [09:19] * seb128 does shotgun snapd [09:20] hum [09:20] ~$ ps aux | grep snapd [09:20] root 11537 0.0 0.0 477372 6116 ? Ssl 11:19 0:00 /usr/lib/snapd/snap-failure snapd [09:20] what is snap-failure? [09:20] (no manpage) [09:24] seb128 it's a program we invoke after snapd fails [09:24] seb128: reverts snapd to a previous revision [09:24] IIRC it should only run on core systmems [09:24] and yes, we should write manual pages [09:24] PR core20#54 closed: static: switch /etc/cloud from synced to persistent [09:24] * zyga happily will [09:25] well, there process is there on my classic desktop [09:25] and not going away [09:25] I guess I do need to restart the machine :( [09:25] hmmm [09:25] mvo ^ [09:25] this is probably very not intentional [09:25] that's annoying, I really don't want to reboot now [09:26] but I need to refresh some snaps to work on things [09:26] * seb128 tries to kill snap-failure, who knows [09:26] i'm setting this to critical [09:26] ah, I got a working snapd back [09:27] pstolowski, zyga, thanks for the help [09:27] pstolowski, well, if it's rare enough that only me have been hitting it that's probably not critical [09:27] zyga: it runs on classic too, otherwise nothing would revert snapd if it fails to start [09:27] seb128: sure, sorry for the inconvienience, we will look into it [09:27] mborzecki hmmmmm [09:27] np, thanks for the help! [09:27] seb128: can you post journalctl -u snapd somewhere? [09:27] zyga: maybe you meant snap-repair? [09:28] no [09:28] hmm [09:28] so what is going on [09:28] why is it running? [09:28] PR snapd#9156 opened: boot: copy boot assets cache to new root [09:28] zyga: snapd crashed & failed probably [09:29] mborzecki, https://people.canonical.com/~seb128/snapjournal.txt [09:29] yes but is snapd-failure finished [09:29] or stuck [09:29] Issue core20#66 closed: Use focal base in travis [09:29] PR # closed: core20#58, core20#73, core20#74, core20#77, core20#78 [09:30] seb128: thanks, can you include all of journal between 11:06 and 11:26? [09:32] mborzecki, https://people.canonical.com/~seb128/journal.txt [09:33] seb128: thanks! [09:33] mborzecki, 11:19 is when I ended up doing a kill -9 snapd [09:33] mborzecki, and :26 when I killed snap-failure [09:34] or :25 rather [09:37] mvo: ^^ some weird interaction, snapd failed, `snap-failure snapd` got started and launched snapd, which then worked as if nothing happened, because original failure was not related to snapd task [09:38] mborzecki: oh, why did the original snapd fail? also that sounds like we need snap-failure more clever and just do nothing but restart snapd if it's unrelated to refreshes [09:38] PR snapd#9137 closed: boot: refactor such that bootStateUpdate20 mainly carries Modeenv [09:38] seb128: i see you tried to referesh ubuntu-bug-triage snap and abort a change 315, what was the change about? [09:39] do we know why snapd failed? nil pointer or something? [09:40] seb128: was there a prompt from gtk polkit asking for a password maybe? [09:41] mvo: looks like snapd was not responding, seb128 tried to restart it, but snapd did not exit so got killed by systemd and triggered snap-failure [09:41] mborzecki, no, but I was on xubuntu session for trying something and polkit prompting seems to not work there [09:41] so maybe it did try and fail [09:43] mvo: i'm speculating, but what would happen if you have a polkit prompt (or snapd goes to ask polkit) and you try to restart the snapd service? [09:44] mborzecki: no idea, but that sounds plausible [09:44] that it would hang then [09:49] mvo: heh, systemctl restart snapd is hanging, snapd is clearly waiting for polkit to answer [09:50] mvo: well, but it was stuck on api activity, so eventually closed [09:50] mvo: so prpbably something else then [09:54] mborzecki: looking at snap-failure I think it is indeed a bit too naive and not considering non-upgrade cases properly [09:59] PR snapd#9157 opened: o/devicestate: wrap asset update observer error [09:59] mvo: we don't look at the state at all, it feels like snapd should do it when it's launched by snap-failure [09:59] mborzecki: yeah [09:59] mvo: perhaps we need snap failure to indicate it's in some sort of failure handling mode and the previous snapd would act accordingly (eg. in this case exit nicely and do nothing?) [10:09] * zyga heads upstairs [10:31] mvo I have a fix for the bug [10:32] mvo I will have a patch with a regression test ready by the standup [10:32] it's something we can easily cherry pick into any release [10:42] mborzecki: yes, I think so [10:42] zyga: for which bug? [10:42] mvo the priority bug from field https://bugs.launchpad.net/snapd/+bug/1891371 [10:42] Bug #1891371: mount namespace issues [10:42] just working on better testing [10:42] and explanations [10:43] zyga: nice! [10:51] PR snapcraft#3203 closed: experimental ros2 extension & colcon v2 plugin [10:59] mvo: updated #9152, spread test passed now locally, i had a silly bug (C is full of traps) [10:59] PR #9152: cmd/snapd-generator: generate drop-in to use fuse in container [10:59] pstolowski: nice [11:03] pstolowski I'll re-review after standup [11:05] ty [11:13] * pstolowski lunch [11:26] looks like we have a bug with "service.console-conf.disable: true" during seeidng of core20, instigating right now [11:29] /me learned something new justn ow [11:36] PR snapcraft#3248 opened: specifications: fix typo in package-repositories [11:36] PR snapcraft#3249 opened: Tiny typo fix [11:44] PR snapd#9158 opened: logger: add support for setting snapd.debug=1 on kernel cmdline [12:02] oh wow, GitLens for ms visual code is nice [12:03] E: Failed to fetch http://us-east1.gce.archive.ubuntu.com/ubuntu/pool/main/a/apparmor/libapparmor-dev_2.13.3-7ubuntu6_amd64.deb Temporary failure resolving 'us-east1.gce.archive.ubuntu.com' [12:03] anyone seen this? [12:04] PR snapd#9159 opened: cmd/snap-update-ns: detach all bind-mounted file [12:11] mborzecki /proc/pid/fdinfo/FD shows the mount id of the filesystem that contains the open file [12:11] TIL [12:11] PR snapcraft#3248 closed: specifications: fix typo in package-repositories [12:11] * zyga coffee and then more bugs [12:11] PR snapcraft#3249 closed: Tiny typo fix [12:16] xnox: do you think you could have a look at https://bugs.launchpad.net/snapd/+bug/1891644 ? what should we do with console-conf when it's disabled [12:16] Bug #1891644: uc20 seeding fails with "service.console-conf.disable: true" [12:28] pstolowski: https://github.com/snapcore/snapd/pull/9152#pullrequestreview-467530317 [12:28] PR #9152: cmd/snapd-generator: generate drop-in to use fuse in container [12:39] PR snapd#9157 closed: o/devicestate: wrap asset update observer error [12:43] zyga-mbp: thanks [12:44] afk, see you at the standup [13:19] PR snapd#9160 opened: boot, o/devicestate: observe existing recovery bootloader trusted boot assets [13:23] I'll grab lunch now if I can [13:34] i need to wrap it up for now and attend to some errands, bbl [15:05] PR snapd#9161 opened: kernel: add kernel.Validate() [15:10] pstolowski, hey [15:10] I see this https://paste.ubuntu.com/p/V2hF7TJzjB/ [15:11] any idea which could be the root cause' [15:11] ? [15:12] cachio: is this master? [15:13] pstolowski, it is my nested branch [15:14] cachio: this test is hacking stuff around snapd inside the preseeded system. it is senstive about what version of snapd runs with snap-preseed, and what version of snapd is used when the preseeded system boots [15:16] pstolowski, ok, I'll try to see if it is caused because any change in the PR [15:17] cachio: it is also injecting snapd into seeds of the preseeded system [15:17] cachio: also, do you have all changes from master in your branch? [15:18] pstolowski, no latest ones [15:22] * cachio lunch [15:49] I have a fix for networkd [15:49] will send and EOD [15:55] PR snapd#9162 opened: gadget: change mountedfilesystemwriter to use resolvedSource (3/N) [16:00] PR snapd#8931 closed: tests/main/install-fontconfig-cache-gen: enhance test by verifying, add fonts to test [17:19] * zyga-mbp goes to exercise [17:20] PR snapd#9163 opened: tests: work around broken update of systemd-networkd [17:26] cachio ^ plz merge when green [17:26] zyga-mbp, sure [17:27] enjoy the exercises [18:01] PR snapd#9165 opened: interfaces: add kernel-crypto-api interface - 2.46 [18:11] PR snapd#9166 opened: interfaces/many: miscellaneous updates for strict microk8s - 2.46 [20:56] PR snapd#9167 opened: many: correctly calculate the desktop file prefix everywhere [21:11] PR snapd#9163 closed: tests: work around broken update of systemd-networkd [21:42] PR snapcraft#3250 opened: cli: implement list-tracks