[06:22] morning [06:23] mborzecki: good morning [06:37] mborzecki: I cherry-picked 8575 to 2.44 [06:37] mborzecki: that is needed, right? we will need a new 2.44.4 with a fix for syscalls with base: core20 [06:38] mvo: thanks! [06:38] mvo: yes, it fixes the problems with libcrypto dependency being injected on centos [06:39] mvo: i'll need to add a similar change in downstream packaging for fedora/epel [06:39] ta [06:40] Good morning [06:41] How are you guys? [06:41] I will start late but I will look at opensuse a little today [06:41] I’ll fix our tests there and update the packaging /release [06:42] zyga: hey, I'm good, thanks! [06:45] qemu 5.0 is out, supports emulating tpm for arm [06:45] Groovy [06:46] Can we snap it? :-) [06:52] mvo: hmm 2.44 travis unit test job runs with different go version? [06:53] ah, it's go master [06:56] mborzecki: yeah, just removed that [06:56] mborzecki: I want to see the spread run [06:57] i'm out for a bit, and yay it's raining! [07:04] morning [07:04] hey pstolowski - good morning === rawr is now known as grumble [07:43] mvo, zyga can one of you help me with that snapd download, please? [08:01] I will try [08:01] Which version do you need? [08:22] juergh: ^ ? [08:29] did xdg-open change recently ? [08:29] error: error running snapctl: Unknown command `user-open'. Please specify one command of: get, is-connected, restart, services, set, set-health, start, stop or unset [08:33] ogra: hmm, weird [08:33] ogra: perhaps a bug [08:34] i'm not yet sure, might be that the app changes ... the snapctl got me hooked though [08:34] *changed [08:37] * zyga breakfast [08:42] zyga, snapd 2.43.3 [08:42] ok [08:42] give me a moment please [08:51] juergh: which architecture do you need? [08:52] zyga, amd64 [08:52] juergh: hmm, but wait, 2.43.3 is latest stable [08:52] ah [08:52] sorry 44 [08:52] * zyga needs coffee [09:59] * zyga looks at fixing the opensuse build issue now [10:01] Hi, is there a reason why snapd /etc/profile.d/snapd.sh *appends* snap_bin_path to PATH, rather than prepends? [10:01] https://forum.snapcraft.io/t/should-snap-bin-be-prepended-to-path/4506 [10:01] tpreston: it's a more conservative choice, I suspect [10:01] I have ctags installed at a system level (because of a package dependency), but I'd prefer to use snapd universal-ctags [10:02] zyga: I guess so, maybe I should alias ctags then [10:04] how can I make how can I make file:///usr/share/doc/python3-doc/html/index.html work with the chromium snap? [10:06] doko: apart from serving it over http, I guess you cannot [10:07] doko: maybe copy it to ~ ? [10:09] zyga: broken by design to access anything in /usr/share/doc ? [10:09] doko: not broken by design [10:09] doko: it's just not designed to access arbitrary host locations [10:10] doko: there are two elements at play, the view of the filesystem and the sandbox [10:10] well, copying everything from /usr/share/doc to ~ is not viable [10:10] doko: chromium and your system see different filesystem [10:10] doko: and while /usr/share/doc is probably safe and could be allowed, it's not a general property [10:10] is there a bug report about that? [10:11] doko: before you file one, do you have an idea of how it should work? [10:11] doko: I mean, I share the sentiment of it feeling broken [10:11] doko: but what could we do? [10:13] well, for me it then not using the snap anymore [10:13] have a "host-documentation" interface that lallows reading /usr/share/doc content ? [10:13] doko: to partially answer my own question, at some point the document portal could be integrated into chromium, so that it could open a file [10:13] but IIRC the portal does not handle directories yet [10:14] doko: while I agee it's a regression I think using chromium to browse debian-style local documentation is fortunately niche [10:15] ogra: if you do that then the snap won't be able to see the /usr/share/doc from the base, I guess that's acceptable but rather obscure [10:15] ogra: but I think that's a good idea [10:17] yeah, i doubt the base will actually have content in /usr/shar/doc apart from copyright files [10:17] doko: I would strongly encourage you to file a bug or open a forum thread, I think ogra's idea is viable and well worth discussing [10:17] unless the building has changed massively (we used to remove everything there in the expectation that nobody would every want to use docs inside a base) [10:18] ogra: yes, I think we are lucky in a sense [10:18] if it was something other than that directory, it's probably not something that we could rely on [10:18] yeah [10:20] btw, using a portal would be horrible i fear, every link inside that index.html would pop up a new portal window [10:20] zyga: where to file a bug? [10:21] ogra: I think there's a plan to open an entire directory as a portal [10:21] ah [10:21] doko: please file the bug on snapd on launchpad [10:24] https://bugs.launchpad.net/snapd/+bug/1875860 [10:24] Bug #1875860: local documentation is not accessible from the chromium snap [10:26] ogra: I added a comment explaining your idea [10:26] doko: thank you! [10:26] :) [11:04] hi, are these error coming from snapd itself https://launchpadlibrarian.net/477374648/DpkgTerminalLog.txt ? [11:08] yes [11:08] ackk: udev in containers [11:08] ackk: it's a known topic that's on the backlog [11:10] zyga, what's the issue there? [11:10] ackk: snapd tries to reconfigure udev but udev is not meant to be used in containers (we pull it as a debian dependency) and it fails [11:11] zyga, for context, https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1875741 is the bug for that trace. basically a bionic->focal dist-upgrade with maas transitioning from deb to snap broke [11:11] Bug #1875741: package maas 2.6.2-7841-ga10625be3-0ubuntu1~18.04.1 failed to install/upgrade: new maas package pre-installation script subprocess returned error exit status 1

[11:11] ackk: on subsequent runs snapd things everything is okay and no longer tries [11:11] zyga, so trying once again should fix it? [11:11] ackk: it's a topic we know about since Fra but we haven't managed to discuss at depth with Security [11:11] ackk: yes [11:11] ackk: there are some disagreements and we frankly didn't have time to sit down and discuss this [11:12] zyga, did anything change recently? I did test maas deb->snap transition in containers, never seen this issue [11:12] not that I know of [11:13] zyga, won't that break anyone dist-upgrading a container with snapd installed to focal? or is some kind of race that might happen? [11:14] ackk: it probably will [11:14] ackk: just a fire more distant than fires close up [11:14] zyga: got a bug number for it? [11:14] sparkiegeek: yes, let me find it [11:14] sparkiegeek: I tried addressing it before [11:14] one sec [11:15] zyga, sure, not implying anything, just trying to understand if there's something on the maas package side we could do, which it seems ther's not? [11:17] https://github.com/snapcore/snapd/pull/8219 [11:17] PR #8219: interfaces: use udev backend if udev socket exists [11:17] this is my idea and the bug is https://bugs.launchpad.net/snapd/+bug/1865503 [11:17] Bug #1865503: snapd (deb) fails to install snapd (snap) inside LXD on a buildd based image [11:17] the discussion there is interesting as well [11:17] my idea is that we should not depend on udev, not pull it in for containers, not use any udev when there's no real udev or real /dev [11:19] sparkiegeek, ackk: if you have opinions I'd love to learn from you as well [11:20] zyga: no educated opinion here :| [11:20] zyga, same here... [11:31] PR snapd#8578 opened: interfaces: add host-docs interface [11:58] pstolowski: does your PR now work on centos 8 with the fix in master? [12:19] mborzecki: which PR? [12:19] pstolowski: don't remember :P one that was blocked by selinux basically, or was there none? [12:20] mborzecki: master was blocked [12:20] ahh cool [12:22] doko: https://github.com/snapcore/snapd/pull/8578 [12:22] PR #8578: interfaces: add host-docs interface [12:22] doko: it even has a test: https://github.com/snapcore/snapd/pull/8578/files#diff-aea859a8dd781e56f1c21b809b953d95R13 [12:31] oh, look someone found the mvo backdorr in ubuntu core !!! once you create an mvo account you can never delete it anymore !! https://forum.snapcraft.io/t/login-and-delete-user-created-with-v2-create-user/17019 [12:33] all devices belong to mvo ;) [12:33] yeah ! [12:33] ogra: haha [12:34] i really wonder what makes people use some random accounts found on the internet for such stuff :) [12:34] it's baffling! [12:37] mvo: lol [12:53] mvo: I wonder if the user found this is our tests or if there is some documentation example [12:54] probably [13:48] I could use a review for rather simple https://github.com/snapcore/snapd/pull/8565 [13:48] PR #8565: osutil: expand FileLock to support shared locks and more [14:02] ok, i have my opensuse install under control now [14:05] +1 [14:05] thanks! [14:09] ijohnson: do you have a moment ^ it's pretty simple [14:09] ijohnson: it would unblock a host of other PRs [14:11] zyga: will look in a bit busy atm [14:11] sure [14:17] mvo: pstolowski: do you think we can land https://github.com/snapcore/snapd/pull/8536 to unblock the rest? with any tweaks in the followups (there's a bunch of PRs stack on top of this one) [14:17] PR #8536: store,asserts,many: support the new action fetch-assertions [14:17] hm pedronis isn't around [14:18] brb [14:18] tea [14:19] mborzecki: best to ask him actually... don't know if there is any risk [14:29] PR snapd#8536 closed: store,asserts,many: support the new action fetch-assertions [14:29] mborzecki: I think it makes sense, done [14:29] thanks! [14:29] jdstrand: https://github.com/snapcore/snapd/pull/8578/files#diff-aea859a8dd781e56f1c21b809b953d95R13 [14:29] PR #8578: interfaces: add host-docs interface [14:30] zyga: cool thanks. I have it on my list to review, but note I'm sprinting/etc [14:31] no rush :) [14:57] cmatsuoka: I hit the infinite loop of failed to transient mount unit for ubuntu-seed you had the other day [15:13] ijohnson: it's quite infrequent, but it's something we'll have to fix [15:13] cmatsuoka: did the issue go away when you rebooted ? [15:14] * ijohnson would like to debug but is afraid that when rebooting the issue goes away [15:14] ijohnson: yes, when I booted the same image again it worked as expected [15:14] looks racy [15:14] mmm yeah [15:17] ah actually I can consistently reproduce this now on my image [15:17] * ijohnson debugs [15:18] oh that's good, I couldn't reproduce it [15:23] * cachio lunch [15:25] cmatsuoka: I video recorded the boot and caught this in the first few seconds of the log [15:25] https://usercontent.irccloud-cdn.com/file/84WtzfpC/loop%20uc20%20edge.png [15:26] the last line `systemd-fsck CP437 Invalid argument` is suspicious [15:26] oh interesting [15:26] yes, very suspicious [15:27] https://usercontent.irccloud-cdn.com/file/uYCPMgpF/loop%20uc20%20later.png [15:28] cmatsuoka: then this is also suspicious where run-mnt-ubuntu\x2dseed.mount fails because wrong fs type, bad option, etc. [15:29] ijohnson: maybe you could inspect the image file? (it could be a good idea to make a copy first in case you end up fixing the problem) [15:30] ijohnson: and check if the partition contents are there or the filesystem seems right [15:30] cmatsuoka: yes I am doing that now [15:30] the partition was mounted fine with kpartx + mount of the loopback device [15:31] does it look good? [15:31] yeah it has all the expected files and such [15:31] hmm [15:31] should I try to run fsck on the partition in the .img file ? [15:32] yeah, let's see what happens [15:34] huh fsck.etx4 says ubuntu-seed is actually a vfat partition? [15:34] https://www.irccloud.com/pastebin/C2v5Dpoz/ [15:35] mm, yes, seed is vfat [15:35] really? I thought seed was ext4 on amd64 ? [15:35] we must EFI boot it [15:36] ah okay so I was just wrong about that then [15:36] seed is vfat, the rest is ext4 [15:36] ok, fsck.vfat doesn't complain about the partition at all then [15:36] cmatsuoka: anything else I should check about the partition you think? [15:37] did it complain about code pages or something like that? [15:37] or maybe systemd-fsck does something different? [15:38] does it still fail if you boot the image again? [15:42] cmatsuoka: yes it reliably fails every time I boot now [15:42] cmatsuoka: all I got from fsck.vfat output was this: [15:43] https://www.irccloud.com/pastebin/nuIWXTi1/ [15:43] hmm [15:44] xnox, when you disabled the ability to disaable console-conf in ubuntu-image, did you make sure it still works for core18 builds ? [15:44] (someone just said it is completely gone, we have customers usign that option to build their images) [15:46] ijohnson: so the fs is good but for some reason it's not being mounted [15:47] yes [15:47] I can mount it fine though when the image is not booting [15:47] actually you know what, since this is still in install mode I can change the kernel cmdline since we never make it out of the initrd [15:47] so unsealing doesn't affect anything at this point [15:48] yes, you're right, let's inspect it while it's happening [15:48] xnox, ah, nevermind ... it still works but the --help info about it is gone [15:49] well damn now it boots fine [15:50] did you make a copy of the original image? [15:50] yes I did [15:50] let me make a copy of that original image and try again [15:51] ah now the copy of the original broken image works too :-/ [15:51] oh noes [15:52] so this definitely seems like a race [15:54] /o\ [15:57] well on the other hand, I notice that in the log of the failed boot that goes into this loop there's this message too: [15:57] [ 1.342368] FAT-fs (vda2): IO charset iso8859-1 not found [15:57] and that message doesn't appear in a good boot [15:57] so perhaps it's a race with some kernel module being loaded? [15:58] that sounds plausible [15:58] but then if it's desperately retrying, at some point it should work [15:59] ijohnson: https://askubuntu.com/questions/372445/fat-fs-io-charset-iso8859-1-not-found-error-while-mounting-fat-drives [16:01] cmatsuoka: what's so weird about this issue, is that it finishes the fsck check on /dev/vda2 in the loop case [16:01] ah but it wouldn't be loading the kernel module from that partition at this point, it would be loading it from the initrd in the kernel [16:02] so perhaps that kernel module in the initrd is corrupted somehow? [16:02] but then why it would work in the next boot? [16:03] well tbh I didn't quite make a perfect backup of the original image that was always reproducing, what I did was this: [16:04] 1. tried booting same img over and over again always reproducing infinite loop [16:04] 2. tried mounting the img file with kpartx and ran fsck.ext4 on it as explained [16:04] 3. unmounted with kpartx and made a backup copy [16:04] 4. re-mounted it and tried with fsck.vfat [16:04] 5. booted it and it didn't work first time [16:04] 6. rebooted it and changed kernel cmdline in grub menu, now it works every time [16:05] well, it still didn't boot in step 5 [16:06] yes but somehow changing the kernel cmdline made it boot fine [16:06] that's weird [16:06] I set the kernel cmdline to this: `console=tty1 console=ttyS0 rd.systemd.journald.forward_to_console=1 systemd.journald.forward_to_console=1 rd.systemd.debug_shell=1 dangerous panic=-1` [16:07] let me try re-creating this image fresh the way I did originally [16:08] I'll break for lunch and hopefully have a good idea while eating [16:08] because right now I don't have any [16:09] yeah just doing same thing again to build the image didn't reproduce it [16:09] * ijohnson runs it in a loop [16:11] hey I reproduced it again! [16:11] * ijohnson saves that img [16:14] you guys are having some fun :) [16:14] uc20 is too much fun [16:17] alright so every few time it hits that loop and fails, I see the IO charset iso8859-1 not found message, but when it does boot, I don't see that message, so it definitely is related [16:26] ogra_: i didn't change anything in ubuntu-image, and do not do any ubuntu-image development there. [16:26] ogra_: open an issue against ubuntu-image repository, with any bugs. [16:27] hi. I see my snapcraft push nightlies/vlc_*.snap --release beta, has succeeded in my CI, and I also see it under "Latest revisions for amd64" list. However it's not in the beta channel. [16:27] why would that happen, on the store? [16:28] thresh: this is more of a snapcraft question but perhaps snapcraft devs are around as well [16:29] thanks zyga, I'll complain in the related channel too :-) [16:31] thresh: I don't use snapcraft that often but I usually separately upload and release [16:47] cachio: are tumbleweed images updated periodically somehow or are we running a snapshot until something happens explicitly? [16:50] zyga, failed yesterday [16:50] sorry, I don't understand [16:51] let me try to fix it [16:51] cachio: can you please answer my question :-) [16:51] I mean [16:51] I' just want to understand what the process is [16:52] you can tell me what failed and about fixing it as well [16:52] but that's not what I was asking about [16:52] the process is: [16:52] the image is updated to a tamporal image [16:52] then the snapd tests are executed [16:53] if tests pass, then we turn that temporal image in a test image [16:53] otherwise the image is discarded [16:53] yesterday it failed running tests [16:54] failed preparing the the snapd teest suite [16:54] is there any notification when that happens? [16:54] no [16:54] that's not great, can we try go get some [16:54] it is in the brnaches board [16:54] ans it happens every monday [16:55] what is "branches board"? [16:55] https://travis-ci.org/github/snapcore/spread-cron/branches [16:55] https://travis-ci.org/github/snapcore/spread-cron/branches [16:55] zyga, this is the last log https://travis-ci.org/github/snapcore/spread-cron/builds/680087171#L2749 [16:56] sometimes I manually run the jobs to update the images [16:57] zyga, I see this Package 'python-docutils' not found. [16:58] I see, I think this package just got removed, it's a python 2 package [16:59] zyga, also 'pkg-config' not found in package names. [17:00] cachio: why are we installing python-docutil? [17:00] I don't see it in anything related to opensuse [17:00] cachio: yes, that's also been removed and replaced by something else === ijohnson is now known as ijohnson|lunch [17:01] zyga, this is the snapd test suite which fails [17:01] cachio: python-docutils is only listed in arch [17:01] cachio: why are we installing python-docutils, do you know? [17:01] zyga, don't know why it is installed [17:01] I guess a test need it [17:01] xnox, oh, sorry, GH tricked me ... you were just a reviewer on https://github.com/CanonicalLtd/ubuntu-image/pull/186 [17:01] PR CanonicalLtd/ubuntu-image#186: Some options are unsupported for UC20 builds [17:01] cachio: do you know which part of the code is installing it? I certainly don't see it in pkgdb.sh [17:02] I see it in opensuse dependencies but that's not managed by pkgdb.sh [17:04] cachio: ok, I see the problem now [17:04] packaging/opensuse-15.0/snapd.spec: [17:04] cachio: I think for sid, tumbleweed and arch we should not stop [17:04] cachio: if we stop and use some ancient version because tests don't pass [17:04] that's pretty much because snapd is in a way broken [17:04] the world has moved on [17:05] staying on an old image without alerting the team could be a problem later [17:05] in a sense those systems are all unstable because they never stop [17:05] zyga, for arch is not a problem because we update in every run the image [17:05] cachio: regardless of how we update [17:06] zyga, but I could remove snapd tests validation for tumbleweed and sid [17:06] it is very easy [17:06] cachio: yeah, I think we should do that [17:06] cachio: we can have them in unstable systems [17:06] zyga, yes [17:06] cachio: in reality, the package is gone from the archive [17:06] totally agree [17:06] cachio: not updating a stale image is not buying us anything [17:06] cachio: the package cannot be built [17:06] cachio: CI is not functioning as CI :) [17:07] cachio: cool, I'll send a patch to fix our packaging [17:07] zyga, nice, thanks! [17:07] cachio: but please let's figure out how to make some notifications flash on everyone's systems [17:07] cachio: so that next time we know earlier [17:07] zyga, sure, I'll configure to send emails [17:07] super, thank you :) [17:07] zyga, yaw [17:17] cachio: https://github.com/snapcore/snapd/pull/8579 [17:17] PR #8579: packaging: stop depending on python-docutils [17:17] PR snapd#8579 opened: packaging: stop depending on python-docutils [17:22] * zyga goes to make coffee ahead of the meeting [17:28] zyga, thanks [17:28] lets test it whith the new image once it is ready [17:31] sure [17:49] If someone runs `snap revert`, will it run the refresh hooks? === ijohnson|lunch is now known as ijohnson [17:49] kyrofa: afaik it won't [17:50] ijohnson, any hooks? Any way the snap to which is being reverted can know that a revert happened? [17:50] no, I think that's by design that the snap doesn't know it got reverted [17:51] Fair enough, thanks ijohnson [17:53] cmatsuoka: ah I think I understand the problem now [17:53] cmatsuoka: what's happening is this sequence of events: [17:54] 1. there is a systemd unit for running fsck on /dev/disk/by-label/ubuntu-seed that just always runs, so it happens to run first [17:55] 2. snap-bootstrap via "the-tool" runs, says to mount /run/mnt/ubuntu-seed _with transient unit_ (that's important) [17:55] 3. that fails because systemd-modules-load has not run, or finished running yet [17:55] 4. so we see IO charset iso8859-1 not found from the kernel because nls_iso8859_1 was not loaded [17:56] 5. systemd-modules-load eventually runs/finishes running and does load nls_iso8859_1: systemd-modules-load[238]: Inserted module 'nls_iso8859_1' [17:56] 6. the-tool keeps running snap-bootstrap, which says to mount /run/mnt/ubuntu-seed, however because of how the-tool tries to do the mount, it just requests a transient unit for the mount [17:57] 7. the transient unit was already started/created by systemd so systemd doesn't try again and just complains thusly: the-tool[380]: Failed to start transient mount unit: Unit run-mnt-ubuntu\x2dseed.mount already exists. [17:58] 8. thus we get stuck in an infinite loop where the-tool never exits because the-tool doesn't run systemd-mount with set -e, it explicitly runs systemd-mount with set +e first, then turns set -e back on after running systemd-mount [17:58] that makes perfect sense [17:59] cmatsuoka: so I have a few thoughts on how to fix this, 1) make the-tool depend on systemd-modules-load, 2) make the-tool eventually give up (because since it never gives up systemd never allows a debug shell to be started either) [17:59] 3) don't have a unit which runs fsck always on ubuntu-seed, instead have the-tool dynamically do that when snap-bootstrap requests something to be mounted [17:59] wdyt? [18:01] 2 looks like a good thing to have in case we run into problems again [18:02] 1 + 2 maybe? [18:02] yes actually I requested a very similar thing when mborzecki's original PR to ubuntu-core-initramfs was proposed, but I confused myself and said it wasn't a problem haha [18:02] (for 2 that is) [18:02] hahah [18:03] well I'm very happy now that this mystery is solved [18:03] it does make sense, i mean there's not gooing to me more than some limited number of mounts [18:04] zyga, hey, I also see this error https://paste.ubuntu.com/p/gcw9dTrRGm/ [18:07] alright I will propose a PR to ubuntu-core-initramfs to fix this, but I was going to file a LP bug first to summarize all this, but seems LP just died, so I will have to do that later [18:07] * ijohnson goes back to console-conf on core20 hacking [18:17] ack [18:19] cachio: please release the image anyway, I will send another change to address that [18:19] zyga, ok [18:26] PR snapd#8579 closed: packaging: stop depending on python-docutils [18:35] mvo, hi, any idea about this ppa ? https://paste.ubuntu.com/p/ttF5CHt56X/ [18:41] cachio: launchpad is/was down for a bit [18:42] ijohnson, ah, thanks [18:44]