[06:22] <mborzecki> morning
[06:23] <mvo> mborzecki: good morning
[06:37] <mvo> mborzecki: I cherry-picked 8575 to 2.44
[06:37] <mvo> mborzecki: that is needed, right? we will need a new 2.44.4 with a fix for syscalls with base: core20
[06:38] <mborzecki> mvo: thanks!
[06:38] <mborzecki> mvo: yes, it fixes the problems with libcrypto dependency being injected on centos
[06:39] <mborzecki> mvo: i'll need to add a similar change in downstream packaging for fedora/epel
[06:39] <mvo> ta
[06:40] <zyga> Good morning
[06:41] <zyga> How are you guys?
[06:41] <zyga> I will start late but I will look at opensuse a little today
[06:41] <zyga> I’ll fix our tests there and update the packaging /release
[06:42] <mvo> zyga: hey, I'm good, thanks!
[06:45] <mborzecki> qemu 5.0 is out, supports emulating tpm for arm
[06:45] <zyga> Groovy
[06:46] <zyga> Can we snap it? :-)
[06:52] <mborzecki> mvo: hmm 2.44 travis unit test job runs with different go version?
[06:53] <mborzecki> ah, it's go master
[06:56] <mvo> mborzecki: yeah, just removed that
[06:56] <mvo> mborzecki: I want to see the spread run
[06:57] <mborzecki> i'm out for a bit, and yay it's raining!
[07:04] <pstolowski> morning
[07:04] <mvo> hey pstolowski - good morning
[07:43] <juergh> mvo, zyga can one of you help me with that snapd download, please?
[08:01] <zyga> I will try
[08:01] <zyga> Which version do you need?
[08:22] <zyga> juergh: ^ ?
[08:29] <ogra> did xdg-open change recently ?
[08:29] <ogra> error: error running snapctl: Unknown command `user-open'. Please specify one command of: get, is-connected, restart, services, set, set-health, start, stop or unset
[08:33] <zyga> ogra: hmm, weird
[08:33] <zyga> ogra: perhaps a bug
[08:34] <ogra> i'm not yet sure, might be that the app changes ... the snapctl got me hooked though
[08:34] <ogra> *changed
[08:37]  * zyga breakfast
[08:42] <juergh> zyga, snapd 2.43.3
[08:42] <zyga> ok
[08:42] <zyga> give me a moment please
[08:51] <zyga> juergh: which architecture do you need?
[08:52] <juergh> zyga, amd64
[08:52] <zyga> juergh: hmm, but wait, 2.43.3 is latest stable
[08:52] <zyga> ah
[08:52] <zyga> sorry 44
[08:52]  * zyga needs coffee
[09:59]  * zyga looks at fixing the opensuse build issue now
[10:01] <tpreston> Hi, is there a reason why snapd /etc/profile.d/snapd.sh *appends* snap_bin_path to PATH, rather than prepends?
[10:01] <tpreston> https://forum.snapcraft.io/t/should-snap-bin-be-prepended-to-path/4506
[10:01] <zyga> tpreston: it's a more conservative choice, I suspect
[10:01] <tpreston> I have ctags installed at a system level (because of a package dependency), but I'd prefer to use snapd universal-ctags
[10:02] <tpreston> zyga: I guess so, maybe I should alias ctags then
[10:04] <doko> how can I make how can I make file:///usr/share/doc/python3-doc/html/index.html  work with the chromium snap?
[10:06] <zyga> doko: apart from serving it over http, I guess you cannot
[10:07] <zyga> doko: maybe copy it to ~ ?
[10:09] <doko> zyga: broken by design to access anything in /usr/share/doc ?
[10:09] <zyga> doko: not broken by design
[10:09] <zyga> doko: it's just not designed to access arbitrary host locations
[10:10] <zyga> doko: there are two elements at play, the view of the filesystem and the sandbox
[10:10] <doko> well, copying everything from /usr/share/doc to ~ is not viable
[10:10] <zyga> doko: chromium and your system see different filesystem
[10:10] <zyga> doko: and while /usr/share/doc is probably safe and could be allowed, it's not a general property
[10:10] <doko> is there a bug report about that?
[10:11] <zyga> doko: before you file one, do you have an idea of how it should work?
[10:11] <zyga> doko: I mean, I share the sentiment of it feeling broken
[10:11] <zyga> doko: but what could we do?
[10:13] <doko> well, for me it then not using the snap anymore
[10:13] <ogra> have a "host-documentation" interface that lallows reading /usr/share/doc content ?
[10:13] <zyga> doko: to partially answer my own question, at some point the document portal could be integrated into chromium, so that it could open a file
[10:13] <zyga> but IIRC the portal does not handle directories yet
[10:14] <zyga> doko: while I agee it's a regression I think using chromium to browse debian-style local documentation is fortunately niche
[10:15] <zyga> ogra: if you do that then the snap won't be able to see the /usr/share/doc from the base, I guess that's acceptable but rather obscure
[10:15] <zyga> ogra: but I think that's a good idea
[10:17] <ogra> yeah, i doubt the base will actually have content in /usr/shar/doc apart from copyright files
[10:17] <zyga> doko: I would strongly encourage you to file a bug or open a forum thread, I think ogra's idea is viable and well worth discussing
[10:17] <ogra> unless the building has changed massively (we used to remove everything there in the expectation that nobody would every want to use docs inside a base)
[10:18] <zyga> ogra: yes, I think we are lucky in a sense
[10:18] <zyga> if it was something other than that directory, it's probably not something that we could rely on
[10:18] <ogra> yeah
[10:20] <ogra> btw, using a portal would be horrible i fear, every link inside that index.html would pop up a new portal window
[10:20] <doko> zyga: where to file a bug?
[10:21] <zyga> ogra: I think there's a plan to open an entire directory as a portal
[10:21] <ogra> ah
[10:21] <zyga> doko: please file the bug on snapd on launchpad
[10:24] <doko> https://bugs.launchpad.net/snapd/+bug/1875860
[10:24] <mup> Bug #1875860: local documentation is not accessible from the chromium snap <regression> <snapd:New> <https://launchpad.net/bugs/1875860>
[10:26] <zyga> ogra: I added a comment explaining your idea
[10:26] <zyga> doko: thank you!
[10:26] <ogra> :)
[11:04] <ackk> hi, are these error coming from snapd itself  https://launchpadlibrarian.net/477374648/DpkgTerminalLog.txt ?
[11:08] <zyga> yes
[11:08] <zyga> ackk: udev in containers
[11:08] <zyga> ackk: it's a known topic that's on the backlog
[11:10] <ackk> zyga, what's the issue there?
[11:10] <zyga> ackk: snapd tries to reconfigure udev but udev is not meant to be used in containers (we pull it as a debian dependency) and it fails
[11:11] <ackk> zyga, for context, https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1875741 is the bug for that trace. basically a bionic->focal dist-upgrade with maas transitioning from deb to snap broke
[11:11] <mup> Bug #1875741: package maas 2.6.2-7841-ga10625be3-0ubuntu1~18.04.1 failed to install/upgrade: new maas package pre-installation script subprocess returned error exit status 1 <amd64> <apport-package> <focal> <third-party-packages> <uec-images> <maas (Ubuntu):New> <https://launchpad.net/bugs/1875741>
[11:11] <zyga> ackk: on subsequent runs snapd things everything is okay and no longer tries
[11:11] <ackk> zyga, so trying once again should fix it?
[11:11] <zyga> ackk: it's a topic we know about since Fra but we haven't managed to discuss at depth with Security
[11:11] <zyga> ackk: yes
[11:11] <zyga> ackk: there are some disagreements and we frankly didn't have time to sit down and discuss this
[11:12] <ackk> zyga, did anything change recently? I did test maas deb->snap transition in containers, never seen this issue
[11:12] <zyga> not that I know of
[11:13] <ackk> zyga, won't that break anyone dist-upgrading a container with snapd installed to focal? or is some kind of race that might happen?
[11:14] <zyga> ackk: it probably will
[11:14] <zyga> ackk: just a fire more distant than fires close up
[11:14] <sparkiegeek> zyga: got a bug number for it?
[11:14] <zyga> sparkiegeek: yes, let me find it
[11:14] <zyga> sparkiegeek: I tried addressing it before
[11:14] <zyga> one sec
[11:15] <ackk> zyga, sure, not implying anything, just trying to understand if there's something on the maas package side we could do, which it seems ther's not?
[11:17] <zyga> https://github.com/snapcore/snapd/pull/8219
[11:17] <mup> PR #8219: interfaces: use udev backend if udev socket exists <Security-High> <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/8219>
[11:17] <zyga> this is my idea and the bug is https://bugs.launchpad.net/snapd/+bug/1865503
[11:17] <mup> Bug #1865503: snapd (deb) fails to install snapd (snap) inside LXD on a buildd based image <core20> <snapd:In Progress by zyga> <https://launchpad.net/bugs/1865503>
[11:17] <zyga> the discussion there is interesting as well
[11:17] <zyga> my idea is that we should not depend on udev, not pull it in for containers, not use any udev when there's no real udev or real /dev
[11:19] <zyga> sparkiegeek, ackk: if you have opinions I'd love to learn from you as well
[11:20] <sparkiegeek> zyga: no educated opinion here :|
[11:20] <ackk> zyga, same here...
[11:31] <mup> PR snapd#8578 opened: interfaces: add host-docs interface <Created by zyga> <https://github.com/snapcore/snapd/pull/8578>
[11:58] <mborzecki> pstolowski: does your PR now work on centos 8 with the fix in master?
[12:19] <pstolowski> mborzecki: which PR?
[12:19] <mborzecki> pstolowski: don't remember :P one that was blocked by selinux basically, or was there none?
[12:20] <pstolowski> mborzecki: master was blocked
[12:20] <mborzecki> ahh cool
[12:22] <zyga> doko: https://github.com/snapcore/snapd/pull/8578
[12:22] <mup> PR #8578: interfaces: add host-docs interface <Created by zyga> <https://github.com/snapcore/snapd/pull/8578>
[12:22] <zyga> doko: it even has a test: https://github.com/snapcore/snapd/pull/8578/files#diff-aea859a8dd781e56f1c21b809b953d95R13
[12:31] <ogra> oh, look someone found the mvo backdorr in ubuntu core !!! once you create an mvo account you can never delete it anymore !! https://forum.snapcraft.io/t/login-and-delete-user-created-with-v2-create-user/17019
[12:33] <mborzecki> all devices belong to mvo ;)
[12:33] <ogra> yeah !
[12:33] <mvo> ogra: haha
[12:34] <ogra> i really wonder what makes people use some random accounts found on the internet for such stuff :)
[12:34] <mvo> it's baffling!
[12:37] <zyga> mvo: lol
[12:53] <zyga> mvo: I wonder if the user found this is our tests or if there is some documentation example
[12:54] <mvo> probably
[13:48] <zyga> I could use a review for rather simple https://github.com/snapcore/snapd/pull/8565
[13:48] <mup> PR #8565: osutil: expand FileLock to support shared locks and more <Created by zyga> <https://github.com/snapcore/snapd/pull/8565>
[14:02] <zyga> ok, i have my opensuse install under control now
[14:05] <pstolowski> +1
[14:05] <zyga> thanks!
[14:09] <zyga> ijohnson: do you have a moment  ^ it's pretty simple
[14:09] <zyga> ijohnson: it would unblock a host of other PRs
[14:11] <ijohnson> zyga: will look in a bit busy atm
[14:11] <zyga> sure
[14:17] <mborzecki> mvo: pstolowski: do you think we can land https://github.com/snapcore/snapd/pull/8536 to unblock the rest? with any tweaks in the followups (there's a bunch of PRs stack on top of this one)
[14:17] <mup> PR #8536: store,asserts,many: support the new action fetch-assertions <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8536>
[14:17] <mborzecki> hm pedronis isn't around
[14:18] <zyga> brb
[14:18] <zyga> tea
[14:19] <pstolowski> mborzecki: best to ask him actually... don't know if there is any risk
[14:29] <mup> PR snapd#8536 closed: store,asserts,many: support the new action fetch-assertions <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8536>
[14:29] <mvo> mborzecki: I think it makes sense, done
[14:29] <mborzecki> thanks!
[14:29] <zyga> jdstrand: https://github.com/snapcore/snapd/pull/8578/files#diff-aea859a8dd781e56f1c21b809b953d95R13
[14:29] <mup> PR #8578: interfaces: add host-docs interface <Created by zyga> <https://github.com/snapcore/snapd/pull/8578>
[14:30] <jdstrand> zyga: cool thanks. I have it on my list to review, but note I'm sprinting/etc
[14:31] <zyga> no rush :)
[14:57] <ijohnson> cmatsuoka: I hit the infinite loop of failed to transient mount unit for ubuntu-seed you had the other day
[15:13] <cmatsuoka> ijohnson: it's quite infrequent, but it's something we'll have to fix
[15:13] <ijohnson> cmatsuoka: did the issue go away when you rebooted ?
[15:14]  * ijohnson would like to debug but is afraid that when rebooting the issue goes away
[15:14] <cmatsuoka> ijohnson: yes, when I booted the same image again it worked as expected
[15:14] <cmatsuoka> looks racy
[15:14] <ijohnson> mmm yeah
[15:17] <ijohnson> ah actually I can consistently reproduce this now on my image
[15:17]  * ijohnson debugs
[15:18] <cmatsuoka> oh that's good, I couldn't reproduce it
[15:23]  * cachio lunch
[15:25] <ijohnson> cmatsuoka: I video recorded the boot and caught this in the first few seconds of the log
[15:25] <ijohnson> https://usercontent.irccloud-cdn.com/file/84WtzfpC/loop%20uc20%20edge.png
[15:26] <ijohnson> the last line `systemd-fsck CP437 Invalid argument` is suspicious
[15:26] <cmatsuoka> oh interesting
[15:26] <cmatsuoka> yes, very suspicious
[15:27] <ijohnson> https://usercontent.irccloud-cdn.com/file/uYCPMgpF/loop%20uc20%20later.png
[15:28] <ijohnson> cmatsuoka: then this is also suspicious where run-mnt-ubuntu\x2dseed.mount fails because wrong fs type, bad option, etc.
[15:29] <cmatsuoka> ijohnson: maybe you could inspect the image file? (it could be a good idea to make a copy first in case you end up fixing the problem)
[15:30] <cmatsuoka> ijohnson: and check if the partition contents are there or the filesystem seems right
[15:30] <ijohnson> cmatsuoka: yes I am doing that now
[15:30] <ijohnson> the partition was mounted fine with kpartx + mount of the loopback device
[15:31] <cmatsuoka> does it look good?
[15:31] <ijohnson> yeah it has all the expected files and such
[15:31] <cmatsuoka> hmm
[15:31] <ijohnson> should I try to run fsck on the partition in the .img file ?
[15:32] <cmatsuoka> yeah, let's see what happens
[15:34] <ijohnson> huh fsck.etx4 says ubuntu-seed is actually a vfat partition?
[15:34] <ijohnson> https://www.irccloud.com/pastebin/C2v5Dpoz/
[15:35] <cmatsuoka> mm, yes, seed is vfat
[15:35] <ijohnson> really? I thought seed was ext4 on amd64 ?
[15:35] <cmatsuoka> we must EFI boot it
[15:36] <ijohnson> ah okay so I was just wrong about that then
[15:36] <cmatsuoka> seed is vfat, the rest is ext4
[15:36] <ijohnson> ok, fsck.vfat doesn't complain about the partition at all then
[15:36] <ijohnson> cmatsuoka: anything else I should check about the partition you think?
[15:37] <cmatsuoka> did it complain about code pages or something like that?
[15:37] <cmatsuoka> or maybe systemd-fsck does something different?
[15:38] <cmatsuoka> does it still fail if you boot the image again?
[15:42] <ijohnson> cmatsuoka: yes it reliably fails every time I boot now
[15:42] <ijohnson> cmatsuoka: all I got from fsck.vfat output was this:
[15:43] <ijohnson> https://www.irccloud.com/pastebin/nuIWXTi1/
[15:43] <cmatsuoka> hmm
[15:44] <ogra> xnox, when you disabled the ability to disaable console-conf in ubuntu-image, did you make sure it still works for core18 builds ?
[15:44] <ogra> (someone just said it is completely gone, we have customers usign that option to build their images)
[15:46] <cmatsuoka> ijohnson: so the fs is good but for some reason it's not being mounted
[15:47] <ijohnson> yes
[15:47] <ijohnson> I can mount it fine though when the image is not booting
[15:47] <ijohnson> actually you know what, since this is still in install mode I can change the kernel cmdline since we never make it out of the initrd
[15:47] <ijohnson> so unsealing doesn't affect anything at this point
[15:48] <cmatsuoka> yes, you're right, let's inspect it while it's happening
[15:48] <ogra> xnox, ah, nevermind ... it still works but the --help info about it is gone
[15:49] <ijohnson> well damn now it boots fine
[15:50] <cmatsuoka> did you make a copy of the original image?
[15:50] <ijohnson> yes I did
[15:50] <ijohnson> let me make a copy of that original image and try again
[15:51] <ijohnson> ah now the copy of the original broken image works too :-/
[15:51] <cmatsuoka> oh noes
[15:52] <ijohnson> so this definitely seems like a race
[15:54] <cmatsuoka>  /o\
[15:57] <ijohnson> well on the other hand, I notice that in the log of the failed boot that goes into this loop there's this message too:
[15:57] <ijohnson> [    1.342368] FAT-fs (vda2): IO charset iso8859-1 not found
[15:57] <ijohnson> and that message doesn't appear in a good boot
[15:57] <ijohnson> so perhaps it's a race with some kernel module being loaded?
[15:58] <cmatsuoka> that sounds plausible
[15:58] <cmatsuoka> but then if it's desperately retrying, at some point it should work
[15:59] <cmatsuoka> ijohnson: https://askubuntu.com/questions/372445/fat-fs-io-charset-iso8859-1-not-found-error-while-mounting-fat-drives
[16:01] <ijohnson> cmatsuoka: what's so weird about this issue, is that it finishes the fsck check on /dev/vda2 in the loop case
[16:01] <ijohnson> ah but it wouldn't be loading the kernel module from that partition at this point, it would be loading it from the initrd in the kernel
[16:02] <ijohnson> so perhaps that kernel module in the initrd is corrupted somehow?
[16:02] <cmatsuoka> but then why it would work in the next boot?
[16:03] <ijohnson> well tbh I didn't quite make a perfect backup of the original image that was always reproducing, what I did was this:
[16:04] <ijohnson> 1. tried booting same img over and over again always reproducing infinite loop
[16:04] <ijohnson> 2. tried mounting the img file with kpartx and ran fsck.ext4 on it as explained
[16:04] <ijohnson> 3. unmounted with kpartx and made a backup copy
[16:04] <ijohnson> 4. re-mounted it and tried with fsck.vfat
[16:04] <ijohnson> 5. booted it and it didn't work first time
[16:04] <ijohnson> 6. rebooted it and changed kernel cmdline in grub menu, now it works every time
[16:05] <cmatsuoka> well, it still didn't boot in step 5
[16:06] <ijohnson> yes but somehow changing the kernel cmdline made it boot fine
[16:06] <cmatsuoka> that's weird
[16:06] <ijohnson> I set the kernel cmdline to this: `console=tty1 console=ttyS0 rd.systemd.journald.forward_to_console=1 systemd.journald.forward_to_console=1 rd.systemd.debug_shell=1 dangerous panic=-1`
[16:07] <ijohnson> let me try re-creating this image fresh the way I did originally
[16:08] <cmatsuoka> I'll break for lunch and hopefully have a good idea while eating
[16:08] <cmatsuoka> because right now I don't have any
[16:09] <ijohnson> yeah just doing same thing again to build the image didn't reproduce it
[16:09]  * ijohnson runs it in a loop
[16:11] <ijohnson> hey I reproduced it again!
[16:11]  * ijohnson saves that img
[16:14] <zyga> you guys are having some fun :)
[16:14] <ijohnson> uc20 is too much fun
[16:17] <ijohnson> alright so every few time it hits that loop and fails, I see the IO charset iso8859-1 not found message, but when it does boot, I don't see that message, so it definitely is related
[16:26] <xnox> ogra_:  i didn't change anything in ubuntu-image, and do not do any ubuntu-image development there.
[16:26] <xnox> ogra_:  open an issue against ubuntu-image repository, with any bugs.
[16:27] <thresh> hi.  I see my snapcraft push nightlies/vlc_*.snap --release beta, has succeeded in my CI, and I also see it under "Latest revisions for amd64" list.  However it's not in the beta channel.
[16:27] <thresh> why would that happen, on the store?
[16:28] <zyga> thresh: this is more of a snapcraft question but perhaps snapcraft devs are around as well
[16:29] <thresh> thanks zyga, I'll complain in the related channel too :-)
[16:31] <zyga> thresh: I don't use snapcraft that often but I usually separately upload and release
[16:47] <zyga> cachio: are tumbleweed images updated periodically somehow or are we running a snapshot until something happens explicitly?
[16:50] <cachio> zyga, failed yesterday
[16:50] <zyga> sorry, I don't understand
[16:51] <cachio> let me try to fix it
[16:51] <zyga> cachio: can you please answer my question :-)
[16:51] <cachio> I mean
[16:51] <zyga> I' just want to understand what the process is
[16:52] <zyga> you can tell me what failed and about fixing it as well
[16:52] <zyga> but that's not what I was asking about
[16:52] <cachio> the process is:
[16:52] <cachio> the image is updated to a tamporal image
[16:52] <cachio> then the snapd tests are executed
[16:53] <cachio> if tests pass, then we turn that temporal image in a test image
[16:53] <cachio> otherwise the image is discarded
[16:53] <cachio> yesterday it failed running tests
[16:54] <cachio> failed preparing the the snapd teest suite
[16:54] <zyga> is there any notification when that happens?
[16:54] <cachio> no
[16:54] <zyga> that's not great, can we try go get some
[16:54] <cachio> it is in the brnaches board
[16:54] <cachio> ans it happens every monday
[16:55] <zyga> what is "branches board"?
[16:55] <cachio> https://travis-ci.org/github/snapcore/spread-cron/branches
[16:55] <cachio> https://travis-ci.org/github/snapcore/spread-cron/branches
[16:55] <cachio> zyga, this is the last log https://travis-ci.org/github/snapcore/spread-cron/builds/680087171#L2749
[16:56] <cachio> sometimes I manually run the jobs to update the images
[16:57] <cachio> zyga, I see this Package 'python-docutils' not found.
[16:58] <zyga> I see, I think this package just got removed, it's a python 2 package
[16:59] <cachio> zyga, also 'pkg-config' not found in package names.
[17:00] <zyga> cachio: why are we installing python-docutil?
[17:00] <zyga> I don't see it in anything related to opensuse
[17:00] <zyga> cachio: yes, that's also been removed and replaced by something else
[17:01] <cachio> zyga, this is the snapd test suite which fails
[17:01] <zyga> cachio: python-docutils is only listed in arch
[17:01] <zyga> cachio: why are we installing python-docutils, do you know?
[17:01] <cachio> zyga, don't know why it is installed
[17:01] <cachio> I guess  a test need it
[17:01] <ogra> xnox, oh, sorry, GH tricked me ... you were just a reviewer on https://github.com/CanonicalLtd/ubuntu-image/pull/186
[17:01] <mup> PR CanonicalLtd/ubuntu-image#186: Some options are unsupported for UC20 builds <Created by sil2100> <Merged by sil2100> <https://github.com/CanonicalLtd/ubuntu-image/pull/186>
[17:01] <zyga> cachio: do you know which part of the code is installing it? I certainly don't see it in pkgdb.sh
[17:02] <zyga> I see it in opensuse dependencies but that's not managed by pkgdb.sh
[17:04] <zyga> cachio: ok, I see the problem now
[17:04] <cachio> packaging/opensuse-15.0/snapd.spec:
[17:04] <zyga> cachio: I think for sid, tumbleweed and arch we should not stop
[17:04] <zyga> cachio: if we stop and use some ancient version because tests don't pass
[17:04] <zyga> that's pretty much because snapd is in a way broken
[17:04] <zyga> the world has moved on
[17:05] <zyga> staying on an old image without alerting the team could be a problem later
[17:05] <zyga> in a sense those systems are all unstable because they never stop
[17:05] <cachio> zyga, for arch is not a problem because we update in every run the image
[17:05] <zyga> cachio: regardless of how we update
[17:06] <cachio> zyga, but I could remove snapd tests validation for tumbleweed and sid
[17:06] <cachio> it is very easy
[17:06] <zyga> cachio: yeah, I think we should do that
[17:06] <zyga> cachio: we can have them in unstable systems
[17:06] <cachio> zyga, yes
[17:06] <zyga> cachio: in reality, the package is gone from the archive
[17:06] <cachio> totally agree
[17:06] <zyga> cachio: not updating a stale image is not buying us anything
[17:06] <zyga> cachio: the package cannot be built
[17:06] <zyga> cachio: CI is not functioning as CI :)
[17:07] <zyga> cachio: cool, I'll send a patch to fix our packaging
[17:07] <cachio> zyga, nice, thanks!
[17:07] <zyga> cachio: but please let's figure out how to make some notifications flash on everyone's systems
[17:07] <zyga> cachio: so that next time we know earlier
[17:07] <cachio> zyga, sure, I'll configure to send emails
[17:07] <zyga> super, thank you :)
[17:07] <cachio> zyga, yaw
[17:17] <zyga> cachio: https://github.com/snapcore/snapd/pull/8579
[17:17] <mup> PR #8579: packaging: stop depending on python-docutils <Created by zyga> <https://github.com/snapcore/snapd/pull/8579>
[17:17] <mup> PR snapd#8579 opened: packaging: stop depending on python-docutils <Created by zyga> <https://github.com/snapcore/snapd/pull/8579>
[17:22]  * zyga goes to make coffee ahead of the meeting
[17:28] <cachio> zyga, thanks
[17:28] <cachio> lets test it whith the new image once it is ready
[17:31] <zyga> sure
[17:49] <kyrofa> If someone runs `snap revert`, will it run the refresh hooks?
[17:49] <ijohnson> kyrofa: afaik it won't
[17:50] <kyrofa> ijohnson, any hooks? Any way the snap to which is being reverted can know that a revert happened?
[17:50] <ijohnson> no, I think that's by design that the snap doesn't know it got reverted
[17:51] <kyrofa> Fair enough, thanks ijohnson
[17:53] <ijohnson> cmatsuoka: ah I think I understand the problem now
[17:53] <ijohnson> cmatsuoka: what's happening is this sequence of events:
[17:54] <ijohnson> 1. there is a systemd unit for running fsck on /dev/disk/by-label/ubuntu-seed that just always runs, so it happens to run first
[17:55] <ijohnson> 2. snap-bootstrap via "the-tool" runs, says to mount /run/mnt/ubuntu-seed _with transient unit_ (that's important)
[17:55] <ijohnson> 3. that fails because systemd-modules-load has not run, or finished running yet
[17:55] <ijohnson> 4. so we see IO charset iso8859-1 not found from the kernel because nls_iso8859_1 was not loaded
[17:56] <ijohnson> 5. systemd-modules-load eventually runs/finishes running and does load nls_iso8859_1: systemd-modules-load[238]: Inserted module 'nls_iso8859_1'
[17:56] <ijohnson> 6. the-tool keeps running snap-bootstrap, which says to mount /run/mnt/ubuntu-seed, however because of how the-tool tries to do the mount, it just requests a transient unit for the mount
[17:57] <ijohnson> 7. the transient unit was already started/created by systemd so systemd doesn't try again and just complains thusly: the-tool[380]: Failed to start transient mount unit: Unit run-mnt-ubuntu\x2dseed.mount already exists.
[17:58] <ijohnson> 8. thus we get stuck in an infinite loop where the-tool never exits because the-tool doesn't run systemd-mount with set -e, it explicitly runs systemd-mount with set +e first, then turns set -e back on after running systemd-mount
[17:58] <cmatsuoka> that makes perfect sense
[17:59] <ijohnson> cmatsuoka: so I have a few thoughts on how to fix this, 1) make the-tool depend on systemd-modules-load, 2) make the-tool eventually give up (because since it never gives up systemd never allows a debug shell to be started either)
[17:59] <ijohnson> 3) don't have a unit which runs fsck always on ubuntu-seed, instead have the-tool dynamically do that when snap-bootstrap requests something to be mounted
[17:59] <ijohnson> wdyt?
[18:01] <cmatsuoka> 2 looks like a good thing to have in case we run into problems again
[18:02] <cmatsuoka> 1 + 2 maybe?
[18:02] <ijohnson> yes actually I requested a very similar thing when mborzecki's original PR to ubuntu-core-initramfs was proposed, but I confused myself and said it wasn't a problem haha
[18:02] <ijohnson> (for 2 that is)
[18:02] <mborzecki> hahah
[18:03] <cmatsuoka> well I'm very happy now that this mystery is solved
[18:03] <mborzecki> it does make sense, i mean there's not gooing to me more than some limited number of mounts
[18:04] <cachio> zyga, hey, I also see this error https://paste.ubuntu.com/p/gcw9dTrRGm/
[18:07] <ijohnson> alright I will propose a PR to ubuntu-core-initramfs to fix this, but I was going to file a LP bug first to summarize all this, but seems LP just died, so I will have to do that later
[18:07]  * ijohnson goes back to console-conf on core20 hacking
[18:17] <zyga> ack
[18:19] <zyga> cachio: please release the image anyway, I will send another change to address that
[18:19] <cachio> zyga, ok
[18:26] <mup> PR snapd#8579 closed: packaging: stop depending on python-docutils <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8579>
[18:35] <cachio> mvo, hi, any idea about this ppa ? https://paste.ubuntu.com/p/ttF5CHt56X/
[18:41] <ijohnson> cachio: launchpad is/was down for a bit
[18:42] <cachio> ijohnson, ah, thanks
[18:44] <mvo> cachio: looks like a LP issue
[18:46] <cachio> mvo, yes
[18:46] <cachio> thanks
[19:12] <mup> PR snapd#8580 opened: data/completion: add `snap` command completion for zsh <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8580>
[19:13] <mborzecki> any zsh users here?
[19:34] <mup> PR snapcraft#3095 closed: plugins: break out rosdep resolve parsing for external use <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3095>
[19:34] <mup> PR snapcraft#3098 opened: build providers: bootstrap with dirmngr <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3098>
[19:45] <cmatsuoka> ijohnson: just hit the problem again here, and in the next boot it's gone
[20:05] <ijohnson> cmatsuoka: yes I'm not sure what changed, but something seems to make it more likely to happen recently as I never saw it before yesterday
[20:06] <cmatsuoka> and it seems that I'm having some dns issues with the store
[20:10] <ijohnson> there was a big outage with launchpad a while ago that maybe propagated to the store? I dunno store seems a little slow but otherwise operational on my end
[20:12] <cmatsuoka> right now I can't resolve snapcraft.io
[20:13] <Lukewh> seems ok here cmatsuoka where are you based?
[20:13] <cmatsuoka> I'm in brazil
[20:13] <cmatsuoka> oh, and now it's back
[20:15] <cmatsuoka> a few days ago there were some routing problems from brazil to the rest of the world so it can be something weird on my side again
[20:16] <mup> PR snapcraft#3099 opened: repo: always refresh cache when build packages are required <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3099>
[20:17] <cmatsuoka> oh, it seems that the ipv4 addresses are resolving correctly but AAAA is not for some reason
[20:18] <cmatsuoka> anyway, let's just use ipv4
[20:22] <zyga> I will fix the damn pulseaudio test tomorrow
[20:22] <zyga> now that other stuff is not red anymore
[20:23] <mup> PR snapd#8565 closed: osutil: expand FileLock to support shared locks and more <Created by zyga> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/8565>
[20:24] <zyga> \o/
[20:24] <zyga> thank you Ian :)
[20:27] <cjwatson> ijohnson: it was a Canonical network issue, not specifically LP, so there'll have been various bits of fallout
[20:27] <ijohnson> yaw zyga
[20:28] <ijohnson> cjwatson: yeah that's kinda what I expected
[20:28] <cjwatson> (GS2 I think)
[20:28] <zyga> ijohnson: https://github.com/snapcore/snapd/pull/8566/files is next on the r-a-a pipe
[20:28] <zyga> but it has probably-poor names
[20:28] <mup> PR #8566: cmd/cmdutil: add run inhibition operations <Created by zyga> <https://github.com/snapcore/snapd/pull/8566>
[20:28] <zyga> I was split between lock naming
[20:29] <zyga> and totally not lock naming
[20:29] <ijohnson> mmm zyga does that need samuele review for names?
[20:29] <zyga> ijohnson: for sure but we can do the technical review
[20:29] <ijohnson> ack will try to get to it today, but probably that one will have to wait for tomorrow
[20:30] <zyga> ijohnson: or maybe better this:
[20:30] <zyga> https://github.com/snapcore/snapd/pull/8571
[20:30] <zyga> this is self-sufficient
[20:30] <mup> PR #8571: overlord/snapstate: warn of refresh/postpone events <Created by zyga> <https://github.com/snapcore/snapd/pull/8571>
[20:30] <zyga> and useful today
[20:30] <zyga> and actually tiny apart from tests :)
[20:30] <zyga> just two print-like calls
[20:31] <zyga> but I should EOD
[20:31] <zyga> or I'll start sending more things
[20:32]  * zyga waves
[20:32] <ijohnson> haha
[20:32] <ijohnson> have a good night zyga
[20:32] <zyga> one last thing: https://github.com/snapcore/snapd/pull/8572#event-3283181589
[20:32] <zyga> I like this line
[20:32] <mup> PR #8572: packaging: add the inhibit directory <Created by zyga> <https://github.com/snapcore/snapd/pull/8572>
[20:32] <zyga> "via automation"
[20:43] <zyga> meh
[20:43] <zyga> I fixed the stupid test
[20:58] <zyga> ijohnson: https://github.com/snapcore/snapd/pull/8581 draft for now
[20:58] <mup> PR #8581: tests: port pulseaudio test to session-tool <Created by zyga> <https://github.com/snapcore/snapd/pull/8581>
[20:59] <mup> PR snapd#8581 opened: tests: port pulseaudio test to session-tool <Created by zyga> <https://github.com/snapcore/snapd/pull/8581>
[21:09] <zyga> so, session-tool on 16.04 - it needs to start doing upstart things, I guess
[21:11] <ijohnson> sad
[21:11] <ijohnson> that's unfortunate
[21:12]  * ijohnson EODs early
[22:35] <mup> Issue core20#48 closed: python3 from base causes segfault for confined snaps on armhf <question> <Created by sergiusens> <Closed by xnox> <https://github.com/snapcore/core20/issue/48>