[05:43] <mborzecki> morning
[06:46] <mup> PR snapd#8636 closed: tests: add dependency needed for next upgrade of bionic <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8636>
[06:48] <mup> PR snapd#8634 closed: usersession/agent,wrappers: fix races between Shutdown and Serve <Test Robustness> <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8634>
[06:49] <mborzecki> mvo: hey
[06:49] <mvo> good morning mborzecki
[06:50] <mup> PR snapd#8630 closed: tests: inject snapd from edge into seeds of the image in manual preseed test <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8630>
[06:53] <mup> PR snapd#8617 closed: tests: new directory used to store the cloud images on gce <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8617>
[06:54] <mup> PR snapd#8611 closed: tests: not fail when boot dir cannot be determined <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8611>
[07:00] <pstolowski> morning
[07:06] <mvo> good morning pstolowski
[07:13] <mup> PR snapd#8577 closed: secboot,cmd/snap-bootstrap: move initramfs-mounts tpm access to secboot <UC20> <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8577>
[07:17] <mvo> mborzecki: there is one comment from ian in 8628, is that still relevant to address? this looks like a fine 2.45 piece
[07:18] <zyga> o/
[07:18] <mborzecki> mvo: yes, i'll be pushing a patch there, filepath.Glob() sorts internally but it isn't mentioned in the doc so it's fair to call it an implementation detail and just sort eplxiciltly outside
[07:18] <zyga> hey guys
[07:22] <mborzecki> zyga: hey
[07:23] <mvo> mborzecki: thanks, I want to cherry pick this to 2.45 so I will wait for this update first :9
[07:23] <mvo> hey zyga
[07:35] <mborzecki> mvo: updated #8628
[07:35] <mup> PR #8628: cmd/snap: coldplug auto-import assertions from all removable devices <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8628>
[07:40] <mvo> mborzecki: thank you
[08:18] <mborzecki> zyga: something random? https://paste.ubuntu.com/p/V7PhBg73zj/
[08:18] <zyga> 1;1
[08:19] <mborzecki> btw. 14.04 deb does not build anymore?
[08:19] <mborzecki> https://paste.ubuntu.com/p/f4CdkXsVwm/ getting 2020-05-11T07:32:20.0517313Z E: Unable to locate package dbus-user-session on 14.04
[08:21] <mborzecki> mvo: the ci job in #8628 has finished, please take a look and merge, there's a couple of unrelated failures in the tests
[08:21] <mup> PR #8628: cmd/snap: coldplug auto-import assertions from all removable devices <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8628>
[08:22] <mvo> mborzecki: thanks, in a meeting
[08:32] <zyga> mborzecki: maybe new image?
[08:33] <zyga> I think it's the new image with a new package preinstalled
[08:37] <zyga> re
[08:37] <mborzecki> zyga: we try to install dbus-user-session in prepare, but there's no such package on 14.04 apparently
[08:37] <zyga> mborzecki: is it on master now
[08:37] <mborzecki> zyga: yeah, it's on master
[08:37] <zyga> mborzecki: is that on 14.04?
[08:37] <zyga> mborzecki: IIRC it only was a factor on 16.04
[08:37] <zyga> ubuntu-16.04-64
[08:37] <zyga> the log says 16.04
[08:37] <zyga> sergio preinstalled that package
[08:37] <zyga> it landed this morning
[08:37] <zyga> so we just need a matching test correction
[08:38] <zyga> the test measures what's the state of our image
[08:38] <zyga> so that we understand what to do in tests and what to expect
[08:38] <mborzecki> zyga: i'm confused, which log?
[08:38] <zyga> ah
[08:38] <zyga> you said: "something random? ..."
[08:38] <mborzecki> zyga: that's separate issue :)
[08:38] <zyga> right
[08:39] <zyga> yeah, this package does not exist in 14.04
[08:39] <zyga> we should not try to install it there
[08:39] <zyga> hmm, didn't this land after a spread run?
[08:39] <zyga> anyway, both need to be fixed
[08:39] <mvo> zyga: 8633 is the one that fails in core with the strange ownership, right?
[08:40] <zyga> mvo: yes
[08:40] <mvo> zyga: so if I run this in core16 I should see the same failure?
[08:40] <zyga> mvo: checkout that PR and run core16 spread tests
[08:40] <zyga> yeah
[08:40] <mvo> zyga: I wonder if it also fails on qemu, maybe it's really something cloud specific
[08:40] <zyga> mvo: aaah
[08:40] <zyga> brilliant idea
[08:40] <zyga> I'm sure it will be a decisive test for the cloud-init theory
[08:42] <mvo> zyga: running it now locally here
[08:42] <zyga> ok, I'll grab one thing from upstairs and look at the thing maciek reported
[08:46] <zyga> ok
[08:46] <zyga> let's get to it
[08:50] <pedronis> #8613 needs a 2nd review and is fairly straightforward
[08:50] <mup> PR #8613: o/configcore: pass extra options to FileSystemOnlyApply <Created by stolowski> <https://github.com/snapcore/snapd/pull/8613>
[09:09] <zyga> mborzecki: testing a fix, should be available shortly
[09:09] <mborzecki> zyga: thanks
[09:20] <mborzecki> pedronis: mvo: should i start looking into snap repair for uc20 perhaps or the cli bits for recovery systems?
[09:21] <mup> PR snapd#8637 opened: tests: test adjustments and fixes for recently published images <Created by zyga> <https://github.com/snapcore/snapd/pull/8637>
[09:22] <zyga> mborzecki: ^
[09:22] <pedronis> mborzecki: probably reviews
[09:26] <pedronis> mborzecki: also I think Ian PR can panic in some cases: https://github.com/snapcore/snapd/pull/8635#discussion_r422906005
[09:26] <mup> PR #8635: o/devicestate: support doing system action reboots from recover mode <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8635>
[09:26] <mborzecki> pedronis: ok, let me check that, i can push a fix if needed
[09:36] <mup> PR snapd#8628 closed: cmd/snap: coldplug auto-import assertions from all removable devices <Squash-merge> <UC20> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8628>
[09:36] <mborzecki> yay
[09:36] <mvo> mborzecki: thank *you* - and cherry-picked to 2.45
[09:37] <mborzecki> mvo: anything else we need to do for auto import right now? (aside from little tweaks)
[09:37] <mvo> mborzecki: I think that's it, this should work now
[09:37] <mvo> mborzecki: once it's on beta/edge I will ask taipei to test it
[09:37] <mborzecki> mvo: great, thanks!
[09:45] <pedronis> we broke 14.04 testing ?
[09:45] <zyga> pedronis: yes
[09:45] <zyga> pedronis: fix is up
[09:45] <pedronis> :/
[09:45] <zyga> 8637
[09:46] <zyga> pedronis: I updated https://github.com/snapcore/snapd/pull/8566
[09:46] <mup> PR #8566: c/snaplock/runinhibit: add run inhibition operations <Created by zyga> <https://github.com/snapcore/snapd/pull/8566>
[09:47] <zyga> pedronis: it was merged earlier today https://github.com/snapcore/snapd/pull/8636
[09:48] <mup> PR #8636: tests: add dependency needed for next upgrade of bionic <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8636>
[09:53] <pedronis> zyga: again a pulseaudio failure as well: https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/4124/signedlogcontent/31?urlExpires=2020-05-11T09%3A53%3A40.8345409Z&urlSigningMethod=HMACV1&urlSignature=CGXZSIWuB1%2B%2FEaA9iyF9qXEMhUFFa5d0nk8fHPh%2BXic%3D
[09:54] <zyga> that's interesting, it looks like a real issue, look that within the same test we did interact with pulse already
[09:54] <pedronis> and also issues with session agent on core16
[09:54] <zyga> I have an idea about that failure
[09:54] <zyga> pedronis: is that a random or have you seen this more than once?
[09:55] <pedronis> is the one when we cannot talk to the socket
[09:55] <pedronis> but this time is the a different socket not root
[09:55] <pedronis> read unix @->/run/user/12345/snapd-session-agent.socket: read: connection reset by peer
[09:57] <zyga> do you think that the agent is crashing or just shutting down at the wrong moment?
[09:58] <zyga> pedronis: perhaps a debug line would help, something like
[09:58] <zyga> session-tool -u test journalctl --user
[10:02] <zyga> brb,
[10:18] <mborzecki> pedronis: hm perhaps we should error out when trying to request a system action and the system is not yet fully seeded
[10:19] <mborzecki> pedronis: looking at current code, the 'current' system would also be incorrectly detected in such case
[10:19] <mborzecki> so any systemLabel == current.Label checks make no sense at all
[10:19] <pedronis> mborzecki: well, it's complicated
[10:19] <pedronis> you don't get any run or recover
[10:19] <pedronis> action
[10:19] <pedronis> if there's no current I think
[10:20] <mborzecki> pedronis: yes, that also
[10:20] <pedronis> it's install that is strange
[10:20] <pedronis> honestly I think we should just error if you try to install while installing
[10:21] <pedronis> even if it's the same system
[10:21] <pedronis> is just a strange request
[10:21] <mborzecki> maybe just error out early if in install mode
[10:21] <mborzecki> yeah
[10:27] <zyga> re
[10:28] <zyga> pedronis: we could do two things that would have better perception of working
[10:28] <zyga> 1) if we are already performing an operation that is being requested (.e.g. refresh while refreshing) we could just display the progress instead of failing
[10:28] <zyga> this sometimes fails when you open gnome software and it tries to refresh snaps but we're already doing that
[10:29] <zyga> 2) we could perhaps queue a install command if we are in a special state (not seeded, but perhaps other states as well)
[10:34] <zyga> https://github.com/snapcore/snapd/pull/8637 fixes master and needs a 2nd review
[10:35] <mup> PR #8637: tests: test adjustments and fixes for recently published images <Created by zyga> <https://github.com/snapcore/snapd/pull/8637>
[10:43] <pedronis> zyga: this is about mode, not normal snap operations
[10:43] <zyga> I see
[10:43] <zyga> so run vs other?
[10:45] <pedronis> mborzecki: actually, I was confused, we set things into seeded-systems even for the ephemeral ones?
[10:46] <pedronis> mborzecki: it sounds slightly wrong (in terms of defining what is current)
[10:46] <zyga> mvo: did your qemu experiment turn up passing or failing?
[10:47] <mborzecki> pedronis: yes, it's set in mark seeded, which ephemeral does too for itself
[10:47] <mvo> zyga: it failed but in a different test :( in snap-user-service-socket-activation on uc16
[10:47] <pedronis> mborzecki: it sounds kind of wrong
[10:47] <mvo> zyga: sounds like I need to run it again
[10:47] <zyga> mvo: that's a known warning
[10:47] <pedronis> mborzecki: we need to think a bit
[10:47] <zyga> mvo: dit it fail on just that? roughly how many tests did you get throguh?
[10:48] <zyga> mvo: in GCE I see the failure really quickly
[10:48] <pedronis> mvo: all kind of snap-user-service tests fail sometimes on UC16
[10:54] <mborzecki> pedronis: upated #8635
[10:54] <mup> PR #8635: o/devicestate: support doing system action reboots from recover mode <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8635>
[10:55] <pedronis> mborzecki: thx
[10:58] <mvo> zyga: around ~100, I can run it again on gce
[10:58] <zyga> mvo: I'm running int on GCE now with some more ideas
[10:58] <zyga> mvo: but I think this shows it's cloud-init and, presumably, around tests that reboot
[10:58] <zyga> mvo: the naive workaround I added just fixes one reboot case (the initial reboot)
[10:59] <zyga> need to think where to put a fix for something that can happen at any time
[10:59] <zyga> I wonder if we could turn off cloud-init somehow as well, not sure if it should do stuff on core during CI
[11:02] <pedronis> in which sense? some our CI flows use cloud-init
[11:02] <zyga> pedronis: it seems that cloud init data in GCE causes creation of an "ubuntu" user with broken home directory
[11:02] <zyga> but only on core16/18
[11:02] <zyga> (though only *broken* there, it may be that we create it everywhere0
[11:03] <pedronis> well, it's easy to know
[11:03] <pedronis> look what's the user data there
[11:03] <zyga> at cloud-init data? I'm not deeply familiar with it but I looked that some on Friday and
[11:04] <zyga> 1) cloud-init.service sets up the password for ubuntu user, so it's likely it is creating it as well (this is evident in the log file)
[11:04] <zyga> 2) user data contains gustavo's account but I was not able to find the string "ubuntu" there - perhaps I looked at the wrong files as I'm not familiar with cloud-init that much
[11:05] <pedronis> ok, but I doubt turning it off completely is an option
[11:05] <zyga> I see
[11:06] <pedronis> anywway I have no time to help with this atm
[11:10] <pedronis> zyga: is this breaking some tests directly? or it's breaking some sanity checks?
[11:10] <zyga> just a sanity check
[11:10] <zyga> I was wondering where it came from
[11:11] <pedronis> given all the things we have to fix it, seems it should go in the ignore for now bucket
[11:13] <zyga> indeed,
[11:16] <mvo> zyga: running it again in qemu/google on core18, maybe that is more robust
[11:22] <zyga> ok, the 14.04 issue is gone
[11:23] <mup> PR snapd#8637 closed: tests: test adjustments and fixes for recently published images <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8637>
[11:27] <mup> PR snapcraft#3112 opened: spread: remove dead code for lxd setup and add debug prints <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3112>
[11:27] <mup> PR snapd#8638 opened: tests: do not add pkg dbus-user-session on ubuntu-14.04 <Created by mvo5> <https://github.com/snapcore/snapd/pull/8638>
[11:29] <mup> PR snapd#8334 closed: many: add core.resiliance.vitality-hint config setting <Needs Samuele review> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8334>
[11:36] <zyga> mvo: I added a workaround as samuele suggested, I'll rebase on top of the master fixes and pushed
[11:36] <zyga> mvo: we can return there if we are really bored one day ;-)
[11:37] <mvo> zyga: heh, maybe what is running already contains this workaround, no failure so far on gce
[11:37] <mvo> zyga: anyway, I have lunch now
[11:37] <zyga> oh?!
[11:37] <zyga> weird
[11:37] <zyga> I saw this very quickly
[11:37] <zyga> (before the workaround)
[11:37] <zyga> on core16 or core18?
[11:37] <zyga> I kept running core16
[11:40] <mup> PR snapd#8639 opened: o/cmdstate: handle ignore flag on exec-command tasks (1/N) <Services ⚙️> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8639>
[11:50] <zyga> re
[11:51] <zyga> in the times of home schooling, fixing the printer is a rather important duty :/
[11:51] <zyga> mvo: we can close https://github.com/snapcore/snapd/pull/8638
[11:51] <mup> PR #8638: tests: do not add pkg dbus-user-session on ubuntu-14.04 <Created by mvo5> <https://github.com/snapcore/snapd/pull/8638>
[11:53] <mup> PR snapd#8572 closed: packaging: add the inhibit directory <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8572>
[11:54] <zyga> pedronis: is there a particular name we can settle on in https://github.com/snapcore/snapd/pull/8578
[11:54] <mup> PR #8578: interfaces: add host-docs interface <Created by zyga> <https://github.com/snapcore/snapd/pull/8578>
[11:55] <zyga> pedronis: I'd like to do a small pass, update the name, adjust the test as pawel suggested, and be done with it
[11:57] <pedronis> zyga: so  :   session-tool -u test journalctl --user  gets a permission error about test not being systemd-journal,  and I haven't found an easy way to see the same logs when driving things as root
[11:58] <zyga> curious, let me look
[11:59] <zyga> I mean, the whole point of journalctl --user is to look at your own logs
[11:59] <zyga> was this on core 16?
[11:59] <pedronis> yes
[12:00] <zyga> can you paste the entire message if you have it handy
[12:00] <zyga> I'm looking now
[12:01] <pedronis> session-tool -u test journalctl --user
[12:01] <pedronis> Hint: You are currently not seeing messages from other users and the system.
[12:01] <pedronis>       Users in the 'systemd-journal' group can see all messages. Pass -q to
[12:01] <pedronis>       turn off this notice.
[12:01] <pedronis> No journal files were opened due to insufficient permissions.
[12:01] <zyga> huh, that's very confusing
[12:02] <zyga> checking what happens under the cover
[12:03] <zyga> interesting, it could be a permission error on core
[12:04] <zyga> user logs are in /var/log/journal/*/user-UID@*
[12:04] <zyga> so presumably part of that path is not available to the test user
[12:04] <zyga> it could be that we are lacking some basic group ownership
[12:05] <zyga> oddly all logs are owned by root and systemd-journal group (on classic)
[12:05] <zyga> I'll check if we can simply create the test user with systemd-journal group to solve it
[12:06] <pedronis> it's core so groups is readonly, maybe there's a way around that
[12:06] <zyga> we work around that alreay
[12:06] <zyga> core has extra setup for it
[12:13] <zyga> pedronis: trying something like this now
[12:13] <zyga> https://www.irccloud.com/pastebin/DL4hLdWP/
[12:16] <mup> PR snapd#8640 opened: wrappers: pass 'disable' flag to StopServices wrapper <Services ⚙️> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8640>
[12:27] <mup> PR snapcraft#3108 closed: tools: install wheel in snapcraft-dev <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3108>
[12:28] <zyga> almost but not quite ;)
[12:30] <mup> PR snapcraft#3103 closed: python v2 plugin: set path to the interpreter <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3103>
[12:39] <zyga> pedronis: re, can we settle on the final name for the host-docs proposal? I'd love to have that branch behind us :)
[12:40] <pedronis> zyga: I made some proposals, nobody gave feedback on them
[12:40] <zyga> pedronis: I think the word "misc" is probably bet to be avoided
[12:40] <zyga> so perhaps just system-doc?
[12:41] <pedronis> no
[12:41] <pedronis> they are eally not the system docs
[12:41] <zyga> though I personally perfer the plural because there's more than one document there
[12:41] <zyga> what do you mean?
[12:41] <pedronis> they are not the main system doc
[12:42] <pedronis> the manpages and mayb share/info are
[12:42] <zyga> I see what you mean
[12:42] <pedronis> as I said,  the FHS call them miscelleanous documentation, I didn't make that up
[12:42] <zyga> arguably we could explore providing both info and man pages but I would not expect those to be much use
[12:43] <zyga> I think FHS naming is not really that great, it's only misc from a mainframe era point of view
[12:44] <zyga> on my system there are nearly 200MB in /usr/share/doc and only about 50 in both info and man pages
[12:45] <zyga> pedronis: I was wondering about "system" as well, because it's a dual-purpose word in snap world - it can mean the "system" as as a whole, including the core part of it, or the "host system" when thinking about classic systems with snapd also installed
[12:47] <pedronis> I made a 2nd proposal as well: system-packages-doc
[12:48] <zyga> I'm happy with that one, because packages are plural and the "doc" refers to the directory name, not to "documentation" or "documents"
[12:48] <zyga> if you are okay I can apply that quickly
[12:48] <mup> PR snapcraft#3105 closed: build provider: clean incompatible build-environments <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3105>
[13:09] <mup> PR snapcraft#3111 closed: elf: fix parsing of notes after patchelf mangling <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3111>
[13:11] <mborzecki> cmatsuoka: if you can access the shell after booting in recovery mode and chooser failed to trigger, can you try to obtain the log from snapd.recovery-chooser-trigger.service ?
[13:11] <cmatsuoka> mborzecki: will do
[13:11] <mborzecki> cmatsuoka: thanks!
[13:14] <cmatsuoka> mborzecki: I'll re-image ian's PR because sudo doesn't work in this image
[13:23] <mup> PR snapd#8638 closed: tests: do not add pkg dbus-user-session on ubuntu-14.04 <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8638>
[13:39] <mup> PR core-build#59 closed: initramfs: add new handle_writable_defaults() <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/core-build/pull/59>
[13:41] <pedronis> mborzecki: this how Pool is going to be used in the end: https://github.com/snapcore/snapd/pull/8569/files#diff-763375e1fe8805703f81dfb94060286f
[13:41] <mup> PR #8569: o/assertstate,asserts: use bulk refresh to refresh snap-declarations <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8569>
[13:56] <mup> PR snapd#8641 opened: tests: improve debugging of user session agent tests <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8641>
[14:08] <mborzecki> cmatsuoka: chooser seems to start correctly with core20_634.snap, pc_100.snap, pc-kernel_495.snap in recover mode here
[14:09] <cmatsuoka> mborzecki: did you remaster the kernel?
[14:09] <mborzecki> cmatsuoka: no, i did not try to repack it
[14:09] <cmatsuoka> mborzecki: in my tests this situation works too
[14:09] <cmatsuoka> mborzecki: once you repack the kernel it starts to fail
[14:10] <cmatsuoka> you don't really need to inject anything or even change the efi
[14:14] <mup> PR snapd#8642 opened: tests: add dbus-user-session to bionic and reorder package names <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8642>
[14:15] <cachio> zyga, hey, when you have 5 minutes, could you please take a look to #8642
[14:16] <mup> PR #8642: tests: add dbus-user-session to bionic and reorder package names <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8642>
[14:16] <zyga> I'm looking
[14:17] <zyga> does it affect anything? I mean session-tool already shows 18.04 has this, or is that just because the current image has recommended packages?
[14:18] <cachio> zyga, I was testing and the images dont have it
[14:19] <zyga> do you mean the upcoming images?
[14:19] <cachio> zyga, yes, upcomming https://paste.ubuntu.com/p/PRTDY963dY/
[14:19] <zyga> ok
[14:19] <zyga> yeah, looks good
[14:19] <zyga> pedronis: this should help https://github.com/snapcore/snapd/pull/8641
[14:19] <mup> PR #8641: tests: improve debugging of user session agent tests <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8641>
[14:20] <cachio> zyga, once we have that, I can push the image which will fail with the mount-ns test
[14:21] <zyga> cachio: sounds good
[14:21] <cachio> zyga, nice, thanks
[14:23]  * zyga goes for lunch
[14:31] <cmatsuoka> mborzecki: https://pastebin.ubuntu.com/p/67KsBSvCP4/
[14:32] <mborzecki> cmatsuoka: does it reboot?
[14:33] <mborzecki> hmm
[14:33] <cmatsuoka> sorry, I didn't wait again
[14:33] <cmatsuoka> I just rebooted it by hand to get the other log
[14:34] <cmatsuoka> mborzecki: the other log: https://pastebin.ubuntu.com/p/Gkz77ksFK3/
[14:36] <mborzecki> cmatsuoka: from what i've seen you need to hold it super long to trigger the chooser in recovery
[14:37] <cmatsuoka> I press it then the tianocore logo is still on screen, and release it only when the key fingerprints appear
[14:38] <cmatsuoka> and I got a lot of 1111111111 during the boot messages
[14:41] <mborzecki> hmm ok i think i know what's going on
[14:42] <cmatsuoka> ah excellent
[14:42] <mvo> pstolowski: fwiw I uploaded https://github.com/snapcore/core-build/pull/59 to bionic,xenial so the next kernel rebuild should have support for _writable_defaults
[14:42] <mup> PR core-build#59: initramfs: add new handle_writable_defaults() <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/core-build/pull/59>
[14:42] <pstolowski> mvo: great! thanks!
[14:43] <mborzecki> cmatsuoka: mvo: looking at the logs here, when system is booting in recovery mode, the trigger watch is started in initramfs (your only chance to trigger the chooser is now!), then once we switch root snapd won't start the trigger monitor since it has `# X-Snapd-Snap: do-not-start`, so there's no chance to trigger it during first boot
[14:44] <cmatsuoka> mborzecki: but why does it work when you don't repack the kernel snap?
[14:53] <pedronis> mborzecki: it's a bit unclear why snapd shouldn't start if it's not started already
[14:53] <pedronis> *start it
[14:54] <mborzecki> cmatsuoka: idk, if the log can be trusted, the key triggered in < 10s? weird
[15:10] <cachio> zyga, does it makes sense to have a script which created the github actions workflows based on a cofiguration file?
[15:10] <zyga> cachio: if we end up editing them a lot, maybe, otherwise I would not bother
[15:11] <zyga> cachio: depending on how actions evolve I would expect to see unification, not further shattering of the workflow files
[15:11] <cachio> so it is easy to have a workflow by system
[15:11] <cachio> and it is easy to restart just 1 system
[15:11] <zyga> cachio: it's not quite easy becaues the script doesn't matter to github
[15:11] <zyga> once still has to edit and run the script and commit the result
[15:12] <zyga> *because
[15:12] <zyga> I mean, if we find editing those files a burden, sure
[15:12] <cachio> the idea is to have an script which will create the workflows, the we push that
[15:12] <cachio> so then we just need to edit the template
[15:12] <mup> PR snapcraft#3113 opened: colcon: use packages.ros.org/ros2 repository <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3113>
[15:13] <zyga> cachio: I do understand but did you read what I said above?
[15:13] <zyga> since it's an additional step it doesn't buy us anything over just editing the script once in a while (you need to commit both the script changes and the actual workflow files)
[15:13] <cachio> yes, we don't need to edit/update those
[15:13] <zyga> and I'm not opposed to it in principle but I don't see the need for it either
[15:15] <cachio> zyga, the idea is to avoid this
[15:15] <cachio>       matrix:
[15:15] <cachio>         system:
[15:15] <cachio>         - ubuntu-14.04-64
[15:15] <cachio>         - ubuntu-18.04-64
[15:15] <cachio>         - ubuntu-core-18-64
[15:15] <cachio>         - debian-9-64
[15:15] <cachio>         - arch-linux-64
[15:15] <cachio>         - amazon-linux-2-64
[15:15] <cachio>         - centos-7-64
[15:15] <cachio> 1 workflow 1 system
[15:16] <cachio> does it makes sense?
[15:16] <cachio> until we are able to restart sytems independently
[15:16] <zyga> so that's not quite what you described earlier
[15:16] <zyga> that's a different cahnge
[15:17] <zyga> the problem here is that now you run unit tests way many more times
[15:17] <zyga> this will perform differently
[15:17] <cachio> zyga, we could run unit tests just on some systems
[15:18] <zyga> then we would run spread if unit tests failed
[15:18] <zyga> because you cannot have cross-workflow relationships
[15:19] <cachio> we could use tags for that
[15:19] <zyga> what kind of tags?
[15:20] <cachio> in a gce bucket
[15:20] <zyga> can we do that without doing the script?
[15:20] <zyga> IIRC mvo has implemented this already, it's under review
[15:20] <cachio> zyga, I'l take a look
[15:20] <cachio> that that change, I didn't see that
[15:20] <mup> PR snapd#8643 opened: wrappers: add RestartServices function and ReloadOrRestart to systemd (3/N) <Services ⚙️> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8643>
[15:26] <mborzecki> cmatsuoka: repacked ekrnel, added some timestamps to the trigger monitor and can't trigger chooser anymore
[15:29] <zyga> pedronis, jdstrand: updated https://github.com/snapcore/snapd/pull/8578
[15:30] <mup> PR #8578: interfaces: add host-docs interface <Created by zyga> <https://github.com/snapcore/snapd/pull/8578>
[15:30] <zyga> mainly renamed to system-packages-doc
[15:32] <zyga> pedronis: ha, we have logs from the faliure now
[15:32] <zyga> *failure
[15:32] <pedronis> there's no there there
[15:32] <pedronis> fascinating
[15:32] <zyga> snap is gone
[15:32] <zyga> what :D
[15:32] <zyga> right?
[15:32] <pedronis> we are probably leaving a mess in the maint script I suppose
[15:32] <pedronis> and things get confused
[15:33] <pedronis> wondering why it doesn't affect other systems though
[15:33] <zyga> I'll get to the bottom of this
[15:33] <zyga> but I need to do an errand now
[15:33] <zyga> but it's paying off
[15:33] <pedronis> ah, no
[15:33] <pedronis> this is core16
[15:33] <pedronis> but is saying /snapd
[15:33] <pedronis> ??
[15:33] <zyga> ha
[15:33] <zyga> I have a hunch :D
[15:33] <zyga> let's see if the log has this
[15:34] <pedronis> we do have a core16->core18 test
[15:34] <pedronis> maybe
[15:34] <pedronis> and leaving stuff around?
[15:34] <zyga> I bet it's something like that
[15:34] <pedronis> anyway I suspect we don't quite the right thing with the user services
[15:34] <pedronis> and cleanup / restore
[15:34] <zyga> google:ubuntu-core-16-64:tests/core/reboot google:ubuntu-core-16-64:tests/core/remodel google:ubuntu-core-16-64:tests/core/os-release google:ubuntu-core-16-64:tests/core/apt google:ubuntu-core-16-64:tests/core/network-config:system google:ubuntu-core-16-64:tests/core/snap-auto-import-asserts google:ubuntu-core-16-64:tests/core/swapfiles google:ubuntu-core-16-64:tests/core/writablepaths
[15:34] <zyga> google:ubuntu-core-16-64:tests/core/device-reg google:ubuntu-core-16-64:tests/core/snap-core-fixup google:ubuntu-core-16-64:tests/core/grub-no-unpacked-assets:vmlinuz google:ubuntu-core-16-64:tests/core/snap-set-core-config google:ubuntu-core-16-64:tests/core/grub-no-unpacked-assets:initrdimg google:ubuntu-core-16-64:tests/core/create-user-2 google:ubuntu-core-16-64:tests/core/services
[15:34] <zyga> google:ubuntu-core-16-64:tests/core/gadget-config-defaults:ssh_oneline google:ubuntu-core-16-64:tests/core/classic-snap16 google:ubuntu-core-16-64:tests/core/gadget-config-defaults:rsyslog google:ubuntu-core-16-64:tests/core/snapd-failover google:ubuntu-core-16-64:tests/core/gadget-update-pc google:ubuntu-core-16-64:tests/regression/lp-1599891 google:ubuntu-core-16-64:tests/main/snap-services
[15:34] <zyga> google:ubuntu-core-16-64:tests/main/system-usernames-illegal google:ubuntu-core-16-64:tests/main/interfaces-personal-files google:ubuntu-core-16-64:tests/main/services-snapctl google:ubuntu-core-16-64:tests/main/security-dev-input-event-denied google:ubuntu-core-16-64:tests/main/install-remove-multi google:ubuntu-core-16-64:tests/main/auto-aliases google:ubuntu-core-16-64:tests/main/interfaces-content-mkdir-writable:common
[15:34] <zyga> google:ubuntu-core-16-64:tests/main/snap-user-service-start-on-install + echo '# free space'
[15:35] <zyga> there's the snapd failover test
[15:36] <zyga> prime candidate :)
[15:37] <zyga> pedronis: looking at the failover  test I don't see what removes snapd actually, perhaps it assumes some global cleanup
[15:37] <zyga> I'll look
[15:37] <zyga> but _after_ an errand, I'll be late soon
[15:37] <mborzecki> cmatsuoka: this is so weird, https://paste.ubuntu.com/p/kB7QyDMmrx/ can't trigger this when i remove the firmware from the kernel snap, same when i pass dangerous systemd.debug-shell=1
[15:38] <pedronis> well, we assume globabl cleanups but probably they don't do the right thin about this services
[15:38] <pedronis> *these
[15:38] <zyga> I'm not entirely sure this is really that, it's just a command that fails to find an executable,
[15:38] <zyga> but we'll find out once we look closer
[15:39] <pedronis> well it's wrong service file content
[15:39] <pedronis> for UC16
[15:41] <cachio> pedronis, do you know is there is a publis user assertion for uc20 beta images?
[15:41] <zyga> pedronis: but the service file is fixed
[15:41] <zyga> pedronis: it's just /usr/bin/snap userd --agent
[15:45] <pedronis> zyga: that's not what the log says
[15:45] <pedronis> or the .in
[15:45] <zyga> can you explain?
[15:45] <zyga> what does the .in file say?
[15:46] <pedronis> ExecStart=/snap/snapd/x1/usr/bin/snap userd --agent
[15:46] <pedronis> @bindir@/snap userd --agent
[15:46] <zyga> that's from the log, I wonder if that's actually true (the first line)
[15:46] <zyga> I don't recall us rewriting those units
[15:46] <zyga> but maybe I'm mistaken
[15:47] <pedronis> we rewrite them for 18
[15:47] <pedronis> that's why as I said I suspect an interaction between testing snapd on UC16 vs cleanup
[15:47] <pedronis> logic
[15:47] <pedronis> but I might be off
[15:48] <zyga> mmm, I see, I grepped for the snapd.session-agent but I didn't find anything in snapd proper, looking at the test code
[15:48] <zyga> anyway, it's much more closer to the light now :)
[15:49] <pedronis> it's the code in wrappers/core18.go
[15:49] <pedronis> it's very indirect so it's not easy to grep
[15:49] <pedronis> I think
[15:49] <zyga> ah, I see it now
[15:49] <zyga> writeSnapdUserServicesOnCore
[15:49] <pedronis> yes
[15:49] <zyga> yeah, it's cleary this
[15:50] <zyga> this makes me wonder why it got stale, perhap the code is just naive and is happy with any file it finds
[15:50] <pedronis> I'm not sure we undo
[15:50] <pedronis> except via the general cleanup
[15:50] <pedronis> restore
[15:50] <pedronis> but as I said I'm not sure that code
[15:50] <pedronis> does things with this files
[15:51] <mup> PR snapd#8641 closed: tests: improve debugging of user session agent tests <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8641>
[15:52] <pedronis> the snap-mgmt tests or something need to grow I suppose
[15:52] <zyga> oh I see
[15:52] <zyga> when we install snapd this happens
[15:52] <zyga> we don't have an equivalent call path for "but I want back to core"
[15:53] <zyga> the core in wrappers is actually pretty smart already
[15:53] <mup> PR snapd#8559 closed: boot, bootloader: adjust comments, expand tests <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8559>
[15:53] <zyga> we could fix this with an explicit test changes
[15:53] <zyga> but I suspect it's also a real bug
[15:54] <zyga> that would show up in practice
[15:54] <pedronis> yes, but I'm not sure apt purge snapd
[15:54] <pedronis> does the right thing
[15:54] <pedronis> was user services
[15:54] <pedronis> atm
[15:54] <pedronis> there's probably a real bug there
[15:55] <pedronis> *fo ruser services
[15:55] <zyga> yes, we don't consider user services
[15:55] <zyga> oh well, so many things
[15:55] <zyga> I'll run now because I'm already late
[16:04] <cachio> mvo, you said I should test pending images on cdimage for beta channel
[16:04] <cachio> mvo, I tried amd64 and didn't work
[16:04] <cachio> mvo, I should use pending images just for rpi?
[16:05] <mvo> cachio: thanks! in a meeting right now, please hop into #uc20
[16:23]  * cachio lunch
[16:23] <zyga> condensed debug data in case someone wants to figure out how we ought to fix the bug pedronis identified:
[16:23] <zyga> user session services debug data https://www.irccloud.com/pastebin/Cbp1aJFm/
[16:23] <zyga> the failover test does indeed corrupt the system
[16:23] <zyga> it's also possible that many other tests that have similar interactions behave the same way
[16:25] <zyga> rm /etc/systemd/user/snapd.session-agent.service && systemctl --user cat snapd.session-agent.service is sufficient to work around it
[16:25] <zyga> but it highlights a deeper bug
[16:25]  * zyga -> AFK
[16:43] <mup> PR snapcraft#3101 closed: package repositories: avoid setting up on host when not the target <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3101>
[17:02] <cachio> zyga, hey
[17:02] <mup> PR snapd#8642 closed: tests: add dbus-user-session to bionic and reorder package names <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8642>
[17:04] <cachio> zyga, once you have the fix for the mount ns test I'll promote the failing image
[17:04] <cachio> zyga, I mean I'll promote the bionic image which fails running mount-ns test
[17:11] <mup> PR snapd#8613 closed: o/configcore: pass extra options to FileSystemOnlyApply <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8613>
[17:34] <mup> PR snapcraft#3113 closed: colcon: use packages.ros.org/ros2 repository <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3113>
[17:37] <mpontillo> hi friends; I noticed today that some of my snaps are out out of date. I thought I was maybe hitting bug #1754345, but logging out and logging in again didn't fix things. any ideas? https://paste.ubuntu.com/p/prCzJTGjnZ/
[17:37] <mup> Bug #1754345: Returns "invalid credentials" error while trying to refresh an invalid macaroon <snapd:Triaged by chipaca> <https://launchpad.net/bugs/1754345>
[19:28] <mup> PR snapd#8644 opened: cmd/snap-bootstrap/initramfs-mounts: append uuid to ubuntu-data when decrypting <UC20> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8644>
[19:55] <mup> PR snapd#8645 opened: secboot: append uuid to ubuntu-data when decrypting <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8645>
[19:58] <mup> PR snapcraft#3114 opened: npm v2 plugin: update doc string <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3114>
[20:58] <mup> PR snapcraft#3112 closed: spread: remove dead code for lxd setup and add debug prints <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3112>
[22:52] <mup> PR snapcraft#3115 opened: build providers: ignore missing LXD instance when cleaning project <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3115>