[05:29] <mborzecki> morning
[06:45] <mup> PR snapd#9054 closed: bootloader/assets: generate bootloader assets from files <UC20> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9054>
[08:43] <jamesh> it's nice to see green ticks from CI again
[08:48] <pedronis> yes
[08:48] <pedronis> mborzecki: catching up on some things, and then I'll review 9006 first
[08:48] <mborzecki> pedronis: thanks!
[08:49] <mborzecki> Saviq: hey, multipass does not plug removable media?
[08:55] <zyga> good morning
[08:56] <mup> PR snapd#9060 opened: interfaces/bultin/multipass: replace U+00A0 no-break space with simple space <Simple 😃> <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9060>
[08:58] <pedronis> zyga: hi
[08:58] <zyga> hey pedronis
[08:58] <zyga> sorry for starting late, it's extremely difficult to sleep lately
[08:58] <zyga> only one day left though
[09:00] <pedronis> zyga: how do you want to proceed, start cutting the release this morning, or in the afternoon? I can provide a cleaned up changelog
[09:01] <zyga> pedronis: I would like to look at the release branch now, run tests manually, review the instructions from Mvo and get everything ready for early afternoon
[09:01] <pedronis> zyga: ok
[09:01] <pedronis> zyga: I can do the part of the instruction that is producing the changelong content and cleaning it up
[09:01] <pedronis> zyga: ping me when you need that
[09:01] <pedronis> zyga: thank you
[09:01] <zyga> thank you, that will be most welcome
[09:05] <zyga> 100 patches between 2.25.2 and release head
[09:37] <mborzecki> zyga: anything unexpected there?
[09:39] <zyga> mborzecki: nothing yet
[09:39] <zyga> I forgot we did some interface changes backports
[09:41] <zyga> most is just noise and test tweaks
[09:44] <zyga> brb
[09:45] <pedronis> zyga: the condesed and edited changelog from mvo tool is: https://gist.github.com/pedronis/c1577c450d53ff3c9fe3086447726232
[09:46] <zyga> ack, looking
[09:47] <zyga> nicely readable
[09:50] <zyga> I've gone through all the patches and everything looks good
[09:50] <zyga> I'll run tests two more times while reading mvo's instructions
[09:58] <pedronis> mborzecki: did a pass on #9006, let me know if you have questions
[09:58] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[10:02] <zyga> one thing I found in the changelogs that we don't have in mvo's doc is the SRU tracking bug
[10:02] <zyga> for 2.45 we used https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1875071
[10:02] <mup> Bug #1875071: [SRU] 2.45 <id-5ef3c1ba59467589703d5ae0> <verification-done> <verification-done-bionic> <verification-done-eoan> <verification-done-focal> <verification-done-xenial> <snapd (Ubuntu):Fix Released> <snapd (Ubuntu Xenial):Fix Released> <snapd (Ubuntu Bionic):Fix Released> <snapd (Ubuntu
[10:02] <mup> Eoan):Fix Released> <snapd (Ubuntu Focal):Fix Released> <https://launchpad.net/bugs/1875071>
[10:02] <zyga> but it seems to mention 2.45.1 as well
[10:02] <zyga> I wonder if I should file a placeholder bug for 2.45.3
[10:03] <zyga> first run: two tests failed in fedora
[10:03] <zyga> well, centos and fedora
[10:03] <zyga>     - google:centos-8-64:tests/main/selinux-clean
[10:03] <zyga>     - google:fedora-30-64:tests/main/snap-run
[10:03] <zyga> but worse, preparing arch failed entirely
[10:03]  * zyga looks 
[10:08] <mborzecki> zyga: ah, w8, there was a patch for zstd
[10:08] <zyga> mborzecki: I think I saw one in the patches
[10:09] <zyga> it's not that
[10:09] <mborzecki> zyga: 4de033c77e742c166dd075fd5e9db507a99b17dc this one?
[10:09] <zyga> https://paste.ubuntu.com/p/wdHDqkXFKy/
[10:09] <mborzecki> zyga: hm that one i cannot fix
[10:10] <zyga> yes, that patch is in the release
[10:10] <zyga> heh, only on release day :)
[10:10] <zyga> that looks like google's own arch repo?
[10:10] <mborzecki> zyga: yeah, not sure where it's coming from
[10:11] <mborzecki> tbh i didn't know there's one even
[10:16] <zyga> different failure now
[10:16] <zyga> Found unused ./vendor packages:
[10:16] <zyga>  vu github.com/snapcore/secboot/internal/pe1.14
[10:16] <zyga> Please fix via 'govendor remove +unused'
[10:16]  * zyga looks
[10:17] <mborzecki> zyga: pacman -Sy seems to work, so maybe there really is a repo, but it's only accessible form the nodes?
[10:21] <mup> PR snapd#9060 closed: interfaces/builtin/multipass: replace U+00A0 no-break space with simple space <Simple 😃> <Skip spread> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9060>
[10:27] <mborzecki> zyga: does it work now?
[10:33] <zyga> now things pass
[10:33] <zyga> mborzecki: it's weird, I removed vendor bits and ran tests without them
[10:33] <zyga> why didn't any other system complain?
[10:34] <mborzecki> zyga: i have no idea, arch should be using go 1.14.6 atm, maybe something about it
[10:37] <zyga> mborzecki: about the other failures, the snap-run test hangs on strace sometimes so that's not new
[10:37] <zyga> mborzecki: and the centos 8 selinux policy situation has not changed, correct?
[10:37] <mborzecki> zyga: nope, i should fix that test sometime
[11:16] <mup> PR snapd#9061 opened: packaging: add placeholder changelog for 2.45.3 <Created by zyga> <https://github.com/snapcore/snapd/pull/9061>
[11:16] <zyga> changelogs are ready, I created a PR for the placeholder entry, I've cloned the trello card and I'm looking at the card now
[11:17] <zyga> I don't think I've ever touched snapd translations on launchpad
[11:17] <zyga> not sure if anyone knows how that operates
[11:27] <mborzecki> zyga: mvo maybe (hopefully)?
[11:27] <zyga> yeah, I wish I knew
[11:27] <zyga> I think we should rotate this more often
[11:27] <zyga> feels a bit "muscle memory"
[11:31] <mup> PR snapcraft#3227 opened: meta: fix content slot to set content when using source key <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3227>
[11:47] <zyga> brb
[11:52] <zyga> re
[12:03] <mborzecki> pedronis: about https://github.com/snapcore/snapd/pull/9006#discussion_r460764489 you mean the same such that the order stays unchanged, i.e. mode/system as specified last?
[12:03] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[12:06] <pedronis> mborzecki: the order yes, or anything else, formulated otherwise my question is whether the final command lines are changing or are the same
[12:06] <pedronis> vs what we have on master right now
[12:09] <mborzecki> pedronis: hm it does change, that's why i bumped the edition number to 2 initially
[12:09] <pedronis> mborzecki: can you make a pastebin of old and new somewhere?
[12:09] <pedronis> side by side
[12:10] <mborzecki> pedronis: grub.cfg and recovery?
[12:10] <pedronis> the command lines
[12:10] <pedronis> for both
[12:11] <mborzecki> pedronis: ok, let me look at the scripts again, this has changed a bunch of times during the reviews
[12:18] <mborzecki> pedronis: heh, i was wrong, it changed twice in the course of reviews, at the end of the day it's the same as in master <mode> <static> <extra> (where <extra args> are a new thing and not used atm), see https://github.com/snapcore/snapd/pull/9006/files#diff-6f9b97376e62185c79af158d2d3c9474 and https://github.com/snapcore/snapd/pull/9006/files#diff-d3f0c595846626e6f78fe9b29f0a61e6
[12:18] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[12:19] <mborzecki> pedronis: that diff chunk in boot is caused by a fix in bootloadertest mocks
[12:20] <pedronis> mborzecki: good, my impression was also that in the end the result was the same
[12:21] <mborzecki> pedronis: while at it, i'll tweak the interface to have CommandLine(mode, system, extraArgs string)
[12:22] <pedronis> sounds good
[12:23] <pedronis> mborzecki: to be clear mode and system there  are full "name=value", right?
[12:26] <mup> PR snapcraft#3228 opened: review tools: fix snap link or copy path <bug> <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3228>
[12:39] <mborzecki> pedronis: yes, just like in the PR now, the only difference is the call will be eg. CommandLine("snapd_recovery_mode=recover", "snapd_recovery_system=123123", "arg=1 other=2")
[12:40] <pedronis> thx
[12:40] <mborzecki> pedronis: and grub bootloader implementation verifies snapd_recovery_mode= and snapd_recovery_system= prefixes
[12:41] <cmatsuoka> mborzecki: the extra args in cmdline should come only from gadget or also from system settings?
[12:41]  * zyga just got the call
[12:41] <zyga> 12:30 tomorrow
[12:41] <zyga> finally
[12:42] <mborzecki> cmatsuoka: gadget for now, i recall we discussed some system args, but so far only gadget was something we though would happen
[12:43] <cmatsuoka> ok. I asked because system settings is mentioned once in the doc intro but there are no details about that
[12:43] <zyga> brb, see you at the standup
[12:43] <mborzecki> zyga: nice, feeling anxious?
[12:50] <zyga> no, just wish it was tomorrow already
[13:12] <mup> PR snapd#9062 opened: releasing package snapd version 2.45.3 <Created by zyga> <https://github.com/snapcore/snapd/pull/9062>
[13:37] <mup> PR snapd#9061 closed: packaging: add placeholder changelog for 2.45.3 <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9061>
[13:41] <pedronis> zyga: sorry I was confused, obviously edge builds from master normally, but there is this: https://launchpad.net/~snappy-dev/+snap/snapd-2.45
[13:41] <zyga> ah, nice, it goes to beta directly
[13:42] <zyga> and it builds automatically
[13:43] <pedronis> it goes to a branch of beta to be clear
[13:44] <zyga> hmm, right, I see that now
[13:45] <pedronis> zyga: and there's https://launchpad.net/~snappy-dev/+snap/core-beta
[13:45] <pedronis> for core
[13:45] <pedronis> that one is manual though
[13:45] <zyga> and this one is built on request, so we can trigger when the ppa is ready
[13:45] <pedronis> yes
[13:45] <pedronis> you can trigger the other one as well if needed
[13:53]  * zyga breaks for 30 minutes for meds and lunch
[14:01] <pedronis> zyga: lp:snapd will need to be imported though for snapd build to work with the right version
[14:01] <ijohnson> pedronis: uc20 status meeting ?
[14:02] <pedronis> omw
[14:32] <mborzecki> pedronis: cmatsuoka: i've updated #9006
[14:32] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[14:55] <pedronis> mborzecki: looked, did you see this comment: https://github.com/snapcore/snapd/pull/9006#discussion_r460770887 ? is that something we should discuss?
[14:55] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[15:02] <mup> PR snapd#9062 closed: releasing package snapd version 2.45.3 <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9062>
[15:09] <pedronis> zyga: I triggered a repo import for that ^
[15:10] <zyga> thank you
[15:10] <zyga> I tried looking at the patch for git-buildpackage, it's a bit unfortunate that we have to carry it
[15:11] <pedronis> zyga: https://code.launchpad.net/~niemeyer/snapd/+git/snapd
[15:11] <zyga> 2.45 is still at the code from a few days ago
[15:12] <pedronis> zyga: yes, look, it's says an import will run soon
[15:12] <pedronis> it doesn't start immeditately
[15:12] <zyga> yeah, refreshing the page shows stuff is changing
[15:13] <zyga> "Repository scan failed"
[15:13] <zyga> clicking the rescan button
[15:13] <zyga> not allowed
[15:13] <zyga> fun
[15:13] <zyga> I guess
[15:14] <zyga> what does that even mean?
[15:14] <zyga> git import into launchpad database?
[15:14]  * zyga asks on #launchpad
[15:17] <pedronis> zyga: the import says it worked, but the branches aren't updated yet
[15:18] <pedronis> ijohnson: are you blocked on reviews by me?
[15:18] <zyga> the pain is under control now, I'll get some tea and be back soon
[15:18] <ijohnson> pedronis: no not at the moment
[15:19] <ijohnson> pedronis: sorting out some things locally then I will push to pawel's debug seeding branch for the devicestate changes then open another PR with the command itself
[15:19] <pedronis> ijohnson: ok, thx
[15:23] <zyga> it worked now
[15:23] <pedronis> zyga: snapd:release/2.45 is up to date on LP now
[15:24] <zyga> yep
[15:24] <pedronis> thee snapd-2.45 should start on its own soon
[15:24] <pedronis> *the snapd-2.45 build
[15:25] <zyga> I
[15:25] <zyga> I'll dput to the ppa now
[15:32]  * cachio lunch
[15:56] <zyga> lintian says we're a bit rusty on various details
[15:56] <zyga> but that's fine
[15:57] <mborzecki> zyga: which package isn't? :)
[15:57] <zyga> mborzecki: hey, my toy package is clean like a ... clean package
[15:58] <zyga> dput complete
[15:58] <zyga> sorry I took forever, I kept struggling with gdp
[15:58] <zyga> *gbp
[16:04] <pedronis> zyga: so my theory was wrong, we have snapd snaps but they have ugly versions
[16:07] <pedronis> zyga: did you create the wrong kind of tag? or wasn't it pushed?
[16:07] <pedronis> I don't see the tag if I grab release 2.45
[16:07] <zyga> hmmmmm
[16:07] <zyga> I opened a PR but that didn't carry the tag!
[16:07] <zyga> I guess I can just push it now
[16:07] <mborzecki> zyga: you need to push the tag
[16:07] <zyga> done
[16:08] <mborzecki> and it's there
[16:08] <zyga> I'm sorry, I entirely missed that the tag is a separate entity
[16:08] <zyga> we need to rebuild the snap now, right?
[16:08] <mborzecki> hm it ended up on a weird commit, but i guess this is not a problem
[16:09] <pedronis> zyga: not sure, I'm still getting weird versions
[16:13] <pedronis> zyga: no, I fear how the tag was attached matters
[16:13] <pedronis> I get the right version if I go to the tag commit but I can the wrong one if I go to the tip of release/2.45
[16:15] <zyga> hmmmm
[16:15] <zyga> I suspect I know why
[16:15] <zyga> mvo pushes that to release directly
[16:15] <zyga> I could not
[16:16] <zyga> so I created a merge PR
[16:16] <zyga> so we have a new commit
[16:16] <pedronis> let me see if I can do something
[16:16] <zyga> ok
[16:18] <zyga> do we need another lp import?
[16:19] <pedronis> I haven't solved the problem, I'm not entirely sure how mvo pushes to those branches
[16:20] <zyga> maybe he has extra permissions
[16:21] <zyga> next time we should record mvo doing all those things and discuss the details
[16:24] <zyga> ppa is still building
[16:25] <zyga> ah no, status was stale
[16:25] <zyga> it failed
[16:25] <zyga> on aarch64
[16:25] <zyga> on unit tests
[16:25] <ijohnson> pedronis: https://github.com/snapcore/snapd/pull/9035 is now ready for you to review
[16:25] <mup> PR #9035: o/devicestate: save seeding/preseeding times for use with debug seeding api (3/N) <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9035>
[16:26] <zyga> TestClientTimeoutLP1837804
[16:26] <zyga> ... error string = "cannot communicate with server: Post http://127.0.0.1:40893/: dial tcp 127.0.0.1:40893: i/o timeout"
[16:26] <zyga> ... regex string = ".* timeout exceeded while waiting for response"
[16:31] <pedronis> zyga: now it should work, forced pushed to 2.45 the commit with the tag, instead of the merge
[16:31] <zyga> great
[16:31] <pedronis> zyga: please double check that 2.45 is sane
[16:31] <zyga> so now we need another lp import
[16:31] <zyga> sure, doing so now
[16:31] <pedronis> yes
[16:32] <zyga> scheduled one just now
[16:32] <pedronis> thx
[16:33] <zyga> did you force push the tag?
[16:33] <zyga> I'm looking at https://github.com/snapcore/snapd/commit/b055d1902de062efe536f2c3f7dd7d2a35db2c61
[16:35] <pedronis> zyga: the tag is there: https://github.com/snapcore/snapd/tree/2.45.3
[16:35] <zyga> ah, it's just UI that confused me
[16:35] <zyga> ok
[16:36] <pedronis> ijohnson: thanks, I'll try to look still today
[16:37] <ijohnson> pedronis: thanks, also https://github.com/snapcore/snapd/pull/9063 is the next one which is the snap debug command bit
[16:37] <mup> PR #9063: cmd/snap/debug: add "snap debug seeding" command for preseeding debugging (4/N) <Preseeding 🍞> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9063>
[16:37] <ijohnson> (which should be the last bit, I included the spread test bit there too, it's fairly small, but if you prefer I can make the spread test another PR)
[16:37] <zyga> oh, ijohnson, can you look at https://github.com/snapcore/snapd/pull/9057 please - I removed the environment entry, it was unjused anyway
[16:37] <mup> PR #9057: overlord: use new tracking cgroup for refresh app awareness <Created by zyga> <https://github.com/snapcore/snapd/pull/9057>
[16:37] <zyga> *unused
[16:37] <mup> PR snapd#9063 opened: cmd/snap/debug: add "snap debug seeding" command for preseeding debugging (4/N) <Preseeding 🍞> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9063>
[16:38] <ijohnson> zyga: sure, I have an in progress review I kept getting distracted from :-)
[16:38] <zyga> OK
[16:38] <pedronis> ijohnson: added to my queue
[16:38] <ijohnson> zyga: and in case I don't get a chance to say it before tomorrow, good luck tomorrow! really hoping things go well for you
[16:38] <ijohnson> thanks pedronis
[16:39] <ijohnson> I will address some feedback and do reviews now, but then I will look into the re-exec bug for preseeding then
[16:41] <pedronis> zyga: I requested a manual build of the snap
[16:42] <pedronis> zyga: snapd is building now
[16:42] <zyga> great, the lp import has succeeded as well
[16:48] <ijohnson> zyga: +1d
[16:48] <zyga> woot
[16:48] <zyga> thank you
[16:48] <ijohnson> no, thank you for sticking with this, we're really close I think :-)
[16:49]  * zyga listens to the thunder and rain outside
[16:49] <zyga> ijohnson: and before the anniversary
[16:49] <zyga> of the v2 branch at least :)
[16:49] <ijohnson> :-D
[16:49] <zyga> once this lands the rest is just code removal
[16:49] <zyga> and some misc changes unrelated to this feature
[17:06] <zyga> the versions now look good
[17:06] <zyga> on the beta/2.45 branch that is
[17:07] <zyga> the ppa is still building for some architectures
[17:07] <zyga> once that is done we should trigger https://launchpad.net/~snappy-dev/+snap/core-beta
[17:07] <zyga> I will break for an hour and come back
[17:07] <zyga> need to double check my hospital bag
[17:08] <zyga> and see what's going on at home
[17:11] <mup> PR snapcraft#3229 opened: errors: introduce HostToolNotFoundError exception <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3229>
[17:17] <cachio> ijohnson, hey
[17:17] <ijohnson> hi cachio
[17:17] <ijohnson> I was just about to break for lunch
[17:17] <ijohnson> (just fyi)
[17:18] <cachio> do you know what happens during initial boot after the line systemd[1]: Started Journal Service.  --> https://paste.ubuntu.com/p/cBK2cnG6NR/
[17:18] <cachio> ijohnson, I se most of the random reboots after that line
[17:18] <cachio> ijohnson, np
[17:18] <ijohnson> cachio: looking quickly
[17:19] <cachio> ijohnson, also after line systemd-journald[185]: Received client request to flush runtime journal.
[17:20] <ijohnson> cachio: I think that just happens to be one of the last things that systemd will do before starting up snapd, etc.
[17:20] <ijohnson> I don't think what is triggering those lines specifically would be what is causing the random reboot
[17:20] <ijohnson> but I dunno maybe it is
[17:20] <cachio> ijohnson, so then snap starts
[17:20] <ijohnson> cachio: but now immediately after that point
[17:21] <ijohnson> cachio: but not immediately after that point
[17:21] <cachio> ijohnson, ok
[17:21] <ijohnson> I have some ideas for more debugging, but I need to break for lunch now
[17:21] <cachio> ijohnson|lunch, is it any way to see a log with that info?
[17:22] <cachio> after that line in the log that I got from the nested vm I see
[17:22] <cachio> [   14.067916] systemd-journald[539]: Received client request to flush runtime journal.
[17:31] <cachio> cmatsuoka, any idea?
[17:32] <mup> PR snapcraft#3228 closed: review tools: fix snap link or copy path <bug> <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3228>
[17:32] <mup> PR snapcraft#3230 opened: lifecycle: check for snap using shutil.which() <enhancement> <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3230>
[17:32] <cmatsuoka> let me see...
[17:34] <cmatsuoka> cachio: this is the entire log, and the system crashed after that?
[17:35] <cachio> cmatsuoka, after that the system works well
[17:35] <cachio> this is the full log
[17:35] <cmatsuoka> but when it crashes, it crashes at that point after userspace is running, is that correct?
[17:36] <cachio> [   62.115886] systemd[1]: Starting Remount Root and Kernel File Systems...
[17:36] <cachio> [   62.154860] systemd[1]: Starting udev Coldplug all Devices...
[17:36] <cachio> [   62.201278] systemd[1]: Finished Create list of static device nodes for the current kernel.
[17:36] <cachio> [   62.237719] systemd[1]: modprobe@drm.service: Succeeded.
[17:36] <cachio> [   62.263067] systemd[1]: Finished Load Kernel Module drm.
[17:36] <cachio> [   62.275837] systemd[1]: Started Journal Service.
[17:37] <mup> PR snapcraft#3227 closed: meta: fix content slot to set content when using source key <bug> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3227>
[17:37] <cachio> after that line at some point it crashed
[17:37] <cmatsuoka> maybe addming something like systemd.log_level=debug systemd.log_target=console to the cmline could help?
[17:37] <cachio> cmatsuoka, but not log
[17:38] <cachio> cmatsuoka, is it possible to update the command line with tpm and secboot enabled??
[17:39] <cmatsuoka> hmm, you would need to add the parameters to the cmdlines used in sealing too
[17:39] <cmatsuoka> unless it crashes before you unlock the encrypted device
[17:40] <cmatsuoka> in this case you could boot with any cmdline
[17:40] <cmatsuoka> but you won't be able to go past beyond the writable fs mount
[17:41] <ijohnson> cmatsuoka: cachio: what I was going to propose is a test PR that changes the cmdline everywhere to include taht systemd.log_level kernel cmdline just to see the boot log
[17:41] <ijohnson> I presume it happens every time so opening a PR with "run nested" label should be enough to trigger it and get some more representative logs
[17:42] <cachio> ijohnson, ok
[17:42] <ijohnson> the other issue is that if it is crashing instead of rebooting, we may lose some of the logs as they don't get flushed to GCE in time
[17:42] <cmatsuoka> ijohnson, cachio: having an instrumented PR is probably the way to go
[17:42] <ijohnson> cachio: I will send a PR then
[17:42] <cachio> nice
[17:42] <cachio> thanks
[17:42] <ijohnson> cmatsuoka: do you know if it's sufficient to change the kernel cmdline inside snapd now with mborzecki's changes, or do I still need to make changes to the gadget grub.cfg ?
[17:43] <ijohnson> cmatsuoka: I don't know how far along mborzecki's changes are
[17:43] <cmatsuoka> ijohnson: uuuhm, I don't know if mborzecki's changes are already effective
[17:43] <cmatsuoka> if not, the hard-coded cmdline is in gadget/install/install.go
[17:44] <ijohnson> I think they might be effective already, I tried something similar late last week and found that snapd was writing the grub.cfg
[17:45] <cmatsuoka> at some point last week I had to change the hardcoded cmdline, but I don't know if I had master merged in that PR
[17:45]  * cmatsuoka checks 
[17:49] <cmatsuoka> ah, I think you're right, in my test I edited the bootloader line by hand
[17:49] <cmatsuoka> (it was a local test, not a test PR)
[17:51] <cmatsuoka> I have a local branch where I changed the hardcoded lines with calls to maciek's command line building, hence the confusion
[17:53] <mup> PR snapd#9064 opened: .github/workflows: move snap building to test.yaml as separate cached job <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9064>
[17:55] <mborzecki> zyga: how's it going?
[17:55] <zyga> mborzecki: let me chcek
[17:55] <zyga> builds were going
[17:56] <mborzecki> nice, ijohnson opened https://github.com/snapcore/snapd/pull/9064
[17:56] <mup> PR #9064: .github/workflows: move snap building to test.yaml as separate cached job <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9064>
[17:56] <ijohnson> hey mborzecki
[17:56] <ijohnson> question on your grub.cfg stuff
[17:56] <mborzecki> zyga: and upstream monitoring already opened https://bugzilla.redhat.com/show_bug.cgi?id=1861024 for the update
[17:56] <ijohnson> how much of that is operational right now ?
[17:57] <ijohnson> i.e. if I wanted to change the kernel cmdline that a uc20 system is booted with for debugging, can I just change the grub.cfg and the internal edition snippet in snapd Go code, or do I still need to make some changes to the gadget's grub.cfg ?
[17:57] <mborzecki> ijohnson: in what sense? grub.cfg (and recovery) is already written from snapd
[17:57] <zyga> https://launchpad.net/~snappy-dev/+snap/snapd-2.45 is building again
[17:57] <zyga> hmm
[17:58] <ijohnson> mborzecki: right so that should mean that if I build a fresh uc20 image with snapd from master with my modified kernel cmdline, it should be used and sealed against, correct ?
[17:58] <mborzecki> ijohnson: soo, https://github.com/snapcore/snapd/pull/8947 hasn't landed yet, so an update won't happen
[17:58] <mup> PR #8947: many: update managed boot config when refreshing snapd <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8947>
[17:58] <ijohnson> mborzecki: but for a new install
[17:58] <ijohnson> no updates
[17:59] <mborzecki> ijohnson: iirc the sealing code in gadget/install.Run is still using the harcoded cmdline
[17:59] <cmatsuoka> mborzecki: it is, I'm updating it now
[18:00] <ijohnson> mborzecki: ah good point yes I will have to change that too, but I was more wondering about the grub.cfg's that get written out to disk, if the recovery or ubuntu-boot grub.cfg's from the gadget are used anymore at all
[18:00] <zyga> mborzecki: yeah, I got that
[18:00] <zyga> fedbus is nice
[18:00] <mborzecki> ijohnson: so with #9006 landed, we'd still need a change for the install.Run()
[18:00] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <Squash-merge> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[18:00] <mborzecki> cmatsuoka: cool, no issues so far?
[18:00] <cmatsuoka> mborzecki: I just updated the code and will boot an image to see if it works as expected
[18:01] <zyga> I wonder when I should click https://launchpad.net/~snappy-dev/+snap/core-beta
[18:01] <mborzecki> ijohnson: then with claudio's change, if you use custom snapd with `dangerous system.debug-shell=1` added to snapd_static_cmdline and the snippet it should work
[18:02] <mborzecki> ijohnson: and you need to edit both the snippet in bootloader/assets/grub.go and bootloader/assets/data/grub{.cfg,-recovery.cfg} because we've decided against extracting the static bits during build time
[18:03] <ijohnson> mborzecki: so do I need 9006 for a fresh install / first boot ?
[18:03]  * ijohnson feels like the answer is no
[18:03] <mborzecki> hmm i think no
[18:04] <mborzecki> ijohnson: though i would not mind if you tried with 9006 :)
[18:04] <ijohnson> mborzecki: haha ok I will try with 9006
[18:04] <mborzecki> ijohnson: this and the branch cmatsuoka is working on
[18:05] <ijohnson> mborzecki: well for the branch cmatsuoka I just made the kernel command line I need show up everywhere manually
[18:05] <cmatsuoka> mborzecki: I got a "not implemented" error in master, is that expected?
[18:05] <ijohnson> so the install code doesn't pull it dynamically yet it just happens to still be the same :-)
[18:05] <mborzecki> cmatsuoka: ah yes, so you need 9006 too
[18:05] <cmatsuoka> mborzecki: ok
[18:10] <pedronis> mborzecki: I commented a bit more: https://github.com/snapcore/snapd/pull/9006#discussion_r461075416 , we can chat in the morning if needed
[18:10] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <Squash-merge> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[18:10] <mborzecki> pedronis: ok, thanks
[18:14] <mborzecki> pedronis: haha, ok i see what you mean now and understand why we seem to have different ideas as to what the thing does, let's chat in the morning
[18:15] <pedronis> mborzecki: remember that we need to be able to know what to expect at the next boot before writing things to disk
[18:17] <mborzecki> pedronis: yes, i wanted to have a separate call for that, something like CandidateCommandLine() with the same fingerprint, that would use the edition number from the internal asset rather than the disk
[18:17] <mborzecki> anyways, let's chat in the morning
[18:19] <mup> PR core20#78 opened: hooks/900-cleanup-etc-var.chroot: rm cloud-init config file which we don't want <Created by anonymouse64> <https://github.com/snapcore/core20/pull/78>
[18:19] <cmatsuoka> mborzecki, ijohnson: I'll have to debug a little bit here, unlocking failed
[18:20] <ijohnson> cmatsuoka: my nested run is still going, not sure how far along it has gotten tbh
[18:20] <ijohnson> it's been here for 5 minutes now
[18:20] <ijohnson> 2020-07-27 13:15:48 Preparing google-nested:ubuntu-20.04-64:tests/nested/core20/ (jul271804-169453)...
[18:26] <mup> PR core18#166 opened: hooks/900-cleanup-etc-var.chroot: rm cloud-init config file which we don't want <Created by anonymouse64> <https://github.com/snapcore/core18/pull/166>
[18:29]  * zyga is unsure what to do with the release now, sorry, my mind is all messed up 
[18:31] <pedronis> zyga: we have snapd in beta/2.45
[18:31] <pedronis> zyga: did the package build in the ppa?
[18:32] <zyga> yes, I think so
[18:32] <zyga> yes, for all arches now
[18:33] <zyga> I can see the beta/2.45 branch builds as well
[18:33] <zyga> I should push those over to beta
[18:33] <mborzecki> zyga: btw. do you have permissions to upload the source tarballs to releases page on github?
[18:33] <pedronis> zyga: the changelog looks like from master though
[18:34] <pedronis> in the ppa, bit confused
[18:34] <zyga> mborzecki: let me check, I don't know for sure
[18:34] <zyga> I think everyone does
[18:34] <zyga> I do
[18:34] <pedronis> zyga: if you look here: https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+packages the changelong is weird
[18:34] <pedronis> it has the placeholder
[18:35] <zyga> it is the placeholder, because that's the package that was built from master, not from the release
[18:35] <cmatsuoka> mborzecki: I can retrieve the recover mode cmdline correctly, but the run mode cmdline is returning empty for some reason
[18:36] <zyga> that's step 8 from https://docs.google.com/document/d/1BCOwqjdUZIg-3x9E8iyojUW_P-ZJJjG0fFLeaDZbp5A/edit#
[18:36] <pedronis> zyga: which one is the one for the release?
[18:37] <zyga> pedronis: only snapd.snap is correct at this point
[18:37] <pedronis> zyga: do we understand how to build a correct core?
[18:38] <ijohnson> mborzecki: cmatsuoka: my nested run finished, it seems that we still need to modify the gadget grub.cfg, my changes only took effect for run mode
[18:38] <ijohnson> which is fine, I can do that
[18:38] <zyga> I'm confused about that - way back when I last did this it was built from the ppa directly but that has changed in ways I didn't follow since
[18:38] <zyga> so back then a dput to that ppa was to provide the actual new snapd
[18:38] <mborzecki> cmatsuoka: hm, can you check that ubuntu-boot contains the right grub.cfg that matches what's in 9006?
[18:38] <zyga> I think the instructions are not complete and we're missing something
[18:39] <zyga> hmm
[18:39] <zyga> the beta/2.45 branch is wrong as well
[18:39] <pedronis> zyga: also why would image build from master? I though edge built from master
[18:39] <zyga> not sure why, maybe it's just broken
[18:39] <zyga> snap    2.45.3
[18:39] <zyga> snapd   unknown
[18:39] <zyga> that's what's in that branch
[18:39] <cmatsuoka> mborzecki: I don't think we had any recent modification to that file, but I'll check
[18:39] <mborzecki> cmatsuoka: actually /run/mnt/ubuntu-boot
[18:39] <zyga> I don't know
[18:40] <pedronis> zyga: I installed the snapd from the branch and it looks correct here
[18:40] <pedronis> snap    2.45.3
[18:40] <pedronis> snapd   2.45.3
[18:40] <pedronis> series  16
[18:40] <pedronis> ubuntu  20.04
[18:40] <pedronis> kernel  5.4.0-42-generic
[18:40] <zyga> oh wait, perhaps it's my local config
[18:40] <mborzecki> cmatsuoka: there should be a header, `# Snapd-Boot-Config-Edition: 1`
[18:40] <zyga> yes sorry
[18:40] <zyga> I've disabled it for development
[18:40] <zyga> it's not broken, uff
[18:41] <ijohnson> mborzecki: w/o 9006, I see that in the /run/mnt/ubuntu-boot grub.cfg
[18:42] <zyga> I think this is wrong though
[18:42] <ijohnson> mborzecki: but I don't see that in grub.cfg for /run/mnt/ubuntu-seed for example
[18:42] <zyga> it's just the 2.45.3 version
[18:42] <zyga> if you follow the manifest
[18:42] <zyga> it leads here https://launchpad.net/~snappy-dev/+snap/snapd-2.45/+build/1060861
[18:42] <pedronis> zyga: anyway you put a snapd into /image and there is only a snapd package build there
[18:42] <zyga> ah, that's correct
[18:42]  * ijohnson tries with more hacks to patch grub.cfg from the gadget in nested.sh
[18:43] <pedronis> zyga: so still not understanding why it built from master
[18:43] <zyga> sorry, I'm paranoid that it's all broken now
[18:43] <zyga> the snap in the branch is correct
[18:43] <pedronis> yes
[18:43] <cmatsuoka> mborzecki: I have "# Snapd-Boot-Config-Edition: 1" in the first line there
[18:43] <pedronis> we need to understand the ppa
[18:43] <pedronis> s
[18:43] <pedronis> I thought that master built into edge, not image
[18:44] <zyga> the ppa contains a changelog bump
[18:44] <zyga> but I think that's only for subsequent edge builds that inherit that somehow
[18:44]  * zyga look at the PPA
[18:44] <mborzecki> ijohnson: have you installed new snap command and built an imag with it?
[18:45] <ijohnson> mborzecki: I just discarded the instance and I'm trying a fresh run with patched gadget snap
[18:45] <ijohnson> mborzecki: I just ran the nested spread test
[18:45] <zyga> pedronis: I think the instructions are wrong, we should dput a real .3 to the ppa or this makes no sense
[18:45] <ijohnson> mborzecki: although good point, maybe the nested spread test needs to be updated to point ubuntu-image at an updated snap command from the spread branch
[18:46] <pedronis> zyga: sorry, I thought that were the instructions, put a real .3 into the ppa
[18:46] <pedronis> are we reading them differently
[18:46] <pedronis> ?
[18:46] <zyga> pedronis: Go to the master branch and create a debian/changelog with a placeholder entry for 2.46 to ensure the next “ppa:snappy-dev/edge” build has the right version number
[18:46] <pedronis> yes
[18:46] <zyga> gbp doesn't run from the release branch
[18:47] <pedronis> that's edge
[18:47] <zyga> yes
[18:47] <pedronis> for image
[18:47] <zyga> there's only one dput in the instructions
[18:47] <pedronis> you need to build from the release branch
[18:47] <zyga> build what? the .deb?
[18:47] <pedronis> yes
[18:47] <zyga> which step is that?
[18:48] <zyga> I mean, something makes no sense, what's the point of the placeholder
[18:48] <zyga> if we also upload the real .3
[18:48] <zyga> are there two ppas?
[18:48] <pedronis> the placeholder is not meant for upload
[18:48] <pedronis> it exist for the next automatic edge build
[18:48] <zyga> so are you saying step 9 is to build the release branch again, not master?
[18:48] <pedronis> yes
[18:48] <cmatsuoka> mborzecki: aha, I think that's missing in my workflow
[18:48] <zyga> but gpb refuses to build from a release branch
[18:49] <pedronis> maybe that's the patch is about
[18:49] <pedronis> anyway right now we have in image a build of master
[18:49] <zyga> no, the patch is about debian being a symlink
[18:49] <pedronis> which I dont think we want
[18:49] <zyga> I agree, but that's "just" edge, right?
[18:50] <zyga> the question now is to decipher what to do to get the real .3
[18:50] <pedronis> core-beta builds from the image ppa
[18:50] <pedronis> so I think we need somehow to upload the right thing there
[18:50] <pedronis> though now it might get confused
[18:50] <pedronis> because we have a 2.45.3 already
[18:51] <zyga> it's a ppa, we can remove a package on demand
[18:51] <zyga> I think I botched the release :/
[18:51] <cachio> zyga, so .3 is almost ready right?
[18:51] <zyga> cachio: no
[18:51] <pedronis> snapd is ready
[18:51] <pedronis> but core not
[18:51] <zyga> cachio: it's somewhat in a mixed state for core
[18:52] <cmatsuoka> mborzecki: hmm, is setting UBUNTU_IMAGE_SNAP_CMD to the new snap binary enough? If it is, I'm doing it
[18:52] <pedronis> zyga: so you say gdp-buildpackage doesn't work for a branch?
[18:52] <zyga> pedronis: I can dpkg-buildpackage -S
[18:53] <zyga> pedronis: yes, it's a part of the git buildpackage scheme
[18:53] <zyga> you build from master
[18:53] <pedronis> and then magic?
[18:53] <zyga> you have several other branches that represent upstream and patch queue
[18:53] <zyga> master contains the imported orig tarball
[18:53] <zyga> and applied patches
[18:53] <zyga> I don't understand how it's used here
[18:53] <zyga> maybe there's a config file we're missing
[18:54] <zyga> I can build a regular source package without gbp
[18:54] <zyga> but at this point we're drifting from the instructions
[18:55] <zyga> pedronis: I use gbp in a toy package I maintain
[18:55] <zyga> but that package has real .orig tarballs and separate packaging git and upstream git
[18:57] <zyga> looking at what dpkg-buildpackage -S produced it looks like a valid source release
[18:57] <mup> PR snapcraft#3231 opened: file utils: rename get_tool_path() to get_snap_tool_path() <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3231>
[18:57] <zyga> shall we remove the .3, dput that and try again?
[18:57] <pedronis> zyga: yes, what we there is master, the diff is full of 2.46 stuff
[18:57] <zyga> ok, starting with package removal
[18:58] <pedronis> zyga: gpd-buildpackage has something called --git-ignore-branch and something called --git-export=TREEISH that might help
[18:58] <zyga> requested
[18:59] <zyga> pedronis: I think we need to ask mvo about this, I don't see what gbp is doing here, there are no orig tarballs anywhere
[18:59] <zyga> perhaps that's just something mvo does out of a habit
[18:59] <zyga> once he is back, that is
[19:00] <zyga> it's a native package so there's no orig anyway
[19:01] <zyga> ok, dputting real .3 now
[19:03] <zyga> hmm, the source package is 148MB
[19:03] <zyga> something is bonkers
[19:03] <pedronis> do you have a cache or something odd in your tree?
[19:04] <zyga> it contains .git
[19:04] <pedronis> that ended up in the source?
[19:04] <pedronis> mmh
[19:04] <zyga> yes
[19:04] <pedronis> I suppose that's something gdp-buildpackage doesn't do?
[19:04] <zyga> that's one particularly descriptive source package
[19:05] <zyga> yeah, gbp exports the tree (which is the imported orig tarball and all changes as exported quilt) and builds that
[19:05]  * zyga tries something 
[19:06] <zyga> gbp buildpackage -S --git-ignore-branch
[19:07] <mup> PR core#115 opened: live-build/hooks: add chroot hook to rm cloud-init config file we don't want <Created by anonymouse64> <https://github.com/snapcore/core/pull/115>
[19:07] <zyga> the config file for gbp is in debian/gpb.conf
[19:07] <pedronis> is that ignore working?
[19:08] <zyga> it's still running
[19:08] <pedronis> I suppose without it fails earlier?
[19:08] <zyga> yes, it doesn't do anything
[19:09] <zyga> ok, it finished
[19:09] <zyga> 3.3M source package
[19:10] <zyga> we need to wait for the package to be removed though
[19:10] <zyga>  File snapd_2.45.3.tar.xz already exists in Packages used in constructing the Ubuntu Core component of Snappy, but uploaded version has different contents. See more information about this error in https://help.launchpad.net/Packaging/UploadErrors.
[19:10] <zyga> removing is async
[19:10]  * zyga looks
[19:11] <zyga> https://help.launchpad.net/Packaging/PPA/Deleting
[19:11] <zyga> this claims we need to wait for up to 6 hours now
[19:12] <zyga> https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+delete-packages?field.name_filter=snapd&field.status_filter=&field.series_filter= shows the wrong package was deleted 13 minutes ago
[19:12] <zyga> I think for now that's that :/
[19:13] <zyga> I can dput again in the morning
[19:14] <zyga> or at night, 6 hours is probably maximum, so on average it will be 3 hours away
[19:14] <zyga> we may get it removed before midnight
[19:16] <ijohnson> cachio: cmatsuoka: so here's a log of uc20 nested boot where we get random reboots with additional systemd logging https://pastebin.ubuntu.com/p/7ftJpJJbzt/
[19:16] <ijohnson> cachio: as expected, the reboots don't all happen at the same or similar points in time, it seems entirely random to me
[19:17] <ijohnson> if you grep for `Linux version 5.4.0-42-` in that log, you will see all the spots were the system was rebooted
[19:19]  * zyga tries to rest for a few hours
[19:19] <zyga> will be back at midnight
[19:19] <zyga> o/
[19:19] <cmatsuoka> ijohnson: so, just after it gets a link-local address?
[19:20] <cmatsuoka> ijohnson: could you run it again to see if the log is similar?
[19:20] <ijohnson> cmatsuoka: well that's one such reboot
[19:20] <ijohnson> cmatsuoka: sure I can run it again
[19:22] <cmatsuoka> ijohnson: are you running that locally? could you check if there are any strange messages at hypervisor level? maybe it's something that's not visible by guests, it's the VM that power cycled
[19:22] <ijohnson> cmatsuoka: I'm running it on GCE
[19:22] <cmatsuoka> ah ok
[19:23] <cmatsuoka> so we'll never know
[19:23] <ijohnson> cmatsuoka: afaik this behavior never happens outside of gce
[19:24] <cmatsuoka> ijohnson: ok, only now I saw the sequence of reboots, it seems pretty random to me
[19:25] <cachio> ijohnson, checking
[19:25] <ijohnson> cachio: cmatsuoka: https://github.com/snapcore/snapd/pull/9065 is what I have been running locally to look at the logs
[19:25] <mup> PR #9065: debug: forward systemd journald output to serial console when booting for uc20 <Precious Logs> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9065>
[19:27] <cachio> ijohnson, it is 100% random
[19:27] <ijohnson> yeah
[19:28] <cachio> ijohnson, I also tried in a different cloud provider
[19:28] <cachio> and I saw the same reboots error
[19:28] <ijohnson> cachio: did you reproduce in a different cloud ?
[19:28] <cachio> yes
[19:28] <ijohnson> woah weird I didn't realize that
[19:28] <mup> PR snapd#9065 opened: debug: forward systemd journald output to serial console when booting for uc20 <Precious Logs> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9065>
[19:28] <cachio> thas's why I am trying to change the servers
[19:29] <cachio> and this kind of configuration because I think it is related to the architecture
[19:32] <ijohnson> hmm interesting
[19:32] <ijohnson> cmatsuoka: when you get a chance, mind taking a really quick look at https://github.com/snapcore/snapd/pull/8998 again ?
[19:32] <mup> PR #8998: tests/cmd/snap-bootstrap/initramfs-mounts: add test case for empty recovery_mode <Simple 😃> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8998>
[19:34] <cmatsuoka> ijohnson: sure, just let me collect some debug messages from the cmdline building code to see why snapd thinks the ubuntu-boot grub assets are not managed
[19:34] <ijohnson> k, no rush
[19:51] <pedronis> ijohnson: I reviewed #9035
[19:51] <mup> PR #9035: o/devicestate: save seeding/preseeding times for use with debug seeding api (3/N) <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9035>
[19:52] <ijohnson> pedronis: thanks I will address that feedback, we should have a 3rd person look at it, since I modified the code too I don't think my +1 is appropriate
[19:55] <pedronis> ijohnson: yes
[20:07] <mup> PR snapcraft#3229 closed: errors: introduce HostToolNotFoundError exception <enhancement> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3229>
[21:44] <mup> PR snapd#9066 opened: o/ifacestate: fix bug in snapsWithSecurityProfiles <Created by pedronis> <https://github.com/snapcore/snapd/pull/9066>
[22:02] <mup> PR snapcraft#3230 closed: lifecycle: check for snap using shutil.which() <enhancement> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3230>