/srv/irclogs.ubuntu.com/2020/07/27/#snappy.txt

mborzeckimorning05:29
mupPR snapd#9054 closed: bootloader/assets: generate bootloader assets from files <UC20> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9054>06:45
jameshit's nice to see green ticks from CI again08:43
pedronisyes08:48
pedronismborzecki: catching up on some things, and then I'll review 9006 first08:48
mborzeckipedronis: thanks!08:48
mborzeckiSaviq: hey, multipass does not plug removable media?08:49
zygagood morning08:55
mupPR snapd#9060 opened: interfaces/bultin/multipass: replace U+00A0 no-break space with simple space <Simple 😃> <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9060>08:56
pedroniszyga: hi08:58
zygahey pedronis08:58
zygasorry for starting late, it's extremely difficult to sleep lately08:58
zygaonly one day left though08:58
pedroniszyga: how do you want to proceed, start cutting the release this morning, or in the afternoon? I can provide a cleaned up changelog09:00
zygapedronis: I would like to look at the release branch now, run tests manually, review the instructions from Mvo and get everything ready for early afternoon09:01
pedroniszyga: ok09:01
pedroniszyga: I can do the part of the instruction that is producing the changelong content and cleaning it up09:01
pedroniszyga: ping me when you need that09:01
pedroniszyga: thank you09:01
zygathank you, that will be most welcome09:01
zyga100 patches between 2.25.2 and release head09:05
mborzeckizyga: anything unexpected there?09:37
zygamborzecki: nothing yet09:39
zygaI forgot we did some interface changes backports09:39
zygamost is just noise and test tweaks09:41
zygabrb09:44
pedroniszyga: the condesed and edited changelog from mvo tool is: https://gist.github.com/pedronis/c1577c450d53ff3c9fe308644772623209:45
zygaack, looking09:46
zyganicely readable09:47
zygaI've gone through all the patches and everything looks good09:50
zygaI'll run tests two more times while reading mvo's instructions09:50
pedronismborzecki: did a pass on #9006, let me know if you have questions09:58
mupPR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>09:58
zygaone thing I found in the changelogs that we don't have in mvo's doc is the SRU tracking bug10:02
zygafor 2.45 we used https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/187507110:02
mupBug #1875071: [SRU] 2.45 <id-5ef3c1ba59467589703d5ae0> <verification-done> <verification-done-bionic> <verification-done-eoan> <verification-done-focal> <verification-done-xenial> <snapd (Ubuntu):Fix Released> <snapd (Ubuntu Xenial):Fix Released> <snapd (Ubuntu Bionic):Fix Released> <snapd (Ubuntu10:02
mupEoan):Fix Released> <snapd (Ubuntu Focal):Fix Released> <https://launchpad.net/bugs/1875071>10:02
zygabut it seems to mention 2.45.1 as well10:02
zygaI wonder if I should file a placeholder bug for 2.45.310:02
zygafirst run: two tests failed in fedora10:03
zygawell, centos and fedora10:03
zyga    - google:centos-8-64:tests/main/selinux-clean10:03
zyga    - google:fedora-30-64:tests/main/snap-run10:03
zygabut worse, preparing arch failed entirely10:03
* zyga looks 10:03
mborzeckizyga: ah, w8, there was a patch for zstd10:08
zygamborzecki: I think I saw one in the patches10:08
zygait's not that10:09
mborzeckizyga: 4de033c77e742c166dd075fd5e9db507a99b17dc this one?10:09
zygahttps://paste.ubuntu.com/p/wdHDqkXFKy/10:09
mborzeckizyga: hm that one i cannot fix10:09
zygayes, that patch is in the release10:10
zygaheh, only on release day :)10:10
zygathat looks like google's own arch repo?10:10
mborzeckizyga: yeah, not sure where it's coming from10:10
mborzeckitbh i didn't know there's one even10:11
zygadifferent failure now10:16
zygaFound unused ./vendor packages:10:16
zyga vu github.com/snapcore/secboot/internal/pe1.1410:16
zygaPlease fix via 'govendor remove +unused'10:16
* zyga looks10:16
mborzeckizyga: pacman -Sy seems to work, so maybe there really is a repo, but it's only accessible form the nodes?10:17
mupPR snapd#9060 closed: interfaces/builtin/multipass: replace U+00A0 no-break space with simple space <Simple 😃> <Skip spread> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9060>10:21
mborzeckizyga: does it work now?10:27
zyganow things pass10:33
zygamborzecki: it's weird, I removed vendor bits and ran tests without them10:33
zygawhy didn't any other system complain?10:33
mborzeckizyga: i have no idea, arch should be using go 1.14.6 atm, maybe something about it10:34
zygamborzecki: about the other failures, the snap-run test hangs on strace sometimes so that's not new10:37
zygamborzecki: and the centos 8 selinux policy situation has not changed, correct?10:37
mborzeckizyga: nope, i should fix that test sometime10:37
mupPR snapd#9061 opened: packaging: add placeholder changelog for 2.45.3 <Created by zyga> <https://github.com/snapcore/snapd/pull/9061>11:16
zygachangelogs are ready, I created a PR for the placeholder entry, I've cloned the trello card and I'm looking at the card now11:16
zygaI don't think I've ever touched snapd translations on launchpad11:17
zyganot sure if anyone knows how that operates11:17
mborzeckizyga: mvo maybe (hopefully)?11:27
zygayeah, I wish I knew11:27
zygaI think we should rotate this more often11:27
zygafeels a bit "muscle memory"11:27
mupPR snapcraft#3227 opened: meta: fix content slot to set content when using source key <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3227>11:31
zygabrb11:47
zygare11:52
mborzeckipedronis: about https://github.com/snapcore/snapd/pull/9006#discussion_r460764489 you mean the same such that the order stays unchanged, i.e. mode/system as specified last?12:03
mupPR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>12:03
pedronismborzecki: the order yes, or anything else, formulated otherwise my question is whether the final command lines are changing or are the same12:06
pedronisvs what we have on master right now12:06
mborzeckipedronis: hm it does change, that's why i bumped the edition number to 2 initially12:09
pedronismborzecki: can you make a pastebin of old and new somewhere?12:09
pedronisside by side12:09
mborzeckipedronis: grub.cfg and recovery?12:10
pedronisthe command lines12:10
pedronisfor both12:10
mborzeckipedronis: ok, let me look at the scripts again, this has changed a bunch of times during the reviews12:11
mborzeckipedronis: heh, i was wrong, it changed twice in the course of reviews, at the end of the day it's the same as in master <mode> <static> <extra> (where <extra args> are a new thing and not used atm), see https://github.com/snapcore/snapd/pull/9006/files#diff-6f9b97376e62185c79af158d2d3c9474 and https://github.com/snapcore/snapd/pull/9006/files#diff-d3f0c595846626e6f78fe9b29f0a61e612:18
mupPR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>12:18
mborzeckipedronis: that diff chunk in boot is caused by a fix in bootloadertest mocks12:19
pedronismborzecki: good, my impression was also that in the end the result was the same12:20
mborzeckipedronis: while at it, i'll tweak the interface to have CommandLine(mode, system, extraArgs string)12:21
pedronissounds good12:22
pedronismborzecki: to be clear mode and system there  are full "name=value", right?12:23
mupPR snapcraft#3228 opened: review tools: fix snap link or copy path <bug> <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3228>12:26
mborzeckipedronis: yes, just like in the PR now, the only difference is the call will be eg. CommandLine("snapd_recovery_mode=recover", "snapd_recovery_system=123123", "arg=1 other=2")12:39
pedronisthx12:40
mborzeckipedronis: and grub bootloader implementation verifies snapd_recovery_mode= and snapd_recovery_system= prefixes12:40
cmatsuokamborzecki: the extra args in cmdline should come only from gadget or also from system settings?12:41
* zyga just got the call12:41
zyga12:30 tomorrow12:41
zygafinally12:41
mborzeckicmatsuoka: gadget for now, i recall we discussed some system args, but so far only gadget was something we though would happen12:42
cmatsuokaok. I asked because system settings is mentioned once in the doc intro but there are no details about that12:43
zygabrb, see you at the standup12:43
mborzeckizyga: nice, feeling anxious?12:43
zygano, just wish it was tomorrow already12:50
mupPR snapd#9062 opened: releasing package snapd version 2.45.3 <Created by zyga> <https://github.com/snapcore/snapd/pull/9062>13:12
mupPR snapd#9061 closed: packaging: add placeholder changelog for 2.45.3 <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9061>13:37
pedroniszyga: sorry I was confused, obviously edge builds from master normally, but there is this: https://launchpad.net/~snappy-dev/+snap/snapd-2.4513:41
zygaah, nice, it goes to beta directly13:41
zygaand it builds automatically13:42
pedronisit goes to a branch of beta to be clear13:43
zygahmm, right, I see that now13:44
pedroniszyga: and there's https://launchpad.net/~snappy-dev/+snap/core-beta13:45
pedronisfor core13:45
pedronisthat one is manual though13:45
zygaand this one is built on request, so we can trigger when the ppa is ready13:45
pedronisyes13:45
pedronisyou can trigger the other one as well if needed13:45
* zyga breaks for 30 minutes for meds and lunch13:53
pedroniszyga: lp:snapd will need to be imported though for snapd build to work with the right version14:01
ijohnsonpedronis: uc20 status meeting ?14:01
pedronisomw14:02
mborzeckipedronis: cmatsuoka: i've updated #900614:32
mupPR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>14:32
pedronismborzecki: looked, did you see this comment: https://github.com/snapcore/snapd/pull/9006#discussion_r460770887 ? is that something we should discuss?14:55
mupPR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>14:55
mupPR snapd#9062 closed: releasing package snapd version 2.45.3 <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9062>15:02
pedroniszyga: I triggered a repo import for that ^15:09
zygathank you15:10
zygaI tried looking at the patch for git-buildpackage, it's a bit unfortunate that we have to carry it15:10
pedroniszyga: https://code.launchpad.net/~niemeyer/snapd/+git/snapd15:11
zyga2.45 is still at the code from a few days ago15:11
pedroniszyga: yes, look, it's says an import will run soon15:12
pedronisit doesn't start immeditately15:12
zygayeah, refreshing the page shows stuff is changing15:12
zyga"Repository scan failed"15:13
zygaclicking the rescan button15:13
zyganot allowed15:13
zygafun15:13
zygaI guess15:13
zygawhat does that even mean?15:14
zygagit import into launchpad database?15:14
* zyga asks on #launchpad15:14
pedroniszyga: the import says it worked, but the branches aren't updated yet15:17
pedronisijohnson: are you blocked on reviews by me?15:18
zygathe pain is under control now, I'll get some tea and be back soon15:18
ijohnsonpedronis: no not at the moment15:18
ijohnsonpedronis: sorting out some things locally then I will push to pawel's debug seeding branch for the devicestate changes then open another PR with the command itself15:19
pedronisijohnson: ok, thx15:19
zygait worked now15:23
pedroniszyga: snapd:release/2.45 is up to date on LP now15:23
zygayep15:24
pedronisthee snapd-2.45 should start on its own soon15:24
pedronis*the snapd-2.45 build15:24
zygaI15:25
zygaI'll dput to the ppa now15:25
* cachio lunch15:32
zygalintian says we're a bit rusty on various details15:56
zygabut that's fine15:56
mborzeckizyga: which package isn't? :)15:57
zygamborzecki: hey, my toy package is clean like a ... clean package15:57
zygadput complete15:58
zygasorry I took forever, I kept struggling with gdp15:58
zyga*gbp15:58
pedroniszyga: so my theory was wrong, we have snapd snaps but they have ugly versions16:04
pedroniszyga: did you create the wrong kind of tag? or wasn't it pushed?16:07
pedronisI don't see the tag if I grab release 2.4516:07
zygahmmmmm16:07
zygaI opened a PR but that didn't carry the tag!16:07
zygaI guess I can just push it now16:07
mborzeckizyga: you need to push the tag16:07
zygadone16:07
mborzeckiand it's there16:08
zygaI'm sorry, I entirely missed that the tag is a separate entity16:08
zygawe need to rebuild the snap now, right?16:08
mborzeckihm it ended up on a weird commit, but i guess this is not a problem16:08
pedroniszyga: not sure, I'm still getting weird versions16:09
pedroniszyga: no, I fear how the tag was attached matters16:13
pedronisI get the right version if I go to the tag commit but I can the wrong one if I go to the tip of release/2.4516:13
zygahmmmm16:15
zygaI suspect I know why16:15
zygamvo pushes that to release directly16:15
zygaI could not16:15
zygaso I created a merge PR16:16
zygaso we have a new commit16:16
pedronislet me see if I can do something16:16
zygaok16:16
zygado we need another lp import?16:18
pedronisI haven't solved the problem, I'm not entirely sure how mvo pushes to those branches16:19
zygamaybe he has extra permissions16:20
zyganext time we should record mvo doing all those things and discuss the details16:21
zygappa is still building16:24
zygaah no, status was stale16:25
zygait failed16:25
zygaon aarch6416:25
zygaon unit tests16:25
ijohnsonpedronis: https://github.com/snapcore/snapd/pull/9035 is now ready for you to review16:25
mupPR #9035: o/devicestate: save seeding/preseeding times for use with debug seeding api (3/N) <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9035>16:25
zygaTestClientTimeoutLP183780416:26
zyga... error string = "cannot communicate with server: Post http://127.0.0.1:40893/: dial tcp 127.0.0.1:40893: i/o timeout"16:26
zyga... regex string = ".* timeout exceeded while waiting for response"16:26
pedroniszyga: now it should work, forced pushed to 2.45 the commit with the tag, instead of the merge16:31
zygagreat16:31
pedroniszyga: please double check that 2.45 is sane16:31
zygaso now we need another lp import16:31
zygasure, doing so now16:31
pedronisyes16:31
zygascheduled one just now16:32
pedronisthx16:32
zygadid you force push the tag?16:33
zygaI'm looking at https://github.com/snapcore/snapd/commit/b055d1902de062efe536f2c3f7dd7d2a35db2c6116:33
pedroniszyga: the tag is there: https://github.com/snapcore/snapd/tree/2.45.316:35
zygaah, it's just UI that confused me16:35
zygaok16:35
pedronisijohnson: thanks, I'll try to look still today16:36
ijohnsonpedronis: thanks, also https://github.com/snapcore/snapd/pull/9063 is the next one which is the snap debug command bit16:37
mupPR #9063: cmd/snap/debug: add "snap debug seeding" command for preseeding debugging (4/N) <Preseeding 🍞> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9063>16:37
ijohnson(which should be the last bit, I included the spread test bit there too, it's fairly small, but if you prefer I can make the spread test another PR)16:37
zygaoh, ijohnson, can you look at https://github.com/snapcore/snapd/pull/9057 please - I removed the environment entry, it was unjused anyway16:37
mupPR #9057: overlord: use new tracking cgroup for refresh app awareness <Created by zyga> <https://github.com/snapcore/snapd/pull/9057>16:37
zyga*unused16:37
mupPR snapd#9063 opened: cmd/snap/debug: add "snap debug seeding" command for preseeding debugging (4/N) <Preseeding 🍞> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9063>16:37
ijohnsonzyga: sure, I have an in progress review I kept getting distracted from :-)16:38
zygaOK16:38
pedronisijohnson: added to my queue16:38
ijohnsonzyga: and in case I don't get a chance to say it before tomorrow, good luck tomorrow! really hoping things go well for you16:38
ijohnsonthanks pedronis16:38
ijohnsonI will address some feedback and do reviews now, but then I will look into the re-exec bug for preseeding then16:39
pedroniszyga: I requested a manual build of the snap16:41
pedroniszyga: snapd is building now16:42
zygagreat, the lp import has succeeded as well16:42
ijohnsonzyga: +1d16:48
zygawoot16:48
zygathank you16:48
ijohnsonno, thank you for sticking with this, we're really close I think :-)16:48
* zyga listens to the thunder and rain outside16:49
zygaijohnson: and before the anniversary16:49
zygaof the v2 branch at least :)16:49
ijohnson:-D16:49
zygaonce this lands the rest is just code removal16:49
zygaand some misc changes unrelated to this feature16:49
zygathe versions now look good17:06
zygaon the beta/2.45 branch that is17:06
zygathe ppa is still building for some architectures17:07
zygaonce that is done we should trigger https://launchpad.net/~snappy-dev/+snap/core-beta17:07
zygaI will break for an hour and come back17:07
zyganeed to double check my hospital bag17:07
zygaand see what's going on at home17:08
mupPR snapcraft#3229 opened: errors: introduce HostToolNotFoundError exception <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3229>17:11
cachioijohnson, hey17:17
ijohnsonhi cachio17:17
ijohnsonI was just about to break for lunch17:17
ijohnson(just fyi)17:17
cachiodo you know what happens during initial boot after the line systemd[1]: Started Journal Service.  --> https://paste.ubuntu.com/p/cBK2cnG6NR/17:18
cachioijohnson, I se most of the random reboots after that line17:18
cachioijohnson, np17:18
ijohnsoncachio: looking quickly17:18
cachioijohnson, also after line systemd-journald[185]: Received client request to flush runtime journal.17:19
ijohnsoncachio: I think that just happens to be one of the last things that systemd will do before starting up snapd, etc.17:20
ijohnsonI don't think what is triggering those lines specifically would be what is causing the random reboot17:20
ijohnsonbut I dunno maybe it is17:20
cachioijohnson, so then snap starts17:20
ijohnsoncachio: but now immediately after that point17:20
ijohnsoncachio: but not immediately after that point17:21
cachioijohnson, ok17:21
ijohnsonI have some ideas for more debugging, but I need to break for lunch now17:21
=== ijohnson is now known as ijohnson|lunch
cachioijohnson|lunch, is it any way to see a log with that info?17:21
cachioafter that line in the log that I got from the nested vm I see17:22
cachio[   14.067916] systemd-journald[539]: Received client request to flush runtime journal.17:22
cachiocmatsuoka, any idea?17:31
mupPR snapcraft#3228 closed: review tools: fix snap link or copy path <bug> <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3228>17:32
mupPR snapcraft#3230 opened: lifecycle: check for snap using shutil.which() <enhancement> <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3230>17:32
cmatsuokalet me see...17:32
cmatsuokacachio: this is the entire log, and the system crashed after that?17:34
cachiocmatsuoka, after that the system works well17:35
cachiothis is the full log17:35
cmatsuokabut when it crashes, it crashes at that point after userspace is running, is that correct?17:35
cachio[   62.115886] systemd[1]: Starting Remount Root and Kernel File Systems...17:36
cachio[   62.154860] systemd[1]: Starting udev Coldplug all Devices...17:36
cachio[   62.201278] systemd[1]: Finished Create list of static device nodes for the current kernel.17:36
cachio[   62.237719] systemd[1]: modprobe@drm.service: Succeeded.17:36
cachio[   62.263067] systemd[1]: Finished Load Kernel Module drm.17:36
cachio[   62.275837] systemd[1]: Started Journal Service.17:36
mupPR snapcraft#3227 closed: meta: fix content slot to set content when using source key <bug> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3227>17:37
cachioafter that line at some point it crashed17:37
cmatsuokamaybe addming something like systemd.log_level=debug systemd.log_target=console to the cmline could help?17:37
cachiocmatsuoka, but not log17:37
cachiocmatsuoka, is it possible to update the command line with tpm and secboot enabled??17:38
cmatsuokahmm, you would need to add the parameters to the cmdlines used in sealing too17:39
cmatsuokaunless it crashes before you unlock the encrypted device17:39
cmatsuokain this case you could boot with any cmdline17:40
=== ijohnson|lunch is now known as ijohnson
cmatsuokabut you won't be able to go past beyond the writable fs mount17:40
ijohnsoncmatsuoka: cachio: what I was going to propose is a test PR that changes the cmdline everywhere to include taht systemd.log_level kernel cmdline just to see the boot log17:41
ijohnsonI presume it happens every time so opening a PR with "run nested" label should be enough to trigger it and get some more representative logs17:41
cachioijohnson, ok17:42
ijohnsonthe other issue is that if it is crashing instead of rebooting, we may lose some of the logs as they don't get flushed to GCE in time17:42
cmatsuokaijohnson, cachio: having an instrumented PR is probably the way to go17:42
ijohnsoncachio: I will send a PR then17:42
cachionice17:42
cachiothanks17:42
ijohnsoncmatsuoka: do you know if it's sufficient to change the kernel cmdline inside snapd now with mborzecki's changes, or do I still need to make changes to the gadget grub.cfg ?17:42
ijohnsoncmatsuoka: I don't know how far along mborzecki's changes are17:43
cmatsuokaijohnson: uuuhm, I don't know if mborzecki's changes are already effective17:43
cmatsuokaif not, the hard-coded cmdline is in gadget/install/install.go17:43
ijohnsonI think they might be effective already, I tried something similar late last week and found that snapd was writing the grub.cfg17:44
cmatsuokaat some point last week I had to change the hardcoded cmdline, but I don't know if I had master merged in that PR17:45
* cmatsuoka checks 17:45
cmatsuokaah, I think you're right, in my test I edited the bootloader line by hand17:49
cmatsuoka(it was a local test, not a test PR)17:49
cmatsuokaI have a local branch where I changed the hardcoded lines with calls to maciek's command line building, hence the confusion17:51
mupPR snapd#9064 opened: .github/workflows: move snap building to test.yaml as separate cached job <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9064>17:53
mborzeckizyga: how's it going?17:55
zygamborzecki: let me chcek17:55
zygabuilds were going17:55
mborzeckinice, ijohnson opened https://github.com/snapcore/snapd/pull/906417:56
mupPR #9064: .github/workflows: move snap building to test.yaml as separate cached job <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9064>17:56
ijohnsonhey mborzecki17:56
ijohnsonquestion on your grub.cfg stuff17:56
mborzeckizyga: and upstream monitoring already opened https://bugzilla.redhat.com/show_bug.cgi?id=1861024 for the update17:56
ijohnsonhow much of that is operational right now ?17:56
ijohnsoni.e. if I wanted to change the kernel cmdline that a uc20 system is booted with for debugging, can I just change the grub.cfg and the internal edition snippet in snapd Go code, or do I still need to make some changes to the gadget's grub.cfg ?17:57
mborzeckiijohnson: in what sense? grub.cfg (and recovery) is already written from snapd17:57
zygahttps://launchpad.net/~snappy-dev/+snap/snapd-2.45 is building again17:57
zygahmm17:57
ijohnsonmborzecki: right so that should mean that if I build a fresh uc20 image with snapd from master with my modified kernel cmdline, it should be used and sealed against, correct ?17:58
mborzeckiijohnson: soo, https://github.com/snapcore/snapd/pull/8947 hasn't landed yet, so an update won't happen17:58
mupPR #8947: many: update managed boot config when refreshing snapd <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8947>17:58
ijohnsonmborzecki: but for a new install17:58
ijohnsonno updates17:58
mborzeckiijohnson: iirc the sealing code in gadget/install.Run is still using the harcoded cmdline17:59
cmatsuokamborzecki: it is, I'm updating it now17:59
ijohnsonmborzecki: ah good point yes I will have to change that too, but I was more wondering about the grub.cfg's that get written out to disk, if the recovery or ubuntu-boot grub.cfg's from the gadget are used anymore at all18:00
zygamborzecki: yeah, I got that18:00
zygafedbus is nice18:00
mborzeckiijohnson: so with #9006 landed, we'd still need a change for the install.Run()18:00
mupPR #9006: bootloader: compose command line with mode and extra arguments <Squash-merge> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>18:00
mborzeckicmatsuoka: cool, no issues so far?18:00
cmatsuokamborzecki: I just updated the code and will boot an image to see if it works as expected18:00
zygaI wonder when I should click https://launchpad.net/~snappy-dev/+snap/core-beta18:01
mborzeckiijohnson: then with claudio's change, if you use custom snapd with `dangerous system.debug-shell=1` added to snapd_static_cmdline and the snippet it should work18:01
mborzeckiijohnson: and you need to edit both the snippet in bootloader/assets/grub.go and bootloader/assets/data/grub{.cfg,-recovery.cfg} because we've decided against extracting the static bits during build time18:02
ijohnsonmborzecki: so do I need 9006 for a fresh install / first boot ?18:03
* ijohnson feels like the answer is no18:03
mborzeckihmm i think no18:03
mborzeckiijohnson: though i would not mind if you tried with 9006 :)18:04
ijohnsonmborzecki: haha ok I will try with 900618:04
mborzeckiijohnson: this and the branch cmatsuoka is working on18:04
ijohnsonmborzecki: well for the branch cmatsuoka I just made the kernel command line I need show up everywhere manually18:05
cmatsuokamborzecki: I got a "not implemented" error in master, is that expected?18:05
ijohnsonso the install code doesn't pull it dynamically yet it just happens to still be the same :-)18:05
mborzeckicmatsuoka: ah yes, so you need 9006 too18:05
cmatsuokamborzecki: ok18:05
pedronismborzecki: I commented a bit more: https://github.com/snapcore/snapd/pull/9006#discussion_r461075416 , we can chat in the morning if needed18:10
mupPR #9006: bootloader: compose command line with mode and extra arguments <Squash-merge> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>18:10
mborzeckipedronis: ok, thanks18:10
mborzeckipedronis: haha, ok i see what you mean now and understand why we seem to have different ideas as to what the thing does, let's chat in the morning18:14
pedronismborzecki: remember that we need to be able to know what to expect at the next boot before writing things to disk18:15
mborzeckipedronis: yes, i wanted to have a separate call for that, something like CandidateCommandLine() with the same fingerprint, that would use the edition number from the internal asset rather than the disk18:17
mborzeckianyways, let's chat in the morning18:17
mupPR core20#78 opened: hooks/900-cleanup-etc-var.chroot: rm cloud-init config file which we don't want <Created by anonymouse64> <https://github.com/snapcore/core20/pull/78>18:19
cmatsuokamborzecki, ijohnson: I'll have to debug a little bit here, unlocking failed18:19
ijohnsoncmatsuoka: my nested run is still going, not sure how far along it has gotten tbh18:20
ijohnsonit's been here for 5 minutes now18:20
ijohnson2020-07-27 13:15:48 Preparing google-nested:ubuntu-20.04-64:tests/nested/core20/ (jul271804-169453)...18:20
mupPR core18#166 opened: hooks/900-cleanup-etc-var.chroot: rm cloud-init config file which we don't want <Created by anonymouse64> <https://github.com/snapcore/core18/pull/166>18:26
* zyga is unsure what to do with the release now, sorry, my mind is all messed up 18:29
pedroniszyga: we have snapd in beta/2.4518:31
pedroniszyga: did the package build in the ppa?18:31
zygayes, I think so18:32
zygayes, for all arches now18:32
zygaI can see the beta/2.45 branch builds as well18:33
zygaI should push those over to beta18:33
mborzeckizyga: btw. do you have permissions to upload the source tarballs to releases page on github?18:33
pedroniszyga: the changelog looks like from master though18:33
pedronisin the ppa, bit confused18:34
zygamborzecki: let me check, I don't know for sure18:34
zygaI think everyone does18:34
zygaI do18:34
pedroniszyga: if you look here: https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+packages the changelong is weird18:34
pedronisit has the placeholder18:34
zygait is the placeholder, because that's the package that was built from master, not from the release18:35
cmatsuokamborzecki: I can retrieve the recover mode cmdline correctly, but the run mode cmdline is returning empty for some reason18:35
zygathat's step 8 from https://docs.google.com/document/d/1BCOwqjdUZIg-3x9E8iyojUW_P-ZJJjG0fFLeaDZbp5A/edit#18:36
pedroniszyga: which one is the one for the release?18:36
zygapedronis: only snapd.snap is correct at this point18:37
pedroniszyga: do we understand how to build a correct core?18:37
ijohnsonmborzecki: cmatsuoka: my nested run finished, it seems that we still need to modify the gadget grub.cfg, my changes only took effect for run mode18:38
ijohnsonwhich is fine, I can do that18:38
zygaI'm confused about that - way back when I last did this it was built from the ppa directly but that has changed in ways I didn't follow since18:38
zygaso back then a dput to that ppa was to provide the actual new snapd18:38
mborzeckicmatsuoka: hm, can you check that ubuntu-boot contains the right grub.cfg that matches what's in 9006?18:38
zygaI think the instructions are not complete and we're missing something18:38
zygahmm18:39
zygathe beta/2.45 branch is wrong as well18:39
pedroniszyga: also why would image build from master? I though edge built from master18:39
zyganot sure why, maybe it's just broken18:39
zygasnap    2.45.318:39
zygasnapd   unknown18:39
zygathat's what's in that branch18:39
cmatsuokamborzecki: I don't think we had any recent modification to that file, but I'll check18:39
mborzeckicmatsuoka: actually /run/mnt/ubuntu-boot18:39
zygaI don't know18:39
pedroniszyga: I installed the snapd from the branch and it looks correct here18:40
pedronissnap    2.45.318:40
pedronissnapd   2.45.318:40
pedronisseries  1618:40
pedronisubuntu  20.0418:40
pedroniskernel  5.4.0-42-generic18:40
zygaoh wait, perhaps it's my local config18:40
mborzeckicmatsuoka: there should be a header, `# Snapd-Boot-Config-Edition: 1`18:40
zygayes sorry18:40
zygaI've disabled it for development18:40
zygait's not broken, uff18:40
ijohnsonmborzecki: w/o 9006, I see that in the /run/mnt/ubuntu-boot grub.cfg18:41
zygaI think this is wrong though18:42
ijohnsonmborzecki: but I don't see that in grub.cfg for /run/mnt/ubuntu-seed for example18:42
zygait's just the 2.45.3 version18:42
zygaif you follow the manifest18:42
zygait leads here https://launchpad.net/~snappy-dev/+snap/snapd-2.45/+build/106086118:42
pedroniszyga: anyway you put a snapd into /image and there is only a snapd package build there18:42
zygaah, that's correct18:42
* ijohnson tries with more hacks to patch grub.cfg from the gadget in nested.sh18:42
pedroniszyga: so still not understanding why it built from master18:43
zygasorry, I'm paranoid that it's all broken now18:43
zygathe snap in the branch is correct18:43
pedronisyes18:43
cmatsuokamborzecki: I have "# Snapd-Boot-Config-Edition: 1" in the first line there18:43
pedroniswe need to understand the ppa18:43
pedroniss18:43
pedronisI thought that master built into edge, not image18:43
zygathe ppa contains a changelog bump18:44
zygabut I think that's only for subsequent edge builds that inherit that somehow18:44
* zyga look at the PPA18:44
mborzeckiijohnson: have you installed new snap command and built an imag with it?18:44
ijohnsonmborzecki: I just discarded the instance and I'm trying a fresh run with patched gadget snap18:45
ijohnsonmborzecki: I just ran the nested spread test18:45
zygapedronis: I think the instructions are wrong, we should dput a real .3 to the ppa or this makes no sense18:45
ijohnsonmborzecki: although good point, maybe the nested spread test needs to be updated to point ubuntu-image at an updated snap command from the spread branch18:45
pedroniszyga: sorry, I thought that were the instructions, put a real .3 into the ppa18:46
pedronisare we reading them differently18:46
pedronis?18:46
zygapedronis: Go to the master branch and create a debian/changelog with a placeholder entry for 2.46 to ensure the next “ppa:snappy-dev/edge” build has the right version number18:46
pedronisyes18:46
zygagbp doesn't run from the release branch18:46
pedronisthat's edge18:47
zygayes18:47
pedronisfor image18:47
zygathere's only one dput in the instructions18:47
pedronisyou need to build from the release branch18:47
zygabuild what? the .deb?18:47
pedronisyes18:47
zygawhich step is that?18:47
zygaI mean, something makes no sense, what's the point of the placeholder18:48
zygaif we also upload the real .318:48
zygaare there two ppas?18:48
pedronisthe placeholder is not meant for upload18:48
pedronisit exist for the next automatic edge build18:48
zygaso are you saying step 9 is to build the release branch again, not master?18:48
pedronisyes18:48
cmatsuokamborzecki: aha, I think that's missing in my workflow18:48
zygabut gpb refuses to build from a release branch18:48
pedronismaybe that's the patch is about18:49
pedronisanyway right now we have in image a build of master18:49
zygano, the patch is about debian being a symlink18:49
pedroniswhich I dont think we want18:49
zygaI agree, but that's "just" edge, right?18:49
zygathe question now is to decipher what to do to get the real .318:50
pedroniscore-beta builds from the image ppa18:50
pedronisso I think we need somehow to upload the right thing there18:50
pedronisthough now it might get confused18:50
pedronisbecause we have a 2.45.3 already18:50
zygait's a ppa, we can remove a package on demand18:51
zygaI think I botched the release :/18:51
cachiozyga, so .3 is almost ready right?18:51
zygacachio: no18:51
pedronissnapd is ready18:51
pedronisbut core not18:51
zygacachio: it's somewhat in a mixed state for core18:51
cmatsuokamborzecki: hmm, is setting UBUNTU_IMAGE_SNAP_CMD to the new snap binary enough? If it is, I'm doing it18:52
pedroniszyga: so you say gdp-buildpackage doesn't work for a branch?18:52
zygapedronis: I can dpkg-buildpackage -S18:52
zygapedronis: yes, it's a part of the git buildpackage scheme18:53
zygayou build from master18:53
pedronisand then magic?18:53
zygayou have several other branches that represent upstream and patch queue18:53
zygamaster contains the imported orig tarball18:53
zygaand applied patches18:53
zygaI don't understand how it's used here18:53
zygamaybe there's a config file we're missing18:53
zygaI can build a regular source package without gbp18:54
zygabut at this point we're drifting from the instructions18:54
zygapedronis: I use gbp in a toy package I maintain18:55
zygabut that package has real .orig tarballs and separate packaging git and upstream git18:55
zygalooking at what dpkg-buildpackage -S produced it looks like a valid source release18:57
mupPR snapcraft#3231 opened: file utils: rename get_tool_path() to get_snap_tool_path() <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3231>18:57
zygashall we remove the .3, dput that and try again?18:57
pedroniszyga: yes, what we there is master, the diff is full of 2.46 stuff18:57
zygaok, starting with package removal18:57
pedroniszyga: gpd-buildpackage has something called --git-ignore-branch and something called --git-export=TREEISH that might help18:58
zygarequested18:58
zygapedronis: I think we need to ask mvo about this, I don't see what gbp is doing here, there are no orig tarballs anywhere18:59
zygaperhaps that's just something mvo does out of a habit18:59
zygaonce he is back, that is18:59
zygait's a native package so there's no orig anyway19:00
zygaok, dputting real .3 now19:01
zygahmm, the source package is 148MB19:03
zygasomething is bonkers19:03
pedronisdo you have a cache or something odd in your tree?19:03
zygait contains .git19:04
pedronisthat ended up in the source?19:04
pedronismmh19:04
zygayes19:04
pedronisI suppose that's something gdp-buildpackage doesn't do?19:04
zygathat's one particularly descriptive source package19:04
zygayeah, gbp exports the tree (which is the imported orig tarball and all changes as exported quilt) and builds that19:05
* zyga tries something 19:05
zygagbp buildpackage -S --git-ignore-branch19:06
mupPR core#115 opened: live-build/hooks: add chroot hook to rm cloud-init config file we don't want <Created by anonymouse64> <https://github.com/snapcore/core/pull/115>19:07
zygathe config file for gbp is in debian/gpb.conf19:07
pedronisis that ignore working?19:07
zygait's still running19:08
pedronisI suppose without it fails earlier?19:08
zygayes, it doesn't do anything19:08
zygaok, it finished19:09
zyga3.3M source package19:09
zygawe need to wait for the package to be removed though19:10
zyga File snapd_2.45.3.tar.xz already exists in Packages used in constructing the Ubuntu Core component of Snappy, but uploaded version has different contents. See more information about this error in https://help.launchpad.net/Packaging/UploadErrors.19:10
zygaremoving is async19:10
* zyga looks19:10
zygahttps://help.launchpad.net/Packaging/PPA/Deleting19:11
zygathis claims we need to wait for up to 6 hours now19:11
zygahttps://launchpad.net/~snappy-dev/+archive/ubuntu/image/+delete-packages?field.name_filter=snapd&field.status_filter=&field.series_filter= shows the wrong package was deleted 13 minutes ago19:12
zygaI think for now that's that :/19:12
zygaI can dput again in the morning19:13
zygaor at night, 6 hours is probably maximum, so on average it will be 3 hours away19:14
zygawe may get it removed before midnight19:14
ijohnsoncachio: cmatsuoka: so here's a log of uc20 nested boot where we get random reboots with additional systemd logging https://pastebin.ubuntu.com/p/7ftJpJJbzt/19:16
ijohnsoncachio: as expected, the reboots don't all happen at the same or similar points in time, it seems entirely random to me19:16
ijohnsonif you grep for `Linux version 5.4.0-42-` in that log, you will see all the spots were the system was rebooted19:17
* zyga tries to rest for a few hours19:19
zygawill be back at midnight19:19
zygao/19:19
cmatsuokaijohnson: so, just after it gets a link-local address?19:19
cmatsuokaijohnson: could you run it again to see if the log is similar?19:20
ijohnsoncmatsuoka: well that's one such reboot19:20
ijohnsoncmatsuoka: sure I can run it again19:20
cmatsuokaijohnson: are you running that locally? could you check if there are any strange messages at hypervisor level? maybe it's something that's not visible by guests, it's the VM that power cycled19:22
ijohnsoncmatsuoka: I'm running it on GCE19:22
cmatsuokaah ok19:22
cmatsuokaso we'll never know19:23
ijohnsoncmatsuoka: afaik this behavior never happens outside of gce19:23
cmatsuokaijohnson: ok, only now I saw the sequence of reboots, it seems pretty random to me19:24
cachioijohnson, checking19:25
ijohnsoncachio: cmatsuoka: https://github.com/snapcore/snapd/pull/9065 is what I have been running locally to look at the logs19:25
mupPR #9065: debug: forward systemd journald output to serial console when booting for uc20 <Precious Logs> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9065>19:25
cachioijohnson, it is 100% random19:27
ijohnsonyeah19:27
cachioijohnson, I also tried in a different cloud provider19:28
cachioand I saw the same reboots error19:28
ijohnsoncachio: did you reproduce in a different cloud ?19:28
cachioyes19:28
ijohnsonwoah weird I didn't realize that19:28
mupPR snapd#9065 opened: debug: forward systemd journald output to serial console when booting for uc20 <Precious Logs> <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9065>19:28
cachiothas's why I am trying to change the servers19:28
cachioand this kind of configuration because I think it is related to the architecture19:29
ijohnsonhmm interesting19:32
ijohnsoncmatsuoka: when you get a chance, mind taking a really quick look at https://github.com/snapcore/snapd/pull/8998 again ?19:32
mupPR #8998: tests/cmd/snap-bootstrap/initramfs-mounts: add test case for empty recovery_mode <Simple 😃> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8998>19:32
cmatsuokaijohnson: sure, just let me collect some debug messages from the cmdline building code to see why snapd thinks the ubuntu-boot grub assets are not managed19:34
ijohnsonk, no rush19:34
pedronisijohnson: I reviewed #903519:51
mupPR #9035: o/devicestate: save seeding/preseeding times for use with debug seeding api (3/N) <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9035>19:51
ijohnsonpedronis: thanks I will address that feedback, we should have a 3rd person look at it, since I modified the code too I don't think my +1 is appropriate19:52
pedronisijohnson: yes19:55
mupPR snapcraft#3229 closed: errors: introduce HostToolNotFoundError exception <enhancement> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3229>20:07
mupPR snapd#9066 opened: o/ifacestate: fix bug in snapsWithSecurityProfiles <Created by pedronis> <https://github.com/snapcore/snapd/pull/9066>21:44
mupPR snapcraft#3230 closed: lifecycle: check for snap using shutil.which() <enhancement> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3230>22:02

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!