/srv/irclogs.ubuntu.com/2020/03/18/#snappy.txt

mborzeckimorning06:40
mupPR snapd#8289 opened: xdgopenproxy: forward requests to the desktop portal <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/8289>06:45
mborzeckihmm the code in master isn't formatted according to gofmt 1.10?06:58
mborzeckijamesh: thanks for opening the PR! thre's some unit tests failures in the travis job07:26
jameshmborzecki: yep.  Just looking into that (borrowing some logic from gio)07:26
mupPR snapd#8290 opened: run-checks: tweak formatting checks <Simple 😃> <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8290>07:27
jameshmborzecki: I was adjusting the testing to run against a real bus rather than trying to mock everything07:30
mborzeckijamesh: right, userd tests did that too iirc07:32
mborzeckimvo: hey, a trivial PR to start your day with: https://github.com/snapcore/snapd/pull/829007:41
mupPR #8290: run-checks: tweak formatting checks <Simple 😃> <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8290>07:41
mvomborzecki: sure!07:42
mupPR snapd#8288 closed: release: 2.44 <Simple 😃> <Skip spread> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8288>07:44
zygagood morning07:55
mvogood morning zyga07:56
zygahow is everyone?07:56
zygait looks like a record-hot day today07:56
zygaI gave up and bought a disk for debian :/07:57
zygaI could not find any spare drives at home, that were not filled with stuff07:57
zygadid you guys noticed the github app went live yesterday07:59
zygaI tried it and it's pretty slick07:59
zygait feels almost better than using the browser on a desktop07:59
mborzeckizyga: instead of just one browser you can run n instances now :)08:05
pstolowskimorning08:06
zygahey :)08:06
zygapstolowski: still waiting?08:06
pstolowskizyga: yep; not even dispatched08:06
zygait will come08:06
zygajust thinking about some stuff I talked about with mborzecki earlier08:07
zygamborzecki: RGB HDD FTW08:07
zygamvo: is 2.44 out?08:11
mvozyga: yes, since last night in beta08:11
zygawoot08:11
zygais that expected to be the 2.44 final that goes out to ISOs?08:11
mvozyga: there will be a 2.44.1 that will probably go on the iso08:12
mvozyga: with search v208:12
zygaaha08:12
zygaok08:12
zygashould I target fixes to it or to 2.4508:12
zygathinking about stuff like EROFS from yesterday08:12
mvozyga: fixes for 2.4408:12
mvozyga: please mark everything that is worth for 2.4408:12
zygahttps://github.com/snapcore/snapd/pull/8285 is like one08:12
mupPR #8285: cmd/snap-update-ns: ignore EROFS from rmdir/unlink <Bug> <Created by zyga> <https://github.com/snapcore/snapd/pull/8285>08:12
mvozyga: we don't really have time for a 2.45 before the release08:12
zygasure08:13
zygaI was thinking if we are doing any .1 or not at all08:13
mvozyga: yeah, most likley a .108:13
mvozyga: (like ~95%)08:13
mvozyga: I added the 2.44 milestone to you pr and will cherry-pick08:13
zygaI'll make a milestone on LP and retarget thebug08:13
mborzeckizyga:  haha rgb hdd, how about a transparent rgb hdd with a led platter  on top of the magnetic ones? :)08:14
zygamborzecki: as soon as we have that transparent aluminum ;)08:15
zygamborzecki: but I would totally buy an RGB HDD08:15
zygabecause, why not :)08:15
mborzeckizyga: heh, already auto-filed https://bugzilla.redhat.com/show_bug.cgi?id=181455208:41
zygayeah I got the mail ping a moment ago08:41
zygaITSABUGFIXITNOOOW08:41
zygamvo: will you make the release tarballs?08:42
zygabrb08:49
mvozyga: sure08:54
zygathank you :)09:01
mvopedronis: I pushed an update to the nocloud, probably needs a lot of naming tweaks09:20
pedronismvo: #8278 ?09:22
mupPR #8278: devicestate: disable cloud-init by default on uc20 <Simple 😃> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8278>09:22
mvopedronis: yes09:22
pedronismvo: ok, I will look at it, looking at the next secboot PR atm09:22
mvopedronis: no rush, thanks!09:23
mvopedronis: actually let me tweak it a bit more, I think it's silly at this point to split it into two09:26
mborzeckii can't decide whether go allowing to embed `*type` and `type` is good or bad thing09:28
pstolowskizyga: question to the env fix PR09:37
zygapstolowski: ok09:41
pstolowskizyga: ah, it's splitN, so it's fine09:42
zygaheh, I just replied the same :)09:42
zygaI believe this is also tested09:42
zygathere's a test like K=V=...09:43
zygaanyway09:43
zygaI need to take half day off09:43
zygauntil we figure out home schooling09:43
zygaLucy is constantly with me09:43
zygacrying if she is with grandparents09:43
zygaand I just cannot focus09:43
zygauntil enough discipline and getting over the fact working at home is working is understood by older kids09:44
zygasorry, I'll file the time off later :/09:44
zygawe're walking up and down the stairs all morning09:44
mupPR snapd#8234 closed: devicestate: support for "cloud.cfg.d" cloud-init in uc20 with grade: dangerous <Needs Samuele review> <UC20> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8234>09:51
pedronismvo: I'm a bit unsure why you think is silly to split it? I asked it to be split that way09:55
pedronismvo: what I mean, please don't merge 8234 into 827810:02
mvopedronis: ok, I will do them separately, sorry10:04
mvopedronis: then 8278 should be ok for a first look, probably lots of tweaks still needed but hopefully it captures what we discussed10:05
pedronismvo: 8278 is not that small anyway, with more stuff it would maybe >500 lines10:06
mvopedronis: yeah, makes sense10:06
pedronismborzecki: how close do you think is #8159, apart your comment?10:24
mupPR #8159: snap-bootstrap: remove created partitions on reinstall <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8159>10:24
mborzeckipedronis: it's just some nitpicks10:24
mborzeckipedronis: i think a helper called from Run() before CreateMising() that explicity mutates LaidOutVolume based on options would be fine there10:25
pedronismborzecki: I see, I sort of feel there already too many helpers but maybe is just me10:25
mborzeckipedronis: perhaps that should even be something in gadget, since gadget is looked at from other places too (eg. boot assets update)10:28
mborzeckipedronis: wdyt?10:28
ijohnsonmorning folks10:28
mborzeckiijohnson: hey!10:29
ijohnsonhey mborzecki o/10:29
pedronismborzecki: I honestly wonder if overreding what the gadget says is worth the trobule10:29
pedronisthe specs says anyway that the normal partitions can be luks10:30
pedronisafaiu10:30
mborzeckipedronis: leave a type as it is then?10:30
pedronisyes10:30
pedronismborzecki: see here: https://www.freedesktop.org/wiki/Specifications/DiscoverablePartitionsSpec/10:31
mborzeckipedronis: mhm, cryptsetup open probably does not even look at the partition type in gpt/mbr10:31
pedronisit's cute to have the "right" type but atm it just seem grief10:32
mborzeckihaha fair enough10:32
pedronisnotice that I didn't request that10:32
pedronisalso as you said, not sure other gadget code will be happy10:32
pedronisabout the changed type10:32
ijohnsonany idea why snap prepare-image would fail to find the account-key here: https://pastebin.ubuntu.com/p/Qz3RvprPK6/, this is from #828610:34
mupPR #8286: tests: cleanup various uc20 boot tests from previous PR <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8286>10:34
ackkzyga, hi, is https://paste.ubuntu.com/p/Yv9dkYThZr/ the same issue with mounts you're fixing?10:41
mvozyga: hrm, hrm, the debian package fails with https://paste.ubuntu.com/p/P674rpMTMB/ which in my pbuilder, which is super strange given that this is building travis, I wonder if we do something silly there like not excluding the vendor stuff or sillyness like this10:41
mborzeckipedronis: ok, let's discuss that with claudio after the standup maybe?10:44
pedronismvo: I +1ed one 8278 but it needs a 2nd review given all the changes, also Opts name need to change. Lots of details are still unclear to me but shouldn't be a blocker for what is there currently10:47
zygaackk: no, you cannot chown $SNAP10:47
zygahmmm10:47
ijohnsonpedronis: I'm reviewing 8278 right now10:48
zygamvo: weird10:48
zygais golang-evdev in the vendor tarball>10:48
zygaor are we de-vendoring?10:48
zygaI mean, are we really building the devendorized tarball in our spread runs?10:48
pedroniswe shouldn't be building things that need on debian10:48
pedronisneed it10:48
pedronisanyway10:48
zygapedronis: good point10:48
zygaLucy is asleep now but I think today is rather hectic :10:48
pedronismaybe we are because10:48
zyga:/10:48
pedronismborzecki: I'm not sure there is more to discuss, that PR is open since forever10:49
mvopedronis: yeah, I'm unhappy that we did not caught this, the sbuild nightly test is failing since days and it was not noticed10:49
mborzeckipedronis: ok, let me add a note that we drop the type chane10:50
ackkzyga, that's not $SNAP, it's $SNAP_DATA10:50
ijohnsonmvo did you see my comment about cloud-init spread test on 8278 ?10:50
ackkzyga, oh wait, I'm dumb10:50
zygaackk: snap/maas/current/usr/lib/postgresql/10/bin seems like $SNAP10:50
ackkzyga, I saw the "current" there and thought it was SNAP_DATA, sorry :)10:50
pedronisijohnson: I think we need core 20 specific cloud-init tests, I suppose that's mvo plan10:50
zyga:D10:50
zygamvo: do you know what is missing in our spread setup?10:51
pedronisijohnson: given the behavior will be tied to grade etc10:51
ijohnsonok, sure I would still like to see a TODO:UC20: somewhere about that so that we remember to write that :-)10:51
mvozyga: not sure yet, just checking one idea10:51
pedronisijohnson: that's ok, my point is that the TODO should simply say re-enable this for UC2010:51
pedronisheh10:51
pedronisshould not simply say10:52
ijohnson:-)10:52
mvozyga: in any case, the nightly suite did notice the issue we just did not pay attention, I think we need alerting here10:52
zyga:/10:52
mvoijohnson: yeah, sorry, need to reply, but I think we need special tests, uc20 behaves differently from the other oses we have10:52
ijohnsonyes that's fine10:52
* ijohnson just likes todos so we don't forget10:53
pedronisI like todos too10:53
pedronisI like to do todos sometimes too10:53
pedronisor even done todos too10:53
pedronis:)10:53
ijohnsonhaha nah I'm not much for that, I just like them to add up10:53
ijohnson:-)10:53
ijohnsonthere's a benjamin franklin quote something like "I love deadlines, I love the whooshing sound they make as they go by"10:54
zygalol pedronis :D10:54
zygawe need to do do more todos10:54
mupPR snapd#8291 opened: packaging,tests: ensure debian-sid builds without vendor/ <Created by mvo5> <https://github.com/snapcore/snapd/pull/8291>11:52
zygamvo: lol :)11:53
zygasorry :)11:53
zygamvo: approved11:54
mvozyga: it's all very sad11:55
* zyga elbow-hugs mvo 11:56
zyganot _all_ very sad :)11:56
mupPR snapd#8292 opened: travis.yml: run unit tests on master as well <Created by mvo5> <https://github.com/snapcore/snapd/pull/8292>11:56
zygamvo: small review for 829212:02
zygamvo: I also wonder if this bug is related https://bugs.launchpad.net/snapd/+bug/186775512:03
mupBug #1867755: snapd fails to build in focal, unit test clientSuite.TestClientFindFromPathErrIsWrapped fails <snapd:New> <https://launchpad.net/bugs/1867755>12:03
mborzeckikinda meh that dh_auto_build is so magical you need to remove parts of the source tree12:04
zygamborzecki: when it works it is useful12:04
zygano packaging is good12:04
mupPR snapd#8293 opened: packaging: add README.source for debian <Created by mvo5> <https://github.com/snapcore/snapd/pull/8293>12:11
zygamborzecki: ha12:14
zygamborzecki: check this out please12:14
zygahttps://www.irccloud.com/pastebin/5hCE1Blq/12:14
zygaspecifically line 412:14
mvozyga: 1867755 is fun, it seems to be a race12:15
zygadoes it seem to you that dbus activation is somehow not working?12:15
mvozyga: which is so strange - I saw it sometimes in a PPA build (rarely) and even once in travis I think12:15
zygaheh12:16
zygamvo: use teasaiding they12:16
cmatsuokamborzecki: for some reason I can't answer your review comment on #8159 but you're right, we're mutating external data that shouldn't be touched there12:17
mupPR #8159: snap-bootstrap: remove created partitions on reinstall <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8159>12:17
mvozyga: can't parse that12:17
zygamvo: use threads they said | scramble12:17
mborzeckicmatsuoka: left a note there, after discussing with pedronis it probably makes most sense to not update the type at all and leave what's defined in the gadget12:17
mvozyga: hahahahaa12:17
cmatsuokamborzecki: ok, will do it that way12:18
mborzeckizyga: hmm session systemd has no love for the session tool?12:19
zygamborzecki: it fails on 18.0412:19
zygabut not in others somehow12:19
zygaI collected some more data, let me go through it12:19
pedronisijohnson: as discussed apart the test renames #8208 seems good to go, it needs 2nd review though12:20
mupPR #8208: boot_test: add many boot robustness tests for UC20 kernel MarkBootSuccessul and SetNextBoot <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8208>12:20
* cmatsuoka hates when you notice something strange in your code, check the branch to see if it's right (it is) and then realize you're on the wrong host12:27
pedronisijohnson: I reviewed #8286, thanks for it12:35
mupPR #8286: tests: cleanup various uc20 boot tests from previous PR <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8286>12:35
mupPR snapd#8285 closed: cmd/snap-update-ns: ignore EROFS from rmdir/unlink <Bug> <Created by zyga> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/8285>12:36
mborzeckiah, damn, should have squash merged12:36
pedronis#8292 (small) needs a 2nd review12:42
mupPR #8292: travis.yml: run unit tests on master as well <Created by mvo5> <https://github.com/snapcore/snapd/pull/8292>12:42
zygamaking progress :)12:45
zygabrb, I'll make coffee and check up on kids12:46
cmatsuokaijohnson: did you change anything related to grubenv yesterday? cachio reports boot errors in the ci tests12:57
cmatsuokaijohnson:  qemu-system-x86_64[62371]: error: file `/EFI/ubuntu/grubenv' not found.12:58
pedronislast things merged 2 days ago, was the start of uboot support13:18
pedronisshouldn't have broken amd6413:18
pedronisdon't know if something was done to the gadget or somewhere else13:19
zygaha, ok I think I got it13:19
zygamborzecki: it works :)13:19
zyganow the only thing I need is to inject one more exec13:20
zygaas the pid I get is not the pid of the app13:20
zygabut of the intermediate bash13:20
pedroniszyga: not urgent, but if you could look at my changes to #8242 it could be otherwise be landable13:29
mupPR #8242: many: improve environment handling, fixing duplicate entries <Bug> <Needs Samuele review> <Squash-merge> <Created by zyga> <https://github.com/snapcore/snapd/pull/8242>13:29
zygaack13:30
pedronisPawel reviewed it as well13:30
zygain a few minutes13:30
zygaI had a look originally but I need to read it again now as it paged out of my memory13:30
zygaoh, right. 2.44 is branched so we can land 2.45 things again13:31
ijohnsoncmatsuoka: that sounds like the same issue you and cachio had at the sprint no?13:44
sil2100zyga, mvo, xnox: as part of my SRU verification for ubuntu-image, I just built an amd64 uc20 image, but I can't get a working system - I just get the grub menu, and whatever I select I get some errors, is that expected?13:45
sil2100I just want to know if it's ubuntu-image broken or something else13:45
zygasil2100: I don't know13:45
cmatsuokacachio: could you check for "bootloader files not found" messages in the system journal?13:46
cmatsuokacachio: of see if the bootloader is actually there13:46
cmatsuokaijohnson: that's a possibility, I asked sergio to confirm if that's the case13:48
cmatsuokacachio: s/of/or/13:48
cachiocmatsuoka, what I see now is that the system is restarting in a loop13:49
ijohnsonsil2100: what are the menu items in the grub you see13:49
cachiothe last thing I see is Mar 18 13:49:05 mar181302-158658 qemu-system-x86_64[61560]: [   16.102470] systemd[1]: Started Create Static Device Nodes in /dev.13:50
cachiothen allways restarts in install mode13:50
cmatsuokacachio: that didn't happen with the bootloader error in the sprint, right?13:50
cachiocmatsuoka, I thinks it is not the same but not totally sure13:51
cachiocmatsuoka, sometimes I even see Mar 18 13:51:27 mar181302-158658 qemu-system-x86_64[61560]: Press enter to configure.13:51
cachioand then reboots again automatically13:51
sil2100ijohnson: 'Recover using /systems/*', 'Install using /systems/*' and 'System setup'13:52
cachiocmatsuoka, this is the last log v13:53
cachiohttps://paste.ubuntu.com/p/6bNqps9Np2/13:53
cmatsuokacachio: that's very strange, what changed from the latest build that worked? is there a new kernel snap, or new ubuntu-image?13:53
sil2100There has not been a new ubuntu-image in stable since a while, 1.9 is the latest13:55
cmatsuokasil2100: thanks for the information13:56
cmatsuokacachio: so let's see what the differences are, something must be different13:57
sil2100cmatsuoka, cachio: when did you start noticing the problems? And are you using ubuntu-image from the snap or deb?13:58
sil2100That being said, I don't think we had any uc20 related changes to 1.9 even13:59
cachiosil2100, snap14:02
cachiocmatsuoka, I think the problem was already there14:03
cachiothe system is restarting and restarting until at some point it does not restart14:03
cachiothen it works14:03
cmatsuokacachio: coming to SU?14:04
sil2100cachio: while we're at it, what's up with core18 the snap? I see it's not marked as ready for beta all the time, is something failing for the pi3?14:04
cachioit started failing now because I reduced the timeouts because I am tryting to speed up a bit the tests because it is taking like 40 minutes each test14:04
sil2100cwayne, plars: ^14:04
cachiosil2100, let me check14:05
cachiosil2100, this is the problem https://trello-attachments.s3.amazonaws.com/5da8bc830df86851446e9d4e/5e688b02c26ce805a8a979ef/8bb39aabc282ceb61f07f599cd7d72b6/core18_20200311_(1705)_pi3_arm32.log14:06
cachiothe system fails to flash the image14:06
sil2100cachio: oh, so not really an issue with the snap itself - should we poke plars around it and see if he can help?14:07
cachioI think the issue is with the device in the lab14:08
cachiosil2100, not related to the image14:08
cachioI trigegred it again14:09
cachiolets see14:09
ijohnsonsil2100: what error do you see when you try to use the `install using ...` menu item?14:11
xnoxsil2100:  did you boot in UEFI mode, with secureboot, and snakeoil UEFI VARS?14:14
xnoxusing OVMF firmware?14:14
xnoxsil2100:  if you see "*" it means you booted in bios mode14:14
* xnox should push a grub change to print something sensible14:14
cwaynecachio: that doesnt make any sense, we use that image on that pi a lot14:18
plarscachio: cwayne: I can't even reach that device at the moment, so I have no idea, but it's dead now. My best guess is that something was wrong with the sd card on it, but I can't login to check14:21
cachioplars, ok14:25
cachiocwayne, I just tried to run smoke tests for core18 and that happend14:26
Eighth_Doctormborzecki, zyga: so this is now a thing: https://docs.fedoraproject.org/en-US/ci/how-to-add-dist-git-test/14:28
Eighth_Doctorperhaps one day we could add some tests for snapd for fedora infra to run (similar to autopkgtests for debian)14:28
mborzeckiEighth_Doctor: oh nice, we had some ideads about testing packages before pushing to distros14:29
zyganice, thank you14:29
Eighth_Doctoryou can kind of thank pitti here, he pushed for this to become a thing, and learned from the mistakes in making autopkgtests14:30
zygayay, so session tool can now track PIDs of the started processes14:31
zygaseparately from the session manager14:32
zygaso that's good14:32
zygaI added that to the cgroup-tracking test14:32
zygalet's see what we get14:32
zygaEighth_Doctor: can you ask him to push it back to Debian ;D14:43
Eighth_Doctorhah, you know as well as I do that debian doesn't do change :P14:45
zygaEighth_Doctor: it does, in debian stable ;)14:49
zyga\o/14:58
zygajdstrand: I reproduced the issue reliably in spread now14:58
zygawoot14:58
zygaMar 18 14:56:41 mar181449-606426 systemd[20017]: snap.8661f9ab-6dbb-4fd5-bb75-9bb871f5dddc.test-snapd-sh.sh.scope: Failed to add PIDs to scope's control group: Permission denied14:58
zygaI can now push some simple fixes for session-tool14:59
zygaand start working on untangling this particular error14:59
zygalet me just quickly verify that it's not affecting other releaes14:59
zyga*releases14:59
cachiopedronis, mvo this is the log for the retries on uc20 https://paste.ubuntu.com/p/kjhYcqcmrw/15:00
pedronisit is saying:  error: file `/EFI/ubuntu/grubenv' not found.15:01
pedronisquite a bit15:01
ijohnsoncmatsuoka: I had a look at the log for PR 8159, and if you merge master that snapd-failover issue should go away15:01
mupPR #8159: snap-bootstrap: remove created partitions on reinstall <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8159>15:01
cmatsuokaijohnson: thanks, will do15:02
zyga16.04 passes15:08
zygaI'm pretty sure 18.04 has a bug15:08
zygathat's not present in 20.04 or 16.0415:08
zygaah, sorry 20.04 passed, 16.04 still running15:08
pedroniscachio: mvo: we don't even get to systemd in the first couple of those reboots15:08
pedronisafaict15:09
mvopedronis: hm, yeah, this looks like initramfs snafu15:09
pedronisor something uefi related15:10
pedronismvo: there first time we get to EFI Variables Facility v0.08 2004-May-1715:11
pedronisand then there's a reboot there15:11
cachiopedronis, could be related to the gadget snap version?15:12
cachioI am retrying with the edge version now, this run was using the stable version15:12
mvocachio: yeah, that sounds like it could be15:13
cachiomvo, I'll have results soon15:16
mvocool15:16
mupPR snapd#8294 opened: seed: make Brand() part of the Seed interface <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8294>15:42
* cachio lunch15:49
cachiomvo, pedronis using gadget from edge I see the same15:57
cachiothis is the full log https://paste.ubuntu.com/p/bsmwC5Q52b/15:58
mvoxnox: does https://paste.ubuntu.com/p/bsmwC5Q52b/ ring any bells?15:58
mvoxnox: the funny part is that after some reboots this goes away, is there maybe something in initramfs that looks for it, if it's missing the unit fails and reboots and sometimes we run stuff with the right timing and things work, would the initramfs reboot if a unit fails?16:03
xnoxin a meeting16:04
mvoxnox: no worries, sry16:04
xnoxmvo:  hey16:34
xnoxmvo:  so that message about lack of grubenv is from grub. it failing to read and load grubenv, seems to indicate that this is not a UC20 boot? because UC20 gadget relies on grubenv to be present and valid.....16:35
mvoxnox: aha, that's interessting (cc cachio) - what's strange is that apprerntly for cachio it works after a bunch of reboots in his VM setup16:36
xnoxmvo:  i am concerned on how you are starting the VM and whether or not you passed which disk is the expected correct bootdevice16:37
xnoxmvo:  because it seems like your tpm state will be corrupted.16:37
cachioxnox, https://paste.ubuntu.com/p/JTGNRh7djq/16:38
cachioxnox, should I change anything in the command?16:39
xnoxmvo:  the boot does not seemed to be complete. as the boot is in install mode, and the next one is as well.....16:39
xnoxdo the tests reboot twice correctly? and again, without grubenv being written correctly, it will not have pointer to boot into run mode.16:39
xnoxcachio:  you have bootindex=1 set correctly there, so boot device is there.16:40
cachioxnox, do you want to inspect the image?16:41
xnoxmvo:  cachio: i'd like to see the VM image ubuntu-core-new.img contents after it was created, but before it is booted. And the file listing of all files on all partitions, and contents of grubenv file.16:41
xnox=)16:41
xnoxcachio:  we think alike ;-)16:41
mborzecki#8249 is super simple and needs a 2nd review ;)16:41
mupPR #8249: interfaces: make gpio robust against not-existing gpios in /sys <Reviewed> <Security-High> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8249>16:41
mborzeckiheh16:41
mborzecki#8294 this one16:42
mupPR #8294: seed: make Brand() part of the Seed interface <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8294>16:42
pedronismborzecki: could you review #8208 when you have time (is not simple)16:45
mupPR #8208: boot_test: add many boot robustness tests for UC20 kernel MarkBootSuccessul and SetNextBoot <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8208>16:45
xnoxcachio:  mvo: the image looks quite small, i don't know if we have enough disk space to complete the install16:47
xnoxit is 5.4GB big, and I usually locally make 8G/10G big images.16:48
cachioxnox, ahh, I can try with 10GB16:49
cachiodoes it makes sense?16:49
xnoxmvo:  cachio: the failing to load grubenv is a red herring, and i should fix our grub.cfg to not print that pointlessly16:52
xnoxcachio:  please try with a 10GB image, yes.16:52
pstolowskipedronis: #8270 updated with the extra fields, ready for review16:53
mupPR #8270: store: support for search API v2 <Needs Samuele review> <Squash-merge> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8270>16:53
cachioxnox, sure, running16:53
pedronis#8286 needs a 2nd review, it's simple (mostly adding tests and improving code behavior a bit)17:01
mupPR #8286: tests: cleanup various uc20 boot tests from previous PR <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8286>17:02
xnoxcachio:  i'm afk / busy until end of day, sorry. Can chat tomorrow, if you will have uc20 logs from a bigger image.17:03
cachioxnox, sure, I'll try again and send you the logs in case it fails again17:03
cachiothanks for your help17:03
zygajdstrand: hey17:11
zygajdstrand: I've debugged the issue17:11
mborzeckipedronis: ack17:11
zygajdstrand: dbus-user-session17:11
zygajdstrand: the new app tracking feature depends on it and we just didn't notice because it's pre installed since eoan (on server)17:11
zygajdstrand: I've expanded the test to completely cover running as user17:12
zygaand with this package it all passes17:12
zygaand without it it behaves just as your system did17:12
zygajdstrand: a bionic desktop has it installed by default17:13
zygajdstrand: but a bionic server does not17:13
zygajdstrand: can you, if you remember, confirm where you were testing my branch at the time?17:13
jdstrandzyga: hi! (sorry, was doing 360s)17:13
zygajdstrand: no worries :)17:14
zygaI'm so glad I made some progress on this branch17:14
jdstrandzyga: iirc, it was a bionic desktop amd64 vm, logging in over ssh17:15
zygajdstrand: do you have dbus-user-session installed in that VM?17:15
jdstrandI'm checking now17:15
jdstrandzyga: it is installed. when I login I see this in the environ: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus17:18
zygahmmm17:18
zygaookay17:18
zygathat's interesting, and it's not installed recently?17:18
zygaI'll double check in any way17:18
zygathank you17:18
jdstrandzyga: let me look at the PR again. I think I said what I was using17:19
zygaright now the tests do pass in all xenial+ systems17:19
zygaas root and as user17:19
zygabut not as user logged in via ssh17:19
zyga(I didn't write that test since it's all kind of remote)17:19
zygabut maybe it's a relevant factor17:19
jdstrand"Unfortunately, on (at least) bionic, when logged in via ssh or directly via the console"17:22
zygaok, I'll get to it17:22
zygaI suspect something else is a factor then17:22
zygabut I'm closer, I think17:22
jdstrandzyga: sounds like you are getting there! :)17:22
jdstrandzyga: it's the feature that won't stop giving :)17:23
zygaoh yes17:23
jdstrandzyga: and for completeness, logging into a vt, DBUS_SESSION_BUS_ADDRESS was set on the console as well17:24
jdstrandI suspect that is logind for both17:24
zygaI took a snapshot of my 18.04 desktop vm and I'll dig in17:24
zygayes, that's pam-systemd17:25
zygathough that's really17:25
zygadbus-session-bus started in the session17:25
zygajdstrand: is your desktop VM logged in as a desktop user?17:26
jdstrandzyga: the vm came up and is sitting at the gdm prompt. I did not login. I then ssh as the normal user (ie, the one I would use at the gdm login)17:27
zygaok17:28
zygathanks17:28
zygaI have the same setup now17:28
zygaand when you are logged in via ssh17:28
mupPR snapd#8295 opened: osutil: do not leave processes behind after the test run <Simple 😃> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8295>17:28
zygaloginctl shows only two sessions?17:28
zygaone for gdm17:28
jdstrandzyga: I can say that /run/user/1000/bus exists and it is /lib/systemd/systemd --user that setup that socket17:28
zygaand one for your ssh17:28
jdstrandzyga: however, there is no dbus-daemon running under this user. just dbys-daemon --system (root) and dbus-daemon --session (gdm)17:29
zygahmmm17:29
zygaI see17:29
zygahttps://www.irccloud.com/pastebin/H3nx0PgY/17:29
jdstrand(well, there is also the accessibility bus for gdm)17:29
zygazyga       1689  0.0  0.1  49792  3840 ?        Ss   18:26   0:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only17:29
jdstrandzyga: loginctl shows gdm and the user I logged in as17:30
jdstrandzyga: you have that running as 'zyga' without logging into gdm?17:31
zygaI ssh'd in17:31
zygayes17:31
zygabut I have dbus-user-session installed17:31
zygalet me log out and double check PIDs recyce17:31
zygaoh wait17:31
zygait's gone now17:31
jdstrandI have dbus-user-session installed, logged in via ssh and do *not* have the above in ps for my user17:32
zygamaybe it's just activated17:32
zygaha17:32
zygawait17:32
zygait didn't17:32
zygazyga@bionic-desktop:~$ systemd-run --user --scope ls17:32
zygaJob for run-re0d33bd5a3334fc39992748ea3053b88.scope failed.17:32
zygaSee "systemctl status run-re0d33bd5a3334fc39992748ea3053b88.scope" and "journalctl -xe" for details.17:32
zygathat's exactly what you saw17:32
zygawhat the...17:32
zygaI think my auto-login spawned dbus17:32
zygaand it was around17:32
zygaI logged out, killed that session17:32
jdstrandah, yes, that would do it17:32
zygalogged out from ssh17:32
zygalogged back in17:32
zygaand no dbus17:32
zyga*progress*17:32
jdstrand\o/17:33
zygathank you, I won't bother you anymore until I crack this17:33
jdstrandzyga: it feels so close! :)17:33
zygamar 18 18:32:12 bionic-desktop systemd[2002]: run-re0d33bd5a3334fc39992748ea3053b88.scope: Failed to add PIDs to scope's control group: Permission denied17:35
zygamvo: small review on 829517:43
zygamvo: will kill-sleeper work without job control in tty?17:44
mvozyga: I don't know, I tested it locally and it was ok17:46
zygalocally it would have your pty17:46
mvozyga: yeah17:46
zygalet's see what happens17:46
zygamvo: I think exec sleep is better17:46
zygathen you can kill it17:46
mvozyga: but I need the pid?17:47
zygaexec :D17:47
zygah17:47
zygaah17:47
zygasorry17:47
zyga:/17:47
zygamissed that17:47
mvozyga: :)17:47
mvozyga: no worries17:47
zygayou could echo $$ > /tmp/pid17:47
zygaand kill that17:47
zygabut uck17:47
mvozyga: happy about better ideas, feel free to push to the PR, it was the best I could think of17:47
zygayeah, that's fine17:47
zygalet's see if it passes17:47
zygaI'm debugging systemd behavior now17:47
zygabut happy to look next17:48
mvozyga: no worries, if it's good enough that would be nice if not we can iterate17:48
zygaI think I found the systemd bug17:54
zygamkdir("/sys/fs/cgroup/pids/user.slice/user-1000.slice/user@1000.service/run-rbabf8a8b75af4ad68cf81da60da72a47.scope", 0755) = -1 EACCES (Permission denied)17:54
zygahttps://bugzilla.redhat.com/show_bug.cgi?id=141307517:54
zygaseems related17:54
mupPR snapd#8294 closed: seed: make Brand() part of the Seed interface <UC20> <Created by bboozzoo> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/8294>18:52
zygaI give up for today18:52
zygaI'm attached to systemd --user that exhibits the error18:52
zygaI have debug symbols and all that18:52
zygaI can reproduce the error at will18:52
zyganeed to jump into this with fresh head tomorrow18:53
zygajdstrand: ^ FYI18:53
zygaI also have the same setup with newer systemd on focal where it doesn't fail18:53
zygathat's all for today folks18:53
zygao/18:53
cmatsuokacachio: booting the test image it worked, but auto-import from /dev/sda failed (with exit code 127)18:55
cachiocmatsuoka, perhaps mvo knows about which other thing to check18:57
cmatsuokacachio: auto-import.assert is indeed in /dev/sda18:57
ijohnsoncmatsuoka: any more output from auto-import ?18:58
cmatsuokaijohnson: not much, let me retrieve the actual log entries18:59
cmatsuokaijohnson: https://pastebin.ubuntu.com/p/8B3JyXMsbx/19:01
zygacmatsuoka: 127 is command-not-found IIRC19:02
cmatsuokaijohnson: /dev/sda is where the assertion is, I mounted it and it's there19:02
cmatsuokaoh19:02
zygaso /bin/snap whatever is not tehre19:02
cmatsuokathat's embarrassing19:02
zyga;D19:02
ijohnsonwell then19:02
* zyga goes away19:02
cmatsuokacachio: I think it's explained then, let me check the initramfs to see if the executable is there...19:03
cmatsuokaI mean19:04
cmatsuokait's on the real system, right? so no, it's not tehre19:04
cmatsuokathere19:04
mvocmatsuoka: this indicates that the snap command is not there, this usually happen on the very first boot when snapd is not seeded yet19:06
cachiocmatsuoka, do you see any error during the seeding?19:07
cmatsuokamvo: ah yes, that makes sense19:07
cmatsuokacachio: seeding seems good to me19:07
cmatsuokacachio: no more auto-import messages later19:09
cachioif you restart it does it work?19:09
zygadoes the auto-import service depend on seeded system?19:09
zygait probably shold19:10
zyga*should19:10
cmatsuokazyga: it seems that it doesn't but it should, from what we see here19:10
pedronisI think mvo has thoughts on it already19:11
ijohnsonthis is the same bug that came up during the sprint19:11
pedronisyes19:11
pedronisit's a known bug atm19:12
mvocmatsuoka, pedronis yeah, I think we either need to just echo the device paths and catchup or do a udev scan of the block devices in snap.autoimport.service. it's mostly jfdi but I didn't manage to find the time for it yet19:12
pedronisit's not an immediate blocker, the cost is rebooting again19:13
cachiopedronis, in uc20 is taking so long to boot19:13
cmatsuokaok then, now I see how this ties to the sprint conversation19:13
cachiowe were researching why19:14
pedroniscachio: well, I new run reboot shouldn't take that longer19:14
pedroniscachio: isn't the issue the multiple reboots?19:14
pedronisor is this something else19:14
pedroniscachio: I mean, did we fix the issue you showed today, were we get a bunch of reboots19:14
pedronisthat don't even reach systemd19:14
cachioit is something else, it was caused becuase the zise of the image was 5g and we needed 10g19:15
cmatsuokacachio: anyway, in all my tests it booted correctly to run mode without any boot loops19:15
pedronisinteresting, that seems big, do we know what's the space needed for?19:15
cachioI don't yet19:16
pedronisok19:16
cmatsuokapedronis: that's curious because the partition grow code is not there yet19:16
cmatsuokaso the partition sizes should stay the same19:17
pedronisdo we have a weird bug in install code that makes a partition of the wrong sizes?19:18
* zyga EODs19:19
cmatsuokacachio: could you generate a 5G image for me?19:19
cmatsuokacachio: so I can check if there's something funky in partitioning19:19
cachiosure19:20
cachiofirst, could you try restart that vm19:21
cachioin my case when I restart the vm it goes in a rrboot loop19:22
cachioand I see error: file `/EFI/ubuntu/grubenv' not found.19:22
pedronisdo we know if it's in the image?19:25
pedronisit should be (at least we put one there end of Feb)19:26
cachioI am generating a new image to check again19:27
pedronisijohnson: cachio: so I made an image and there's no grubenv, but that's expected I think, there will be one now with the new code19:41
ijohnsonpedronis: there won't be a grubenv on the root ubuntu-seed grubenv19:42
pedronisyes19:42
pedronisnot until we set mode to run19:43
ijohnsonerr sorry there won't be any variables set, I guess I don't know for certain whether there would be an empty grubenv or not19:43
pedronisthere isn't one19:43
ijohnsonyes when we run makeBootable20RunMode then we set grubenv on ubuntu-seed19:43
pedronisI just made an image and mouunted19:43
ijohnsoniirc19:43
pedronisit19:43
pedronishttps://paste.ubuntu.com/p/w3NbfYdt5V/19:46
pedroniscmatsuoka: is #8159 ready for a re-review ?20:20
mupPR #8159: snap-bootstrap: remove created partitions on reinstall <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8159>20:20
cmatsuokapedronis: yes, I also fixed issues raised my maciek (the struct pointer issue and better attribute parsing)20:27
pedroniscmatsuoka: ok, thanks, I'll look at it in my morning20:28
cmatsuokapedronis: I'm currently skipping partitions that have a type that's not in our list. the alternative would be to fail and abort installation20:29
* cachio afk21:03
mupPR snapd#8296 opened: httputil/client_test.go: add two TLS version tests <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8296>21:09
pedronisijohnson: seems here failover failed again, but it seems that PR got the latest code for that: https://api.travis-ci.org/v3/job/664035070/log.txt21:10
ijohnsonpedronis: which pr21:11
pedronisijohnson: your latest21:11
ijohnson8287 ?21:11
pedronis#828621:11
mupPR #8286: tests: cleanup various uc20 boot tests from previous PR <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8286>21:11
pedronisalso the ubuntu-image test change breaks it21:13
pedronissu => session-tool21:13
pedroniss/ubuntu-image/prepare-image/21:13
zygahmm?21:13
zygasession-tool broke something?21:13
pedronisI'm also not sure why using session-tool there21:13
pedronisit seems overkill, but I may be missing something21:14
zygawhere?21:14
zygaI just came to check on something and noticed21:14
pedroniszyga: https://github.com/snapcore/snapd/pull/8286/files21:14
mupPR #8286: tests: cleanup various uc20 boot tests from previous PR <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8286>21:14
pedroniszyga: ijohnson replaced a su invocation with session-tool, do the test is just calling prepare-image, and things break21:14
zygado you know why it is invoked as user in the first place?21:14
ijohnsonmvo wrote the original test fwiw21:15
pedronisto check that it works21:15
pedronisas a normal user21:15
zygaaha21:15
zygaand how does it break/21:15
pedronisit's not finding an assertion, the breakage is strange21:15
pedronisanyway not sure it needs to use session-tool21:15
pedroniszyga: there's spread logs there21:16
ijohnsonI mean isn't this why we have session-tool though ...21:16
zygaok21:16
ijohnsonzyga: https://pastebin.ubuntu.com/p/sJzvXy3dWw/21:16
pedronisijohnson: do break working tests?21:16
zygathanks!21:16
pedronisto break working tests?21:16
zygapedronis: true always works ;)21:16
ijohnsonno I mean to have a tool that "does all the right things"21:16
ijohnsonsu doesn't do all the right things21:16
pedroniswhat is right or wrong is very contextually dependent21:17
zygahmm21:17
zygais /home/test/tmp//model.assertion sensible21:17
zygaas in // that feels like something not expanded21:17
pedronisit's just a var ending in /21:17
zygaah, $ROOT has / at the end21:17
pedronisanyway works with just su21:17
ijohnsonit's easy enough to back out that change but it's just frustrating that we have these problems :-/21:18
zygawhat places model.assertion there?21:18
ijohnsonalso the snapd-failover failure is very depressing tbh21:19
pedronisijohnson: yes, that one is21:19
pedronisdefinitely21:19
zygaijohnson: I made some improvements to session-tool, fixed some issues with quoting21:19
zygabut I don't suspect it's a factor in this case21:19
pedronisijohnson: the other one I would just move back to su and be happy for a while21:19
zygaoh wait21:19
zygahmm21:20
pedronisI'm quite all the other tests like this one use su anyway21:20
pedronisso we would have to change all or none21:20
zygasession-tool expects stuff normally, not as one big string21:20
zygamaybe my quoting fix would actually help21:20
pedronisand it doesn't seem a good use of time at this moment21:20
zygaijohnson: yeah, leave it to me21:20
zygaI plan to do a pass21:21
ijohnsonawesome, thanks zyga21:21
ijohnsonI wonder if we should just stop snapd.socket then start it again rather than a restart21:21
ijohnsonI just really really wish we understood why that keeps happening anyways21:21
pedronisI reverted to su for now21:23
pedroniswe have 29 places using it still21:24
zygaI pushed some fixes to session tool21:25
zygahttps://github.com/snapcore/snapd/pull/829721:25
mupPR #8297: tests: session-tool improvements <Created by zyga> <https://github.com/snapcore/snapd/pull/8297>21:25
pedronisthat's good, I don't think converting su to session-tool is somethign anybody should work on unless there is a good reason for specific tests at this point in time21:25
pedronisthough21:26
pedronistoo many other things21:26
mupPR snapd#8297 opened: tests: session-tool improvements <Created by zyga> <https://github.com/snapcore/snapd/pull/8297>21:26
zygapedronis: I think it's that they are definitely not really testing the feature21:26
pedroniswhich feature?21:26
zygapedronis: su is not representative of a user running the code21:26
zygapedronis: anything21:26
zygapedronis: it's really 90% root21:27
zygapedronis: 10% user21:27
pedronismaybe21:27
pedronisit depends21:27
zygapedronis: if the point of the test is to check if it runs for normal people21:27
zygasu is not it21:27
zygait does depend21:27
zygabut we should generally not use su, because it depends is complicated21:27
pedronisis still not a good use of time all things considered21:27
pedronisat this point in time21:27
zygasure, at some point it will though21:27
pedronisyea21:27
pedronislet's try to get there21:28
zygaindeed :)21:28
* zyga wrote a comment on https://github.com/snapcore/snapd/pull/7825#issuecomment-600861003 that is sad but relevant21:28
mupPR #7825: many: use transient scope for tracking apps and hooks <Security-High> <Created by zyga> <https://github.com/snapcore/snapd/pull/7825>21:28
zyga(not to this discussion though)21:28
zygaand I'm back to being away21:29
mvozyga: we prefer "win the lottery" instead of "hit by bus" - just saying :)21:40
zygamvo: if I win a lottery I will perpetually send patches for things I like and not worry about paycheck ;)21:41
zygamvo: *then* you get the bus :)21:41
zygaI've installed buster now21:42
zygaand installing nvidia driver21:42
ijohnsonhaha21:42
mvoha!21:44
* mvo really calls it a day21:44
zygamvo: can you dput to buster-backports?21:44
* zyga wonder if he saw that21:44
zyganvidia + debian = :(21:54
zyga+ snaps21:54
ppdzyga: Not so nice to hear. Thanks for investigating the Nvidia problem though22:41
cmatsuokacachio: #8260 tests failed23:23
mupPR #8260: tests: enable nested on core20 and test current branch <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8260>23:23
cachiocmatsuoka, I'll push a small change now to retrigger this23:36
cachioneed to wait until tests pass here23:36
cmatsuokait seems that tests are especially slow today23:40

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!