| zyga | good morning | 06:08 |
|---|---|---|
| zyga | battery running low, brb | 06:37 |
| zyga | re | 06:39 |
| zyga | mvo: quiet day, eh? | 06:42 |
| mvo | zyga: good morning | 06:45 |
| mvo | zyga: yes, pretty quiet | 06:45 |
| mvo | zyga: did you enjoy your long weekend? | 06:46 |
| zyga | yes, though lucy had fever yesterday and we were worried | 06:46 |
| zyga | it's all meant to be quite different, my wife returned to work today and (before I got sick) I was supposed to take two weeks off | 06:47 |
| zyga | but reality struck and here we are :) | 06:47 |
| mvo | zyga: yeah, reality is annoying sometimes - but you can't argue with it - it's always right :/ | 06:48 |
| zyga | :D | 06:48 |
| zyga | yep | 06:48 |
| pstolowski | morning | 07:01 |
| mvo | good morning pstolowski | 07:04 |
| mvo | pstolowski: hope you had a nice long weekend | 07:04 |
| pstolowski | mvo: hey! yes, was very nice | 07:10 |
| * zyga had breakfast | 07:25 | |
| zyga | Lucy still sleeping :) | 07:25 |
| zyga | I'm working on the udev integration branch | 07:25 |
| zyga | it's open for so long and should not take long to finish | 07:26 |
| mvo | zyga: nice! | 07:27 |
| zyga | hmm, some store EOFs | 07:27 |
| zyga | - Download snap "core18" (1885) from channel "stable" (unexpected EOF) | 07:28 |
| zyga | is unexpected EOF something we retry on? | 07:29 |
| mvo | pstolowski: unless you disagree I will (squash) merge 9152 | 07:29 |
| mvo | zyga: probably, might be worthwhile to check the logs | 07:29 |
| zyga | mvo: here it was during "snap install shellcheck" before we ran spread | 07:29 |
| zyga | so no logs | 07:29 |
| pstolowski | mvo: fantastic! yes, go for it, thanks | 07:30 |
| mup | PR snapd#9152 closed: cmd/snapd-generator: generate drop-in to use fuse in container <Bug> <Preseeding 🍞> <Squash-merge> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9152> | 07:34 |
| zyga | store is really unhappy today | 07:49 |
| mup | PR snapd#9102 closed: corecfg: add "system.timezone" setting to the system settings <Run nested> <Squash-merge> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9102> | 07:49 |
| pstolowski | jamesh: hi! minor comment to your snapctl is-connected PR | 08:25 |
| jamesh | pstolowski: looking | 08:25 |
| jamesh | pstolowski: would it actually be possible to call snapctl in a context where implicit plugs and slots existed? | 08:31 |
| pstolowski | jamesh: we have a 'hack' for implicit slots (AddImplicitSlots helper) that we explicitly call over SnapInfo of system snap in various places. if core had interface hooks for slots than I image it would be needed. but i'd leave it as is for now because it's not going to be needed any time soon | 08:34 |
| pstolowski | s/image/imagine/ | 08:35 |
| jamesh | I suppose core has a configure hook where it could make a difference if it actually checked | 08:36 |
| pstolowski | jamesh: it has but the hook is "hijacked" and handled internally | 08:40 |
| * zyga moves to the office, lucy is now awake and handled by grandparents | 08:43 | |
| pstolowski | mvo: there is one question from Samuele pending your input on https://github.com/snapcore/snapd/pull/9084, can you take a look? | 08:53 |
| mup | PR #9084: o/snapstate: check disk space before creating automatic snapshot on remove (3/N) <Disk space awareness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9084> | 08:53 |
| mvo | pstolowski: sure, looking | 08:54 |
| mvo | pstolowski: replied | 08:58 |
| pstolowski | ty | 08:58 |
| zyga | mvo: uh | 09:47 |
| zyga | we have a very silly bug in master | 09:47 |
| zyga | mvo: patch coming right up | 09:48 |
| mvo | zyga: oh, ok | 09:48 |
| mup | PR snapd#9171 opened: [RFC] "virtual" configuration for timezone <Needs Samuele review> <⛔ Blocked> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9171> | 09:50 |
| mup | PR snapd#9172 opened: tests: update spread test for unknown plug/slot with snapctl is-connected <Simple 😃> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9172> | 09:55 |
| pstolowski | jamesh: i've updated snapctl spread test to test the new error message ^ | 09:58 |
| zyga-mbp | mvo https://github.com/snapcore/snapd/pull/9173 | 10:03 |
| mup | PR #9173: cmd: compile snap gdbserver shim correctly <Bug> <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9173> | 10:03 |
| mup | PR snapd#9173 opened: cmd: compile snap gdbserver shim correctly <Bug> <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9173> | 10:05 |
| mvo | zyga-mbp: thank you | 10:19 |
| zyga-mbp | I really wonder what tests will say | 10:19 |
| zyga-mbp | I never tried this | 10:19 |
| mvo | 9084 needs a second review if someone has some spare cycles | 10:20 |
| mup | PR snapd#9120 closed: interfaces: add kernel-crypto-api interface <Needs Samuele review> <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9120> | 10:40 |
| mup | PR snapd#9165 closed: interfaces: add kernel-crypto-api interface - 2.46 <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9165> | 10:40 |
| zyga-mbp | looking | 10:44 |
| mup | PR snapd#9167 closed: many: correctly calculate the desktop file prefix everywhere <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9167> | 10:45 |
| zyga-mbp | pstolowski https://github.com/snapcore/snapd/pull/9084#pullrequestreview-469207489 | 10:50 |
| mup | PR #9084: o/snapstate: check disk space before creating automatic snapshot on remove (3/N) <Disk space awareness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9084> | 10:50 |
| zyga-mbp | brb, I'll fetch some water | 10:50 |
| mvo | 9081 also needs a second review | 10:52 |
| mvo | jdstrand: 8301 has conflicts now, could you please have a look? | 11:00 |
| mvo | pstolowski: just fyi, I uploaded a new snapd to groovy with the lxd generator fix so that should allow groovy lxd preseed images. could you please let the relevant people know (still needs some hours before it enters groovy proper, still building of course) | 11:13 |
| pstolowski | mvo: sure, thanks | 11:18 |
| * zyga fetched lunched instead | 11:36 | |
| zyga | re | 11:36 |
| zyga | opensuse tumbleweed seems to be unhappy | 11:40 |
| zyga | error: cannot query the store for updates: got unexpected HTTP status code 503 via POST to "https://api.snapcraft.io/v2/snaps/refresh" | 12:46 |
| zyga | more store woes | 12:46 |
| zyga | pstolowski: ping | 12:49 |
| zyga | I have this in a debug shell | 12:49 |
| zyga | + lxd.lxc exec my-nesting-ubuntu -- snap set lxd waitready.timeout=240 | 12:49 |
| zyga | error: cannot perform the following tasks: | 12:49 |
| zyga | - Run configure hook of "lxd" snap (snap "lxd" has no "configure" hook) | 12:49 |
| zyga | nested snaps are all broken, not mounted | 12:50 |
| pstolowski | zyga: ok, that's weird | 12:50 |
| pstolowski | zyga: is it master? | 12:51 |
| zyga | yep | 12:51 |
| zyga | but seems random | 12:51 |
| zyga | https://paste.ubuntu.com/p/3YspQRfGvT/ | 12:51 |
| zyga | interesting failed unit | 12:51 |
| zyga | no snap mounted | 12:51 |
| zyga | I guess one idea would be to not run hooks on broken snaps | 12:52 |
| zyga | the snaps are on disk | 12:52 |
| zyga | looking at the mount units | 12:52 |
| zyga | there are no mount units?!? | 12:53 |
| pstolowski | did i broke something with systemd generator thing..? | 12:53 |
| zyga | maybe | 12:54 |
| zyga | looking | 12:54 |
| pstolowski | mvo: was #9152 green when it landed, or did you merge manually? | 12:54 |
| mup | PR #9152: cmd/snapd-generator: generate drop-in to use fuse in container <Bug> <Preseeding 🍞> <Squash-merge> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9152> | 12:54 |
| zyga | fstab has only LABEL=cloudimg-rootfs/ ext4defaults0 0 | 12:55 |
| zyga | but there's no /dev/disk/by-label | 12:55 |
| zyga | I guess that's because of a container | 12:55 |
| zyga | and also explains the failed unit | 12:55 |
| zyga | seems like a separate container bug | 12:55 |
| zyga | I don't see any evidence that snapd generator ran | 12:56 |
| zyga | well | 12:56 |
| zyga | actually | 12:56 |
| zyga | I do see snap.mount | 12:56 |
| mvo | pstolowski: I thought it was, maybe I was mistaken | 12:56 |
| zyga | and nothing else | 12:56 |
| mvo | pstolowski: do you see errors now? | 12:56 |
| zyga | what's the other file we should have created? | 12:57 |
| zyga | the file /run/systemd/container is present | 12:57 |
| zyga | and has the word "lxc" | 12:57 |
| zyga | I guess that's why nothing mounted btw | 12:57 |
| zyga | but it's funny | 12:57 |
| zyga | I guess terrible | 12:57 |
| zyga | that we INSTALLED snaps in broken state after mounting failed | 12:57 |
| zyga | mounting should have undone everything | 12:57 |
| zyga | changes and tasks from the container https://paste.ubuntu.com/p/VPH4DwbGvx/ | 12:58 |
| pstolowski | zyga: there should drop-ins in /run/systemd/generator for all the snap mount untis for /etc/systemd/system | 12:58 |
| pstolowski | zyga: are there mount units in /etc/../system ? | 12:59 |
| zyga | there are no drop ins | 12:59 |
| zyga | https://paste.ubuntu.com/p/ft8pr6qXrC/ | 12:59 |
| zyga | and there are no mount nits | 12:59 |
| zyga | but we remove them in undo | 12:59 |
| zyga | although this makes no sense because "snap list" shows all the snaps installed and broken | 13:00 |
| zyga | this is tests/main/lxd | 13:00 |
| zyga | perhaps it's using older non-repackaged core18 + snapd | 13:00 |
| zyga | hmm hmm | 13:00 |
| pstolowski | zyga: ok if there are no mount units then root cause is elsewhere, not in the generator | 13:00 |
| zyga | can you just spread 16.04 + tests/main/lxd | 13:00 |
| zyga | and see if that fails | 13:00 |
| zyga | pstolowski: well, mount units do get removed on undo | 13:01 |
| zyga | but I really don't know what to make of this | 13:01 |
| zyga | the generator only has the /snap sharing mount | 13:01 |
| zyga | but doesn't have the drop ins | 13:01 |
| zyga | ah | 13:01 |
| zyga | 1337.2.46~pre2 | 13:01 |
| zyga | well | 13:01 |
| zyga | 1337 is the repackaged | 13:01 |
| zyga | so... | 13:01 |
| zyga | hmmm | 13:01 |
| mvo | zyga, pstolowski standup | 13:01 |
| zyga | it seems broken | 13:01 |
| zyga | oh | 13:01 |
| zyga | joining | 13:01 |
| jdstrand | mvo: hi! certainly, I'd love to. thanks for the reviews/merges! :) | 13:17 |
| mvo | jdstrand: thank you! | 13:17 |
| jdstrand | mvo: fyi, the cups one should good for re-review whenever people have a chance (though I'm going to merge master real quick). I'll quickly respond to any feedback there. moving on to 'misc policy updates' then after, will go through any PR reviews that need me | 13:20 |
| zyga | jdstrand: could you try to look at https://github.com/snapcore/snapd/pull/7614 again | 13:21 |
| mup | PR #7614: cmd/snap-confine: implement snap-device-helper internally <Needs security review> <Created by zyga> <https://github.com/snapcore/snapd/pull/7614> | 13:21 |
| zyga | jdstrand: I pushed it forward | 13:21 |
| jdstrand | zyga: sure | 13:23 |
| pstolowski | zyga: main/lxd failed for me as well | 13:23 |
| zyga | good | 13:23 |
| zyga | so it's a real thing | 13:23 |
| zyga | (reproducible) | 13:23 |
| zyga | pstolowski: try reverting the generator changes to see if that does something | 13:24 |
| jdstrand | "corecfg: add "system.timezone" setting to the system settings" | 13:25 |
| jdstrand | nice! :) | 13:25 |
| zyga | jdstrand: snap regedit ... | 13:25 |
| zyga | ;-) | 13:25 |
| jdstrand | ah, I wonder if that lxd thing is what was causing my spread tests to fail... | 13:25 |
| jdstrand | zyga: heh | 13:25 |
| zyga | jdstrand: I think we may have broken master | 13:25 |
| pstolowski | zyga: ok will do just in case, although it would contradict what ijohnson said | 13:26 |
| zyga | pstolowski: sure - just a sanity check | 13:26 |
| pstolowski | oh it wasnt't squash-merged | 13:26 |
| * jdstrand will keep an eye on commits to master for something to unbreak things then | 13:27 | |
| ijohnson | zyga: yeah I saw it on my PR's last night before the generator stuff had landed | 13:27 |
| zyga | ah, sorry, I couldn't hear that exactly | 13:27 |
| zyga | that's good, maybe something environmental changed | 13:27 |
| zyga | jdstrand: and comments in that PR are a bit in a weird state, I could not reply to some | 13:36 |
| zyga | jdstrand: so please open new comments on anything that you find wrong | 13:36 |
| mup | PR snapcraft#3165 closed: Update cmake plugin to support Ninja generator <enhancement> <Created by GamePad64> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3165> | 13:37 |
| zyga | pstolowski, ijohnson: interesting | 13:39 |
| zyga | in a debug shell look at the other container (my-ubuntu) | 13:39 |
| zyga | it has core installed as well | 13:39 |
| zyga | and guess what, core works | 13:39 |
| zyga | all the seeded snaps are broken | 13:39 |
| zyga | core is not seeded | 13:39 |
| zyga | does this make _any_ sense to you? | 13:40 |
| zyga | maybe what is really broken | 13:40 |
| zyga | is that /etc/systemd/system just doesn't have any mount units | 13:40 |
| zyga | maybe the image is broken to begin with | 13:40 |
| * zyga looks | 13:40 | |
| zyga | pstolowski: confirmed | 13:42 |
| zyga | the image is busted | 13:42 |
| zyga | however it was made is wrong | 13:43 |
| zyga | pstolowski: should I expect to see mount units inside a squashfs lxd image? (in a container image downloaded by lxd) | 13:43 |
| zyga | I see the snaps and everything else | 13:43 |
| zyga | but the mount units are simply missing | 13:43 |
| zyga | as if it was seeded but then something filtered out the .mount units | 13:44 |
| zyga | pstolowski: to reproduce look at /var/snap/lxd/common/lxd/images/97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540.rootfs on /tmp/dupa type squashfs (ro,relatime) | 13:44 |
| zyga | cc stgraber ^ | 13:44 |
| jdstrand | mvo: fyi, PR 8301 is deconflicted | 13:44 |
| mup | PR #8301: interfaces/many: deny arbitrary desktop files and misc from /usr/share <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8301> | 13:44 |
| zyga | stgraber: it seems that 20.04.1 images that lxd pulls in are snap seeded in a broken way | 13:45 |
| zyga | stgraber: etc/systemd/system does not have any mount units | 13:45 |
| stgraber | zyga: that'd be a bit concerning as the squashfs we use is supposed to be identical filesystem as the cloud image | 13:45 |
| zyga | stgraber: can you double check that I'm not making some rookie mistake | 13:45 |
| zyga | I mounted the .rootfs as downloaded by lxd | 13:46 |
| zyga | and looked at /etc/systemd/system therein | 13:46 |
| stgraber | it's this image in your case: http://cloud-images.ubuntu.com/releases/focal/release-20200804/ | 13:46 |
| zyga | are they all identical? | 13:47 |
| * zyga pulls the squashfs | 13:47 | |
| stgraber | they should be | 13:47 |
| zyga | a bit slow to pull, I'll let it do its thing | 13:48 |
| pstolowski | i wonder if they started preseeding them and something went wrong | 13:48 |
| zyga | the date says 4th of August | 13:48 |
| zyga | maybe it is before we had some fixes? | 13:48 |
| zyga | and it's just busted \ | 13:49 |
| ogra_ | zyga, my yesterdays focal image that i installed looks fine here ... why would mount units be n the readonly squashfs though ? | 13:50 |
| zyga | ogra_: IIRC because we pre-seed things | 13:50 |
| zyga | so they start faster | 13:50 |
| zyga | isn't the squashfs expanded anyway, it's just a template | 13:50 |
| zyga | it's not a snap | 13:50 |
| ogra_ | root@focal:~# ls /etc/systemd/system/*.mount|wc -l | 13:51 |
| ogra_ | 8 | 13:51 |
| zyga | ogra_: that's not what I'm seeing | 13:51 |
| ogra_ | root@focal:~# snap list|wc -l | 13:51 |
| ogra_ | 6 | 13:51 |
| zyga | what are the mount units? | 13:51 |
| zyga | I had 0 mount units | 13:51 |
| zyga | (though I'm still getting the image stgraber suggestd) | 13:51 |
| ogra_ | root@focal:~# ls /etc/systemd/system/*.mount | 13:51 |
| ogra_ | root@focal:~# | 13:51 |
| ogra_ | bah | 13:51 |
| ogra_ | let me pastebin that | 13:51 |
| zyga | I saw the failure in a spread test that was just "lxd launch ..." things | 13:51 |
| zyga | thanks | 13:51 |
| ogra_ | https://paste.ubuntu.com/p/HJJ3NN9hxm/ | 13:52 |
| zyga | right | 13:52 |
| zyga | this is more like what I would expect | 13:52 |
| zyga | I think an image is broken then, the only question is why and how to fix it | 13:52 |
| stgraber | stgraber@castiana:~/Downloads$ tar Jtvf ubuntu-20.04-server-cloudimg-amd64-root.tar.xz | grep etc/systemd | grep mount | 13:52 |
| ogra_ | thats what i got with "lxc launch ubuntu:20.04 focal" ... | 13:52 |
| stgraber | stgraber@castiana:~/Downloads$ | 13:52 |
| pstolowski | zyga: also, it's from 4th Aug but started failing yesterday? | 13:52 |
| zyga | pstolowski: yeah, I think we are missing something | 13:53 |
| zyga | stgraber: 97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540.rootfs can we map that hash to something on cdimage? | 13:53 |
| stgraber | pstolowski: assuming it's using ubuntu:, the images are manually promoted by CPC IIRC, though indeed feels a bit old | 13:53 |
| stgraber | zyga: I did it for you | 13:53 |
| stgraber | zyga: that's the URL I gave you | 13:53 |
| zyga | ah | 13:54 |
| * zyga looks at one more idea | 13:54 | |
| stgraber | current ubuntu:focal is indeed 20200804 on cloud-images.u.c | 13:54 |
| stgraber | ubuntu-daily:focal would be 20200814 | 13:54 |
| stgraber | there's a pretty good chance that the cloud images would similarly be missing those files, it's really meant to be identical across the board | 13:55 |
| stgraber | so you probably should be talking to CPC :) | 13:55 |
| mup | PR snapd#9173 closed: cmd: compile snap gdbserver shim correctly <Bug> <Simple 😃> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9173> | 13:56 |
| zyga | thanks mvo | 13:57 |
| ijohnson | sorry had to step out for IRL things | 13:57 |
| ijohnson | so is the lxd image just busted then ? | 13:57 |
| zyga | ijohnson: it seems so but unclear why | 13:58 |
| zyga | I noticed something buggy in the test | 13:58 |
| zyga | looking now | 13:58 |
| zyga | lxd.lxc launch --quiet "ubuntu:${VERSION_ID:-}" my-ubuntu | 13:58 |
| zyga | VERSION_ID is not defined anywhere | 13:58 |
| zyga | what does that fetch? | 13:58 |
| zyga | latest? | 13:58 |
| stgraber | ah so you went from bionic to focal last week then | 13:59 |
| zyga | I think it's a bug and it was meant to fetch the one matching | 13:59 |
| zyga | so it's likely that this bug triggers another one | 13:59 |
| stgraber | "ubuntu:" fetches the latest recommended LTS | 13:59 |
| stgraber | that was bionic until 20.04.1 | 13:59 |
| stgraber | now is focal | 13:59 |
| ijohnson | zyga: I think we get VERSION_ID from /etc/os-release which gets sourced somewhere in the prepare for a spread run iirc | 14:00 |
| zyga | ijohnson: I doubt that | 14:00 |
| zyga | thre was some code in that task.yaml | 14:00 |
| zyga | it's gone now | 14:00 |
| zyga | must have been killed by accident | 14:00 |
| zyga | and ${VERSION_ID:-} masked that | 14:00 |
| ijohnson | haha fair enough | 14:00 |
| zyga | it used to load version id from the host | 14:00 |
| zyga | because it was about alignment of the container to the system | 14:00 |
| zyga | as we copy stuff inside | 14:00 |
| pstolowski | zyga: i'm looking at ubuntu-20.04-server-cloudimg-amd64-root.tar.xz (also dated 4/08) is this as good as any other image there? fwtw it is NOT preseeded, it's still seeding the old way | 14:01 |
| zyga | I see | 14:01 |
| zyga | I'll start by fixing that VERSION_ID | 14:02 |
| zyga | but it seems we have more bugs | 14:02 |
| zyga | pstolowski: can you focus on trying to understand what's going on in the inner layer | 14:02 |
| zyga | I'll send a patch fixing the symptom we see | 14:03 |
| zyga | ijohnson: you broke it ;D | 14:05 |
| * ijohnson runs and hides | 14:06 | |
| ijohnson | zyga: was it my use the same version of lxd everywhere PR ? | 14:06 |
| zyga | d4a802934b1e62ec8924ea1c165a627695df5044 | 14:06 |
| zyga | that's the broken patch | 14:06 |
| zyga | anyway, fixes coming | 14:06 |
| ijohnson | zyga: haha to be fair though you approved the PR :-D | 14:07 |
| zyga | no hard feelings | 14:07 |
| zyga | at this point I'm beyond shame | 14:07 |
| zyga | I made so many bugs | 14:07 |
| ijohnson | yeah bugs happen | 14:07 |
| ijohnson | no worries | 14:07 |
| pstolowski | zyga: so what's actually broken? do i need to keep looking? | 14:08 |
| zyga | pstolowski: that test is broken when invoked with a focal image | 14:08 |
| zyga | pstolowski: on a xenial host | 14:08 |
| zyga | pstolowski: it should work | 14:08 |
| zyga | pstolowski: to reproduce just grab xenial VM | 14:08 |
| zyga | pstolowski: install lxd the same way | 14:08 |
| zyga | pstolowski: and boot focal | 14:08 |
| zyga | pstolowski: that should work and give you a working focal container with working snaps | 14:09 |
| pstolowski | zyga: ok so it's a test issue, not a real problem? | 14:09 |
| zyga | pstolowski: it's a real problem | 14:09 |
| zyga | pstolowski: the test just now stumbled onto it | 14:09 |
| zyga | pstolowski: it was not exercising that before | 14:09 |
| pstolowski | ah ok | 14:09 |
| zyga | pstolowski: but effectively started doing that because of another issue and recent promotion of focal | 14:09 |
| zyga | pstolowski: PR coming up, I've explained it there as well | 14:10 |
| zyga | just confirming that my patch fixes it | 14:11 |
| ijohnson | zyga: so the right thing for this lxd test is for a xenial host to run a xenial container and a focal host to run a focal container ? | 14:17 |
| zyga | ijohnson: yes | 14:17 |
| zyga | there's a comment there explaining why | 14:17 |
| ijohnson | zyga: I wonder if we should expand the lxd test to have more variants i.e. xenial host launches focal, bionic and xenial | 14:18 |
| ijohnson | oh | 14:18 |
| * ijohnson goes to read that comment | 14:18 | |
| zyga | yes but then we should not just blindly copy snapd | 14:18 |
| zyga | but I agree | 14:18 |
| ijohnson | ah yes | 14:18 |
| ijohnson | I see | 14:18 |
| ijohnson | see if we built the snapd snap and used that as part of our tests we could avoid this | 14:18 |
| zyga | yes | 14:18 |
| zyga | I'm sure we could | 14:18 |
| zyga | it's just a limitation of the current setup | 14:18 |
| zyga | all proper builds will work | 14:18 |
| zyga | (properly built rpm will work on rpm distros) | 14:19 |
| zyga | geez that test is sloooooow | 14:19 |
| ijohnson | mmmm also this test is disabled for uc20 too | 14:19 |
| zyga | (network) | 14:19 |
| ijohnson | we should re-enable that and make sure lxd works on uc20 | 14:19 |
| zyga | I agree | 14:19 |
| zyga | maybe worth splitting lxd test | 14:19 |
| zyga | and thinking of a cache for lxd images if that helps | 14:19 |
| zyga | FYI https://github.com/snapcore/snapd/pull/9174 | 14:19 |
| mup | PR #9174: tests: fix lxd test wrongly tracking 'latest' <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9174> | 14:19 |
| zyga | opened while local test is still going | 14:20 |
| zyga | so likely it works now | 14:20 |
| zyga | but I want to break now as my wife just returned from her first day of work after Lucy was born | 14:20 |
| zyga | I'll work later in the evening, to focus on bug triage | 14:20 |
| zyga | I think it's still broken though | 14:20 |
| zyga | just got this | 14:20 |
| zyga | https://paste.ubuntu.com/p/yJ6HThPdN6/ | 14:21 |
| mup | PR snapd#9174 opened: tests: fix lxd test wrongly tracking 'latest' <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9174> | 14:21 |
| * zyga akf | 14:21 | |
| ijohnson | zyga: approved | 14:21 |
| zyga | ijohnson: let's see it pass | 14:21 |
| ijohnson | well it "looks correct" :-) | 14:21 |
| zyga | I agree :) | 14:22 |
| zyga | oh | 14:22 |
| zyga | it passed | 14:22 |
| zyga | woooot | 14:22 |
| zyga | Ok | 14:22 |
| zyga | I'm really afk now | 14:22 |
| mvo | zyga \o/ | 14:22 |
| mvo | jdstrand: one quick question in 8301 | 14:23 |
| cmatsuoka | ijohnson: do you have any idea on why chooser triggering is not working in recover mode? the trigger file in /run is not there but I still didn't check if it's not being generated at all, or it's not being moved from initramfs to the real system | 14:30 |
| ijohnson | cmatsuoka: oh hey I tried that and it seemed to work for me with all edge snaps just now | 14:31 |
| cmatsuoka | oh really? then it must be something I'm doing locally, let me recheck | 14:31 |
| ijohnson | cmatsuoka: yes it took like a minute to trigger the chooser, etc. but it definitely wasn't 10 minutes or longer before the chooser came up | 14:41 |
| jdstrand | mvo: answered | 14:46 |
| mup | PR snapcraft#3251 opened: build providers: honour http proxy settings for snapd <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3251> | 14:47 |
| cmatsuoka | ijohnson: indeed it works with edge, so probably my remastering of initramfs is the source of the problem | 14:52 |
| cachio | mvo, hey | 14:57 |
| cachio | failover tests failing on master as well | 14:58 |
| mvo | cachio: hey | 14:58 |
| mvo | cachio: oh no! can you please paste the full error log? | 14:58 |
| cachio | mvo, https://paste.ubuntu.com/p/7c56xfYGwC/ | 14:59 |
| cachio | I just merged before running the tests | 14:59 |
| cachio | so my master is up to date | 14:59 |
| mup | PR snapcraft#3252 opened: snapcraft: use system certificates by default for https requests <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3252> | 15:02 |
| mvo | cAug 18 14:55:53 localhost.localdomain snapd[3776]: Aug 18 14:55:28 localhost.localdomain systemd[3584]: snapd.service: Failed at step EXEC spawning /snap/snapd/x1/usr/lib/snapd/snapd: Exec format error | 15:04 |
| mvo | hm, but maybe this is just the message that it failed correctly | 15:05 |
| mvo | cachio: the "external:" image, how was this build? | 15:10 |
| mvo | cachio: also, what is the output of "journalctl -u snapd-failover.service"? | 15:10 |
| cachio | mvo, ubuntu-image snap | 15:11 |
| mvo | cachio: I need to be afk for some minutes but I guess my question is, what commands do I need to run to trigger this error myself (to reproduce :) | 15:11 |
| * mvo is afk for some minutes but will read scrollback | 15:12 | |
| cachio | let me run again | 15:14 |
| cachio | mvo, wget https://storage.googleapis.com/spread-snapd-tests/images/pc-amd64-16-beta/pc.img.xz | 15:15 |
| cachio | sudo kvm -snapshot -smp 2 -m 1500 -net nic,model=virtio -net user,hostfwd=tcp::8022-:22 -serial mon:stdio pc.img | 15:15 |
| cachio | then run | 15:15 |
| cachio | export SPREAD_EXTERNAL_ADDRESS=localhost:8022 | 15:16 |
| cachio | ./tests/lib/external/prepare-ssh.sh localhost 8022 <your-lp-id> | 15:16 |
| cachio | spread -debug external:ubuntu-core-16-64:tests/core/core-to-snapd-failover16 | 15:16 |
| cachio | mvo, following those step you can reproduce 100% | 15:16 |
| pstolowski | zyga: i'm playing with this broken lxc container. in this state i cannot remove or refresh broken snapd snap, a dead end | 15:19 |
| cmatsuoka | ijohnson, mvo: I ran some extra tests here and apparently what makes the difference in the reboot from chooser is using a local, unchanged core20 snap vs the edge asserted core20 | 15:19 |
| pstolowski | zyga: interestingly, i could manually snap install lxd and core18 snaps from seeds, and after that snap set lxd waitready... woks | 15:20 |
| pstolowski | *works | 15:20 |
| cmatsuoka | ijohnson, mvo: erm, wait. that's strange. let me do it again | 15:20 |
| pstolowski | zyga: also saw this, not sure if it means anything https://pastebin.ubuntu.com/p/sJvDhxp5gy/ | 15:21 |
| cmatsuoka | ijohnson, mvo: the actual problem with the server connection timeout is caused by snapd snap injection, not sure exactly why, but it's local, so sorry for the noise | 15:41 |
| zyga | pstolowski: re | 15:55 |
| zyga | pstolowski: that's a known issue, I can explain but I think it is neither new nor a problem for this test | 15:56 |
| zyga | pstolowski: today it is late so I won't dig deeper | 15:56 |
| zyga | I will focus on bug triage and a small thing I was working on earlier | 15:56 |
| zyga | let's attack this tomorrow morning | 15:57 |
| pstolowski | zyga: yeah i think i'll stop here as well | 15:57 |
| zyga | https://github.com/snapcore/snapd/pull/9174 is making progress, though 16.04-64 is still in progress | 15:58 |
| mup | PR #9174: tests: fix lxd test wrongly tracking 'latest' <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9174> | 15:58 |
| mup | PR snapd#9175 opened: tests: find -ignore_readdir_race when scanning cgroups <Simple 😃> <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/9175> | 16:16 |
| mup | PR snapcraft#3253 opened: extensions: prepend the snapd glvnd path <Created by Saviq> <https://github.com/snapcore/snapcraft/pull/3253> | 16:18 |
| zyga | Saviq: o/ I noticed your question yesterday but I was off | 16:18 |
| zyga | Saviq: can you grab me tomorrow - we can discuss that then | 16:18 |
| zyga | ijohnson: commented and asked a question on https://bugs.launchpad.net/snapd/+bug/1889092 | 16:32 |
| mup | Bug #1889092: getent does not support extrausers on uc18 <snapd:Confirmed> <https://launchpad.net/bugs/1889092> | 16:32 |
| ijohnson | zyga: interesting point, perhaps getent works fine with uc20 or even uc22 | 16:34 |
| ijohnson | zyga: also your lxd test PR failed again same problem | 16:36 |
| zyga | https://github.com/snapcore/snapd/pull/9174#issuecomment-675586745 | 16:37 |
| mup | PR #9174: tests: fix lxd test wrongly tracking 'latest' <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9174> | 16:37 |
| zyga | no | 16:38 |
| zyga | not the same problem | 16:38 |
| zyga | mvo: we may need to revert pawel's generator | 16:38 |
| zyga | mvo: now that we test it better | 16:38 |
| zyga | it doesn't work on 16.04 | 16:38 |
| zyga | it was just never tested correctly | 16:38 |
| ijohnson | zyga: sorry how do you know that it is the generator? | 16:38 |
| zyga | the generator never ran | 16:38 |
| zyga | or ran and did nothing | 16:38 |
| ijohnson | zyga: your PR tests failed with the same configure hook problem | 16:38 |
| zyga | this test passed when we were testing against a different, more recent container (by accident) | 16:38 |
| zyga | ijohnson: https://github.com/snapcore/snapd/pull/9174/checks?check_run_id=998982860 shows a different error | 16:39 |
| mup | PR #9174: tests: fix lxd test wrongly tracking 'latest' <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9174> | 16:39 |
| ijohnson | Run configure hook of "lxd" snap (snap "lxd" has no "configure" hook) | 16:39 |
| zyga | I see this failure: | 16:40 |
| zyga | 2020-08-18T15:30:31.5864620Z Sanity check that mount overrides were generated inside the container | 16:40 |
| zyga | 2020-08-18T15:30:31.5864910Z + MATCH '/var/run/systemd/generator/snap-core-.*mount.d/container.conf' | 16:40 |
| zyga | 2020-08-18T15:30:31.5865202Z + lxd.lxc exec my-ubuntu -- find /var/run/systemd/generator/ -name container.conf | 16:40 |
| zyga | 2020-08-18T15:30:31.5865318Z grep error: pattern not found, got: | 16:40 |
| zyga | which suggests that what I wrote above is likely | 16:40 |
| ijohnson | zyga: ok so 16.04 failed but 20.04 failed same way with the configure hook | 16:40 |
| ijohnson | zyga: I was only looking at the 20.04 failure | 16:40 |
| zyga | I see | 16:40 |
| zyga | well | 16:40 |
| zyga | 20.04 is broken because the image is broken :) | 16:40 |
| ijohnson | hey bionic is working well then | 16:40 |
| zyga | we are now testing image matching the host | 16:40 |
| ijohnson | with pawel's generator | 16:40 |
| zyga | so it's totally expected that focal is broken | 16:40 |
| zyga | that's the issue pawel was looking into just a moment ago | 16:41 |
| zyga | and that's what we immediately saw | 16:41 |
| zyga | the PR doesn't change that, it just makes host match the container | 16:41 |
| ijohnson | right ok so this makes sense then | 16:41 |
| zyga | yeah | 16:41 |
| zyga | I think we need to fix more than one issue here | 16:41 |
| zyga | and if this is something we badly need for 2.46 we need to delay | 16:41 |
| ijohnson | so focal is totally broken because broken images, and xenial is broken because the generator didn't run | 16:41 |
| zyga | ijohnson: if focal is preseeded then yes | 16:42 |
| zyga | if it's not pre-seeded then I thing something else is at play | 16:42 |
| zyga | mvo: ^ please ack if this is a release blocker | 16:42 |
| zyga | I'll copy this log to pawel | 16:42 |
| zyga | so that he knows about it | 16:42 |
| * cachio lunch | 16:43 | |
| * zyga EODs | 16:44 | |
| === ijohnson is now known as ijohnson|lunch | ||
| mup | PR snapd#9176 opened: cmd/snap: use ⬏ instead of ↑ where applicable <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9176> | 17:42 |
| mup | PR snapcraft#3253 closed: extensions: prepend the snapd glvnd path <Created by Saviq> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3253> | 18:03 |
| mvo | zyga: hey, I was just looking at the scrollback | 18:31 |
| zyga | mvo: mmm | 18:31 |
| mvo | zyga: so the generator needs reverting? | 18:31 |
| zyga | mvo: it might | 18:32 |
| zyga | mvo: I have a hunch it doesn't work on 16.04 | 18:32 |
| mvo | zyga: oh, ok | 18:32 |
| zyga | mvo: the test was flaky, it never ran on a 16.04 container | 18:32 |
| zyga | mvo: we should discuss with pawel when he reads this tomorrow | 18:32 |
| zyga | mvo: I just wanted to let you know | 18:32 |
| zyga | mvo: as this seems to be relevant to 2.46 | 18:32 |
| mvo | zyga: totally | 18:33 |
| mvo | zyga: let's tackle this in the morning when pawerl and you and me are around | 18:33 |
| zyga | indeed | 18:33 |
| mvo | zyga: thanks for the heads up | 18:33 |
| zyga | :) | 18:33 |
| mup | PR snapcraft#3219 closed: meta: detailed warnings for resolution of commands <enhancement> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3219> | 18:58 |
| dariball | hey hey, I am building a ubuntucore image with some snaps targetting raspi4, is there some best practice to test the image in some virtualisation/emulation ? qemu seems not to be trivial. the ubuntucore docs propose `multipass` but I assume this is then x86 only as long as I do not run it on a raspi, right ? | 19:06 |
| zyga-mbp | ijohnson|lunch https://listed.zygoon.pl/17533/measuring-coverage-of-shell-scripts | 19:09 |
| === ijohnson|lunch is now known as ijohnson | ||
| zyga-mbp | dariball for raspi4 you pretty much need a raspi4 | 19:09 |
| ijohnson | zyga-mbp: nice! :-) | 19:10 |
| ijohnson | I'm really curious now how much of our shell scripts actually get run :-) | 19:13 |
| zyga-mbp | ijohnson so was I :) | 19:14 |
| zyga-mbp | although I have a second agenda here, that's a bit more annoying to accomplish | 19:14 |
| zyga-mbp | I want to write unit tests for makefiles | 19:14 |
| zyga-mbp | but the way make executes stuff makes that hard | 19:15 |
| zyga-mbp | this was a step towards that | 19:15 |
| zyga-mbp | ijohnson an aggregate mode would be nice | 19:15 |
| ijohnson | you mean a unit test for a makefile itself ? | 19:15 |
| zyga-mbp | something like bashcov --cumulative ... | 19:15 |
| zyga-mbp | yes | 19:15 |
| zyga-mbp | I wrote a build system a while ago | 19:16 |
| ijohnson | interesting what's the use case for it? | 19:16 |
| zyga-mbp | and I really want to get to 100% coverage and documentation | 19:16 |
| zyga-mbp | https://github.com/zyga/zmk/ | 19:16 |
| zyga-mbp | documentation is still sparse | 19:16 |
| zyga-mbp | but testing is pretty solid | 19:16 |
| dariball | zyga-mbp: means using a raspi4 with a regular ubuntu where I then run multipass, right ? | 19:16 |
| zyga-mbp | I know some things are not tested as I didn't write any tests yet (for certain modules) | 19:16 |
| zyga-mbp | but for those that are, I want to see if anything is missing | 19:17 |
| zyga-mbp | dariball no, I mean you need to run it on real metal | 19:17 |
| zyga-mbp | the best way forward is to automate deployment | 19:17 |
| zyga-mbp | we're doing that for tests of snapd and ubuntu core itself | 19:17 |
| zyga-mbp | but it's not something that's easy or fun | 19:17 |
| zyga-mbp | dariball CPU architecture is just one aspect of the problem | 19:17 |
| zyga-mbp | you really need an emulated "raspberry pi" virtualized but nobody made one that's not broken | 19:18 |
| zyga-mbp | so deployment on the real thing is really the only viable option | 19:18 |
| zyga-mbp | there are hardware add-ons that allow you to deploy to a SD card and boot a PI with that card | 19:18 |
| dariball | yeah this was my impression until now ... tried some qemu stuff, but thanks for confirming my impression | 19:18 |
| zyga-mbp | some companies manufacture them | 19:18 |
| zyga-mbp | some people make more or less successful implementations of that idea | 19:19 |
| zyga-mbp | IIRC canonical even has the most successful as a test engineer now :) | 19:19 |
| zyga-mbp | but anyway, it's not something I can recommend | 19:19 |
| zyga-mbp | unless you have a budget to spend | 19:19 |
| zyga-mbp | ijohnson as for testing | 19:19 |
| zyga-mbp | ijohnson I wrote unit tests in make | 19:19 |
| zyga-mbp | for make | 19:19 |
| zyga-mbp | ijohnson for example, how to compile a library written in C++ | 19:20 |
| zyga-mbp | https://github.com/zyga/zmk/blob/master/examples/libhello-cpp/Test.mk | 19:20 |
| zyga-mbp | but there's some complexity involved in making sure all combinations are covered | 19:20 |
| ijohnson | interesting I remember you mentioning zmk before | 19:20 |
| zyga-mbp | it's slowly growing | 19:21 |
| zyga-mbp | I paused all development while I was ill as working was hard as-is | 19:21 |
| zyga-mbp | anyway :) | 19:21 |
| zyga-mbp | i think bashcov is more interesting for us | 19:21 |
| zyga-mbp | ijohnson (although to be fair, that was a smoke test for an example, unit tests are more complex as they try to cover all the features, some being optional) | 19:23 |
| ijohnson | yeah bashcov seems very interesting in combination with our spread-shellcheck | 19:24 |
| zyga-mbp | for spread task.yaml's the problem will also be the fact that it just streams a bunch of text and not let us trace much | 19:26 |
| zyga-mbp | but we could find ways around that | 19:26 |
| zyga-mbp | we could patch spread to generate something that mimics the tests tree | 19:26 |
| zyga-mbp | and has actual scripts for everything | 19:26 |
| zyga-mbp | that source each other or what not | 19:26 |
| zyga-mbp | then bashcov could trace the whole execution | 19:26 |
| zyga-mbp | please read my post, play with the code, the idea came to me yesterday | 19:27 |
| zyga-mbp | and I implemented a working copy half an hour ago | 19:27 |
| zyga-mbp | I'm sure there's room for improvement | 19:27 |
| zyga-mbp | (bashcov handles ". foo.sh" sourcing today) | 19:27 |
| zyga-mbp | time to slack now | 19:28 |
| * zyga-mbp goes away | 19:28 | |
| mup | PR snapd#9177 opened: tests: remove support for ubuntu 19.10 from spread tests <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9177> | 20:32 |
| mup | PR snapd#9178 opened: secboot: document exported functions <Simple 😃> <Skip spread> <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/9178> | 21:07 |
| mup | PR snapcraft#3254 opened: tests: restrict colcon / ros2-foxy test to amd64 & arm64 <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3254> | 22:33 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!