mupPR snapcraft#3323 opened: electron-builder spread test: sync expected snapcraft.yaml <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3323>00:03
mupPR snapcraft#3311 closed: storeapi: add releases endpoint <enhancement> <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3311>00:13
mupPR snapcraft#3306 closed: storeapi: add support for reporting status of progressive releases <Created by maxiberta> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3306>00:18
mupPR snapcraft#3323 closed: electron-builder spread test: sync expected snapcraft.yaml <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3323>01:28
=== benfrancis9 is now known as benfrancis
zygagood morning06:39
mvopstolowski: good morning07:05
zygahey mvo07:13
mvogood morning zyga07:13
zygasorry for the lag, a bit ucooperative kids today07:14
mvozyga: no worries07:14
zygamborzecki can you approve https://github.com/snapcore/snapd/pull/9499 - it'd be my most approved change ever ;-)07:27
mupPR #9499: tests: add tests for fsck; cmd/s-b/initramfs-mounts: fsck ubuntu-seed too <UC20> <Created by zyga> <https://github.com/snapcore/snapd/pull/9499>07:27
zygaalso reviewing ijohnson's fsck change for uc20 would be really good07:27
zygaI've de-conflicted and updated https://github.com/snapcore/snapd/pull/942207:35
mupPR #9422: overlord: add link participant for linkage transitions <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9422>07:35
zygaI hope it's in a good shape now and we can make progress on the export manager07:35
mborzeckizyga: hm am i missing something? https://github.com/snapcore/snapd/pull/9499#discussion_r50827337307:35
mupPR #9499: tests: add tests for fsck; cmd/s-b/initramfs-mounts: fsck ubuntu-seed too <UC20> <Created by zyga> <https://github.com/snapcore/snapd/pull/9499>07:35
zygahow did I do that07:35
zygaI understand what my editor asked me to save now07:36
zygaI resolved and closed without saving07:36
zygapedronis good morning07:49
zygapedronis I've updated https://github.com/snapcore/snapd/pull/942207:49
mupPR #9422: overlord: add link participant for linkage transitions <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9422>07:49
zygapedronis I think it should be good now, at least in principle07:50
pedronisthx, I put it back into my queue07:51
pstolowskilxd snap was fixed yesterday,  #9468 is unblocked07:53
mupPR #9468: tests: lxd smoke test <Simple πŸ˜ƒ> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9468>07:53
zygapedronis thanks :)07:56
* zyga warmed up the tea07:56
zygait's still very cold in the office, heating up slowly07:56
zygapstolowski reviewed07:57
zygaI'll work on the pi kerenl thing now07:59
pstolowskizyga: thanks, adding some comments. not sure about restore, snap remove is the main check, i think the new comments will make it clear08:18
mborzeckipedronis: i've updated #9476 with some additional tweaks to make nosecboot happy08:50
mupPR #9476:  many: have install return encryption keys for data and save, improve tests <Run nested> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9476>08:50
zygagoogle:ubuntu-20.10-64:tests/main/snapd-reexec-snapd-snap fails on something silly, probalby assumes core is installed09:19
zyga20.10 has snapd preinstalled09:19
zygashould we disable that test there?09:19
zygamvo ^09:19
mvozyga: in a meeting right now09:19
mvozyga: sorry, can think in a bit but not right now09:20
mupPR snapd#9499 closed: tests: add tests for fsck; cmd/s-b/initramfs-mounts: fsck ubuntu-seed too <UC20> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9499>09:21
mupPR snapd#9522 opened: o/snapstate: ignore remove errors in clear-snap handler, only log them <Created by stolowski> <https://github.com/snapcore/snapd/pull/9522>09:21
zygapstolowski: check my comment please09:23
pstolowskipedronis: hi, if you have a moment, what's your take on warnings vs task log in #9522?09:34
mupPR #9522: o/snapstate: ignore remove errors in clear-snap handler, only log them <Created by stolowski> <https://github.com/snapcore/snapd/pull/9522>09:34
zygabooting rogrer's configuration now09:44
zygagosh I love serial lines09:44
zygais this a bug: /init: /scripts/init-bottom/ORDER: line 3: /scripts/init-bottom/disable-getty: Permission denied09:44
zygathat's from initrd09:44
zygaseems like missing +x09:44
zygasecond error: /usr/share/subiquity/console-conf-wrapper: line 13: snap: command not found09:45
zygathird error (trying to use wifi) https://pastebin.ubuntu.com/p/qXbB66tfjz/09:47
zygaI cannot get past this09:52
zygamborzecki do you recall this subiquity erorr ^ /09:53
amurrayzyga: thanks for finishing off the snapd/apparmor3 stuff :)09:53
zygaamurray you were right after all :)09:54
zygahmm, I wonder if subiquity will refresh09:55
zygadoes anyone know if there's something I can set in uboot to bypass subiquity on core18?09:57
zygaI think the snapd snap I'm getting crashes on startup09:58
zyga[FAILED] Failed to start /tmp/tmp.spBbI7OTQH/usr/lib/snapd/snapd.09:58
zygapstolowski ^ any ideas?09:58
zygaideally on how to debug any of this09:58
pstolowskizyga: is this a private build of snapd?10:00
zygapstolowski nope10:01
zygapstolowski ubuntu-image with the three snaps I got from mvo10:01
zygaand snapd downloaded from the store (automatically)10:01
mvozyga: still in a meeting but i can give you more snaps if you need them10:01
zygamvo: I think whatever is wrong here also is interesting10:02
pstolowskizyga: maybe mount the image and see if the snaps looks sane ?10:02
zygayeah, trying now10:02
pstolowskiseems like something fundamental failed10:02
zygaI think snapd did not seed10:06
zygaI've enabled journal10:06
zygalet's see what we get10:06
pstolowskimborzecki: do you have a moment for #9468?10:07
mupPR #9468: tests: lxd smoke test <Simple πŸ˜ƒ> <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9468>10:07
mborzeckipstolowski: sure, will take a look10:08
zygaOct 20 10:05:57 localhost run-snapd-from-snap[607]: /usr/lib/core18/run-snapd-from-snap: 42: /usr/lib/core18/run-snapd-from-snap: /usr/bin/snap: not found10:08
zygapstolowski I'd like to "reset" seeding10:08
zygacan I just remove state json?10:09
zygawait there is no state.json10:09
zygaI only see this10:09
zygapstolowski note that the snaps/* are symlink10:09
zygaand there are only two10:09
mborzeckizyga: try talking to ondra, he probably has some hacks for postprocessing the image to add users etc10:12
zygaI've added a ssh key for root10:13
zygashould be enough10:13
mborzeckipstolowski: does the test need to launch a container?10:14
pstolowskiyes, symlinks are expected, this is what makes runnable on first boot, but i'm not sure why snapd isn't symlinked; indeed it seems seeding did't run10:14
pstolowskimborzecki: no10:15
pstolowskimborzecki: it was failing pretty reliably with these steps10:15
zygaI'm in10:16
pstolowskimborzecki: thanks!10:16
mupPR snapd#9468 closed: tests: lxd smoke test <Simple πŸ˜ƒ> <Test Robustness> <Created by stolowski> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9468>10:16
zygaexec format error10:16
zygaubuntu image pulled x86-64 snapd!??!10:16
zygano sorry10:17
zygaI have arm7l kernel10:17
zygaand aarch64 snapd10:17
zygathat's fun10:17
zygamy fault10:18
zygabooting fixed image10:21
zygareported https://github.com/snapcore/pi-gadget/issues/5310:24
zygaI went for coffee10:30
zygaand the device is updating and rebooting10:30
zygabut still no console-conf10:30
zygaI'll keep it running10:31
zygait's progressing10:32
zygait's mounting snapd now10:32
zygathis is what I got on the serial log so far10:33
zygaI've hit newline a few times when it took a long time to make any progerss10:33
zygaall this time no console conf10:34
zygamore getty changes10:34
zygahmmm hmm10:35
zygaI wonder if this makes any sense to anyone10:35
zygagetty changes again10:36
zygaI'll reboot10:36
zygathis makes zero sense10:36
zygaok, I have console conf now10:37
zygaand it crashes like before10:37
* zyga debugs10:37
zygathe system cannot seed10:40
* zyga looks at why10:40
zygaok, I'll enable journal and ssh and try again10:43
zygathere's nothing I can see that explains what failed10:43
zygastate.json doesn't have any useful messages10:43
zygatrying again10:46
zygaok, I'm in while it's still trying to seed10:53
zygasnapd is not seeded correctly again10:54
zygathis is so weird10:55
zygaOct 20 10:56:31 localhost snapd[750]: taskrunner.go:271: [change 2 "Make snap \"snapd\" (9731) available to the system" task] failed: open /etc/dbus-1/session.d/snapd.session-services.conf.7PTDJQPMf10:56
zygait fails on this!10:56
mupPR snapd#9523 opened: store, snap-repair: fix use of retry strategies <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9523>10:57
zygaright, because the fixed snapd is only later10:57
pstolowskibut wait10:57
pstolowskiwith old core18 it should be fine?10:57
zygabut this is old core18 and new snapd10:57
pstolowskionly a problem once you refresh no?10:57
zygabtw, offtopic, why do we have to seeding services?10:57
zyga  snapd-seeding.service                                                                                loaded    active     running         /tmp/tmp.jFHQH5ZiJr/usr/lib/snapd/snapd10:57
zyga  snapd.seeded.service                                                                                 loaded    active     exited          Wait until snapd is fully seeded (core18)10:57
zygaperhaps we should give them better names (cc mvo)10:57
pstolowskimvo: i've set 2.47 milestone on #952310:58
mupPR #9523: store, snap-repair: fix use of retry strategies <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9523>10:58
zygapstolowski one with dot, one with dash10:58
zygapstolowski snapd-seeding and snapd.seeding10:58
pstolowskizyga: yeah, got it, you said "to", wasn't sure10:59
zygaah, sorry10:59
zygaswitching between macbook and hacker keyboard10:59
zygamind confused :)10:59
pedronis? one is "seeded" not seeding10:59
zygathat's right11:00
pedronisthey do very different things11:02
zygayes, just have inconsistent naming11:02
zygaOct 20 11:03:20 localhost systemd[1]: snapd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.11:04
zygasnap tasks NNN is super slow due to locks11:05
zygait's not a good experience :/11:05
zygait's practically unusable when there are changes in progress downloading snaps11:05
mvopstolowski: thank you11:06
mvozyga: still in meeting :/ but +1 to anything that adds clarity11:07
zygamvo hard to change that11:07
* mvo has not read backlog :/11:07
zygamanaged to get snap tasks after several minutes of waiting11:08
zygarefreshing core18 explicitly11:08
zygalet's see if we learn something11:08
zygaI've set refresh time now11:10
zygaso no auto updates for a while11:11
zygaI'll get core18 refreshed in a few minutes11:11
zygawith a bit of luck that will show what was failing for Roger11:11
zygahere we go, rebooting11:16
zyga[ 1839.110777] systemd-shutdown[1]: Failed to wait for process: Protocol error11:17
zygapstolowski, mborzecki11:17
zygamvo I've reproduced the failure on that pi11:18
mborzeckizyga: so old kernel/gadget?11:18
zygacheck this out11:19
zygamborzecki but before rebooting11:21
zygathere was a message about system-shutdown11:21
zygait seems it really did fail11:21
zygaand now fat is fcks up11:21
mvozyga: \o/ you rock!11:21
zygalet me plug the device back11:21
mborzeckizyga: you think that boot may be corrupted?11:22
zygafuck yeah11:22
zygatwo power cycles11:22
zygaand it's back11:22
zygaI'm not kidding11:22
zygathis is exactly what happened on that other pi11:22
zygaI'll paste the more complete log now11:22
zygaso now, some research time11:24
zygaI'll install core to have fsck available11:25
mborzeckihm what mounts /run/mnt/ubuntu-seed on /var/lib/snapd/seed?11:27
zygamborzecki dunno11:28
zygacore20 bit for sure11:28
mborzeckihaha :P11:30
zygayeah, this is totally reproducible11:31
zygacore18 cannot refresh, we shutdown uncleanly11:31
zygaw[FAILED] Failed to start Ubuntu core (all-sn…tem shutdown helper setup service.11:31
zygaSee 'systemctl status snapd.system-shutdown.service' for details.11:31
zygaI'll report a detailed bug capturing this11:33
zygaI think core18 is good later11:33
zygabut not yet11:33
mborzeckiso i mount ubuntu-save under /run/mnt/ubuntu-save, and bind mount it under /run/data/system-data/var/lib/snapd/device/save11:33
zygaremember this is old coe1811:33
zygamborzecki it's a propagating mount11:33
mborzeckiquite sure there's some rbind missing somewhere11:34
mborzeckiprobably sysroot-writable.mount (in core20-initrds aka ubuntu-core-initramfs), or/and handle-writable-paths11:35
mborzeckibut i see /var/lib/snapd/seed is mounted, and maybe we could do the same instead, not sure though what mounts it there11:35
pstolowskizyga: amazing stuff!11:39
mupBug #1900693: snapd cannot refresh on some SD cards due to uboot bug <snapd:New> <https://launchpad.net/bugs/1900693>11:39
pedronismborzecki: is done here I think (via fstab): https://github.com/snapcore/core20/blob/master/static/usr/lib/core/handle-writable-paths#L18311:39
zygaok, I'll try to update uboot now11:39
mborzeckipedronis: thanks!11:40
zygamborzecki who is our PI maintainer?11:41
mborzeckiheh, noticed it's at the end of the file and was looking for something that may modify fstab after populate-writable runs11:41
mborzeckizyga: try poking waveform11:41
zygaI suspect we could use a repair assertion11:42
zygawell, we can see11:43
pstolowskidang, i'm not sure my diagnosis re retry strategies was correct11:43
mborzeckimeh, shellcheck is unhappy about handle-writable-paths12:01
cachiomborzecki, hey12:10
cachioabout the tumbleweed image12:10
mborzeckicachio: hey12:10
cachiomborzecki, this is the script we use to create that image12:11
cachiomborzecki, what could we do about that to make tumbleweed work better?12:11
mborzeckicachio: we start from leap?12:11
cachiomborzecki, yes opensuse-15.212:12
cachiomborzecki, I am importing this image opensuse-cloud/opensuse-leap-15-2-v2020070212:13
mborzeckicachio: i guess there's no tumbleweed image we could use?12:13
cachiomborzecki, no12:14
cachiolet me check again12:14
cachiojust to make sure12:14
cachiomborzecki, 15 15.1 and 15.212:17
zygaFYI I asked in #opensuse-admin12:17
mborzeckithat sucks a bit12:18
zygaI asked sysrich about this a while ago12:18
zygaand he did produce a link to TW images12:18
zygabut said they are not QAd12:18
mborzeckii mean, upgrading from a stable release is supposed to be supported so maybe we just need to figure out which pacakges are incorrect?12:19
mborzeckizyga: got link?12:19
mborzeckicachio: when was the last time you built an image?12:19
zygamy memory was rusty12:19
cachiomborzecki, the current tumbleweed is  opensuse-tumbleweed-64-v2020092312:20
cachioand the base we use  opensuse-tumbleweed-64-base-v2020083112:20
cachioso 1 month ago is the last one12:20
zygacachio IIRC the problem was that it had a leap kernel12:21
zygaand was some sort of frankenstein that should not be needed if we had a real TW system12:21
zyga(real = vanilla)12:21
cachiozyga, yes12:21
mborzeckiso i ran zypper dup and it wants to install this kernel: `kernel-default   5.8.14-1.2           x86_64  repo-oss  openSUSE`12:21
mborzeckiwhich is tw12:22
mborzeckicachio: so maybe the image build should actually run more often (weekly?), and run zypper dup once more after the first reboot?12:23
cachioit runs weekly12:23
cachiobut it is failing with differnet errors12:24
cachioI can trigger a new one12:24
zygamaybe we could work through those errors12:24
zygaare you talking about test failures?12:24
zygaor about upgrade failures12:24
cachiozyga, upgrade failures12:25
cachiosome incompatibilities with google packages12:25
cachiothis is the kernel in tumbleweed now12:25
zygapstolowski: upgrading the kernel is equally futile12:25
zygapstolowski and as I discussed with maciek, upgrading the gadget is impossible now12:26
zygamvo: pi is busted :P12:26
zygamvo: needs some coordination to unbreak12:26
cachiozyga, mborzecki this is the last error upgrading12:27
pstolowskizyga: that's really terrible12:27
pstolowskizyga: maybe add your findings under the original bug from Roger?12:28
zygayeah, I will, still looking at some ideas12:28
mborzeckicachio: hm not sure why you're seeing those, i'm running zypper dup on the tw image we have and it seems tow ork12:34
cachiomborzecki, I think it is because we force the leap 15.2 kernel12:35
mborzeckicachio: why do we have that?12:35
zygamborzecki, cachio: https://download.opensuse.org/tumbleweed/appliances/12:35
cachiothis is because of a bug12:35
zygaI think we want to try https://download.opensuse.org/tumbleweed/appliances/openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow212:35
cachiomborzecki, https://github.com/snapcore/spread-images/blob/master/tasks/google/update-opensuse-tumbleweed/task.yaml12:36
cachioit used to get stuck when doing govendor sync12:36
cachioso we had to force that kernel12:36
mborzeckicachio: hm can we try with the current kernel?12:37
cachiomborzecki, yes12:37
cachiomborzecki, in few minutes the new image will be available12:55
zygacachio: perhaps missing call to lxd cleanup13:38
* zyga breaks for lunch 13:52
zygacachio I can talk later13:52
cachiozyga, sure, I am reviewing the code now13:52
zygaafter lunch I will fix xdg-settings13:52
zygathe test fails for real with app tracking on fedora13:52
zygabut probably for good reasons13:52
mborzeckioff to pick up the kids13:59
* zyga is back and looks at xdg-settings14:20
mupPR snapcraft#3322 closed: package repositories: drop $SNAPCRAFT_APT_HOST_ARCH variable <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3322>14:20
pedronisijohnson: I did a pass on both #9418 and #9489 now14:27
mupPR #9418: many: implement snap routine console-conf-start for synchronizing auto-refreshes <Needs Samuele review> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9418>14:27
mupPR #9489: daemon,client: write and read a maintenance.json file for when snapd is shut down <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9489>14:27
ijohnsonpedronis: thanks I'm still on lk this morning but will take a quick look now14:28
mupPR # closed: snapcraft#3303, snapcraft#3308, snapcraft#3310, snapcraft#331314:30
* zyga errand 14:35
pstolowskipedronis: interesting issue with assertions rest api  - https://bugs.launchpad.net/snapd/+bug/189915415:03
mupBug #1899154: importing assertions in snapd: no errors reported <snapd:Triaged> <https://launchpad.net/bugs/1899154>15:03
pedronispstolowski: probably easy to fix if we have a reproducer ?15:27
pstolowskipedronis: for missing/bad body yes, i haven't checked other cases15:27
pstolowskimaybe errors do not bubble up somewhere15:28
pedronisthat sound strange, but easy to check15:29
pstolowskipedronis: yes i can get back to it later, just triaging now15:30
pedroniswe do have unit tests for the basic stuff fwiw15:31
pedronismaybe a problem in client15:31
pedronisbut they were using the api directly?15:32
pstolowskipedronis: yes, that's my understanding15:33
pstolowskipedronis: anyway, i'll add to my list to look at later15:33
pstolowskipedronis: btw, wrt another bug, do you know if it was intentional that we do not automatically carry-over devmode flag when you manually refresh a snap that was previously installed with --devmode?15:34
pstolowskiseems to me like this could be by design15:35
pedronisthat is intentional I think15:37
pstolowskiok, i think it's sensible given that devmode is cleary described in our docs as special mode for developers etc15:39
pedroniswe don't even auto-refresh devmode snaps iirc15:41
* cachio lunch15:58
mupPR snapcraft#3324 opened: set ROS_PYTHON_VERSION for rosdep <Created by artivis> <https://github.com/snapcore/snapcraft/pull/3324>16:05
mupPR snapd#9422 closed: overlord: add link participant for linkage transitions <Needs Samuele review> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9422>16:18
mupBug #1900730 opened: "snap list" hangs if snapd.socket service isn't running <Snappy:New> <https://launchpad.net/bugs/1900730>16:21
mupPR snapcraft#3325 opened: snapcraftctl: add checks for empty string for set-version & set-grade <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3325>16:50
zygathank you mvo16:53
zygarebased the export manager17:20
zygarunning a few smoke tests now17:20
zygabut should be good17:20
zygaand I thought of a new test to write17:20
zyga to ensure the fallback is not wrong17:20
mupPR snapcraft#3326 opened: lxd unit tests: simplify command checking pattern <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3326>17:45
zygaI've tested that we can disable the export manager, and use the corresponding old logic and apparmor profiles18:18
zygaI think it's not worth the cost of a full duplication of all tests but it was a useful run18:19
zygahttps://github.com/snapcore/snapd/pull/9384 needs review18:32
mupPR #9384: overlord: export and use snapd tools <Created by zyga> <https://github.com/snapcore/snapd/pull/9384>18:32
* zyga EODs18:32
cachioijohnson, hi19:11
cachioquick question19:11
ijohnsonhey cachio19:12
cachiouboot is used just in arm devices right'19:12
cachioijohnson, is there any other device where is it used like some amd64 once?19:13
ijohnsoncachio: uboot as we use it is just for armhf / arm64 devices19:13
ijohnsoncachio: but I mean maybe it could be used for other arches, I honestly don't know I've never tried it, I believe actually it works also for riscv64 and mips and such but we don't support ubuntu core on those architectures19:14
ijohnsoncachio: why do you ask ?19:14
cachioijohnson, ok, thanks, that answer my question19:14
cachioit is just for a test19:14
mupPR snapcraft#3327 opened: Fix use case when multiple packages are compiled with catkin <Created by facontidavide> <https://github.com/snapcore/snapcraft/pull/3327>19:21
mupPR snapcraft#3328 opened: package repositories: drop $SNAPCRAFT_APT_RELEASE variable <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3328>19:46
mupPR snapcraft#3328 closed: package repositories: drop $SNAPCRAFT_APT_RELEASE variable <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3328>21:36
mupPR snapcraft#3317 closed: plugin handler: properly handle snapcraftctl errors <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3317>21:46
mupPR snapd#9524 opened: tests: new boot state tool <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9524>23:15

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!