[11:41] <zyga> waveform hello
[11:41] <zyga> waveform I have a curious bug
[11:41] <zyga> and I need some help
[11:41] <zyga> https://bugs.launchpad.net/snapd/+bug/1900693
[12:42] <waveform> zyga, will take a look
[12:42] <zyga> waveform thanks
[12:42] <zyga> I'm happy to aid, I think we are in a bit of a tough spot now
[12:42] <zyga> on updating the gadget
[12:42] <zyga> and the kernel
[12:42] <zyga> and gadget assets
[12:43] <zyga> cc mborzecki
[12:43] <zyga> waveform: I tried updating the gadget snap by hand, updating uboot
[12:43] <zyga> the kernel boots but there are some errors and there's no network at all
[12:43] <zyga> probably because the kernel is now out of sync with the device tree
[12:43] <waveform> very likely
[12:43] <zyga> I'm trying to think about two separate topics:
[12:44] <zyga> 1) what the correct things should be, in all the snaps, so that people can upgrade to a good working set
[12:44] <zyga> 2) what to do about the devices that are stuck in a failed state
[12:44] <zyga> and may not even update snapd (for now)
[12:44] <zyga> eventually snapd update will succeed
[12:45] <zyga> waveform please examine the bug and tell me what you think
[12:45] <zyga> waveform my hunch is to start with the uboot / kernel that is old and broken
[12:46] <zyga> and update uboot only, with gadget update, to resolve sd write issue
[12:46] <zyga> then we can attempt to update other bits
[12:46] <zyga> with synchronized kernel/gadget assets thank to what mvo did
[12:46] <zyga> but that's just my quick idea, not sure if that's workable
[12:46] <waveform> I didn't think the gadget *could* be updated on core18? Or have I misunderstood?
[12:47] <zyga> waveform it can except that it cannot because partition size was changed
[12:47] <zyga> and because actually upgrading it failed as I described now
[12:47] <zyga> cc mborzecki
[12:48] <zyga> gadget updates also require explicit bump of the edition field
[12:48] <waveform> ah right, and then there was the separate issue that the gadget contained the boot-time d-t, but it could be upgraded distinctly to the kernel snap and thus the d-t and kernel could get out of sync
[12:48] <zyga> simply building an image and upgrading an old image results in different boot assets
[12:48] <zyga> yes
[12:49] <zyga> waveform: I can experiment with an older uboot (such as that in revision 14 of pi snap)
[12:49] <zyga> with a patch that improves SD writing
[12:49] <zyga> but I need help to know how to build that
[12:49] <zyga> I can also share a vanilla image that fails
[12:49] <zyga> at least on my card
[12:49] <zyga> linux-side writes are okay
[13:43] <zyga> waveform, mvo_ I tried to capture my analysis of the pi problem in https://bugs.launchpad.net/snapd/+bug/1900693/comments/1
[13:51] <zyga> waveform added one more idea to the bug thread
[13:53] <waveform> zyga, afraid I don't know enough about snap construction to help much with that side of things but one thing I do note is that the env (or more precisely script) we use in core 20 doesn't *always* try and save the env; only when it's changed
[13:54] <zyga> waveform: yeah but that doesn't help in any way
[13:54] <waveform> I could whip up a similar env for 18 which might avoid some of the issue in future?
[13:54] <zyga> this is also true in core18
[13:54] <zyga> no
[13:54] <zyga> because the issue is when we write, we cannot
[13:54] <waveform> I thought core18 *always* tries to write
[13:54] <waveform> hmmm
[13:54] <zyga> waveform why would it
[13:54] <zyga> it's only writing when we are trying
[13:54] <zyga> try->trying change
[13:54] <zyga> anyway, the issue is not fixed by not writing
[13:55] <zyga> we must be able to write around some critical operations
[13:56] <waveform> sorry, you're right - it's only on the try/trying transitions
[14:19] <zyga> waveform, yeah, the error only happens on such boot operations that need to write to FAT from uboot
[15:30] <zyga> waveform do you think we could have a HO about the pi issue
[15:30] <zyga> waveform perhaps tomorrow morning
[15:30] <zyga> waveform or even maybe yoday
[15:30] <zyga> *today
[15:30] <zyga> I'd like to understand a few constraints better
[15:30] <zyga> and learn a few things enough to be able to debug the issue locally, if I can
[15:31] <zyga> my first goal would be to figure out how to reconstruct the pi gadet at revision 14, so that I can try to apply the patch that I referenced
[16:00] <waveform> zyga, sorry was in another meeting - I'm happy to jump on a HO but I can't say I know how to reconstruct a particular revision of the pi-gadget (I've no idea how those revision numbers map to commits in the original repo)
[16:53] <zyga> waveform: let's try to sync tomorrow
[16:58] <waveform> zyga, ok - feel free to stick something in my calendar
[16:58] <zyga> waveform thanks, will do