=== harrisj_ is now known as harrisj [08:23] hey Chipaca, good morning! [08:24] 7096 needs a second review, should be an easy one [08:24] mvo: looking [08:24] ta [08:36] mmh [08:44] pedronis: mmh? (and good morning :) [08:44] I have vague memories that the ! would go in front , but I'm probably misremembering, the notes says after [08:45] also the shell would scream if in front [08:45] so I'm misremembering for sure [08:46] pedronis: oh, indeed [09:18] also in the context of the unset work, we shouldn't forget we decided to fix snap set a.x=... a=... by sorting [09:19] pedronis: good point, I wonder if we captured this in one of the trello cards [09:20] pedronis: what kind of sorting? [09:25] ogra: mvo: do you know, offhand, how to pause the live cdimage before it boots into the actual live system? wanting to set SNAPD_DEBUG :) [09:26] not sure you can ... but there should be a console on tty4 [09:27] (so you can do your edits and restart all involved bits) [09:28] ogra: the thing i'm wanting to see is the seeding, and i'm not fast enough [09:28] Chipaca: uh, thats a tricky one. I wonder if you could break into initramfs [09:29] yeah, thats a possibility but also a lot of time consuming work [09:29] (manually assmbling the rootfs, chrooting and editing etc) [09:29] eugh [09:29] I've asked laney, maybe they have a trick for it [09:29] try break=bottom [09:29] on the kernel commandline [09:30] yeah, break=bottom was what I was thinking, then tweak and exit the shell and hope for the best [09:30] trying that [09:30] (there should be a way to enter this at the very start) [09:33] break=bottom and then echo SNAPD_DEBUG=1 >> /root/etc/environment worked [09:37] Chipaca: \o/ [09:38] need to step out for a bit, will bbiab [09:40] Chipaca: by depth [09:49] 6878 (health status in info/list has two +1, I will merge it, I left one bikesheed comment though, but its fine to ignore that) [09:53] mvo: it's in the card https://trello.com/c/yxC3omC9/245-snap-unset-support [09:53] but not sure it was done yet (haven't looked at the PRs) [09:53] pedronis: excellent, thank you! [09:54] pedronis: not done yet in any of the PRs I have seen so far [09:54] fwiw 7098 and 7102 should be easy wins [09:56] we are also missing snapctl unset [09:57] pedronis: good catch [09:58] anyway I added actual todos to the card [09:58] pedronis: thanks for adding it to the card! [10:03] fwiw - I closed 5915 now (netplan apply) now that we have #7107 [10:03] * mvo really misses mup btw [10:04] mvo: thank you [10:05] 7100 is now also green - its "funny" one, it requires that we update gorilla too [10:06] it needs Chipaca to look at it, then I will look at it as well [10:10] sergiusens, hi, would you consider merging https://github.com/sergiusens/snapcraft-preload/pull/32? we'd need it for postgres, which tries to create /dev/shm/PostgreSQL.XXXXX files [10:20] mvo: wrt 7100, why not have the prereq helpers also take context? [10:21] mvo: they're called from a do* handler so you've got a tomb so you've got a natural context already [10:22] Chipaca: aha, let me check that [10:22] mvo: i mean, if they break all the tests in the world, maybe it's worth leaving them for later :-) [10:22] but otherwise, ...? [10:23] Chipaca: let me explore that [10:25] Chipaca: I'm a bit slow (sry!) do you have an example helper name that could take the context for me? not sure I know what you have in mind [10:25] mvo: installOneBaseOrRequired [10:25] mvo: via installPrereqs [10:25] Chipaca: aha, yes [10:27] Chipaca: I can do that, however I think the reason its not done (now) is that the client-user-agent will only be set (in this PR) in the SnapAction store API call. once the change is created the client-user-agent is no longer available in the change. we could fix that but its more work and the win seems to be small. or am I missing something? [10:28] Chipaca: we would have to persist the client-user-agent somewhere in the change if we want to keep it and use it in the handlers [10:30] mvo: hm... [10:30] ah so it's not in the tomb at all [10:30] sary, missed that disconnect again [10:31] mvo: in any case +1 [10:44] pedronis: does request-serial in classic hit the actual store? [10:46] Chipaca: \o/ thank you! [10:46] 7098 is green and has one +1 already and is small… [10:50] mvo: it's got two +1's though [10:50] Chipaca: yes [10:50] pedronis: that's taking ~30s [10:51] pedronis: which is the same as seeding takes [10:51] on this machine [10:51] pedronis: seems slow for a single request [10:51] ¯\_(ツ)_/¯ [10:51] that is interesting [10:52] pedronis: https://snapforum.s3.amazonaws.com/original/2X/5/55ca16db9b44fe934424c27999c3386948b4fb4e.png [10:53] Chipaca: heh :) [10:53] Chipaca: we are down to 51! [10:53] pedronis: it's prepare-serial-request, not submit-serial-request, that takes that long though [10:53] Chipaca: ah, but we don't split retries [10:53] so a bit unclear [10:53] if that's one request or many retries [10:53] pedronis: you can ask [10:53] pedronis: you can ask me, even [10:54] or you can get the iso and look for yerself :-p [10:54] prepare-serial-request [10:54] that is even weirder [10:55] anyway i need to go comb my hair (just one of them) and get ready for the show [10:55] are we doing or changed something silly [10:55] we ask only a nonce from the store there [10:55] I need to dig/look around a bit [10:56] pedronis: you ok to repro this on your own, or should i get the logs somewhere [10:56] pedronis: instructions here: https://forum.snapcraft.io/t/snap-seeding-is-slow-racy/12310 [10:56] Chipaca: well, I'm not promising to repro it [10:56] I'm promosing to understand what we need to look [10:56] more precisely [10:56] pedronis: fair enough [10:56] i look forward to learning more :-D [10:56] but now i need to run [10:58] Chipaca: I added a comment to 7083 that might be worth doing (not sure though, depends on the plans with 7092) [11:06] mvo: ok, i'll take a look in a bit [11:06] afk from irc but github will work :-) [11:15] I left some comments there [11:17] Chipaca: it takes ~200ms to get a nonce from the store, so something else is going on === ricab is now known as ricab|lunch [12:41] 7102 is another easy win [13:33] xnox: snapd PR#7031 (re-enable the system PATH generator) is fixed now, right? or do we need some initramfs fix too (I have some vague memories that this was discussed but don't know the conclusion) [13:34] xnox: once you +1 it I will merge it [13:56] sil2100, could we get this into all gadgets (core16 and 18) soon ? https://github.com/snapcore/pi3-gadget/pull/28 === ricab|lunch is now known as ricab [13:57] (i asked ondra for a signoff too, to get more eyes on it, but in general someone should really monitor the pi firmware and push regular updates to the gadgets, they often fix over/under vlotage issues, overheating etc) [14:00] ogra: sil2100 i thought we just landed the same thing [14:01] cwayne, well, i couldnt make any of our official images boot on the CM3 on my desk ... neither core16 nor core18 (though i dont look much at 18 anyway given the quality) [14:01] CM3+ [14:01] sorry [14:02] using my universal pi-kiosk snap with the latest firmware (also for pi4 support) boots it fine though [14:02] otherwise it always hangs on the rainbow splashscreen (meaning the rpi bootloader itself dies before it can load anything) [14:07] ogra: did you try with a very recent image? think the updated gadget landed in stable ~5hrs ago [14:07] worked for 18 for us [14:09] i tried both, 16 and 18 from the current dir on cdimage [14:11] Hi! one of my snap failed to publish because I added a new hook, is that normal will have to go through a review or is that something that needs changing in my yaml ? [14:12] the warning was "unknown hooks in meta/hooks: 'disconnect-plug-services-content' lint-snap-v2_unknown_hook" [14:12] i also dont get why we still have separate images for pi2/3/cm3 and dont use the universal gadget, i thought we moved to that [14:19] jdstrand, hi, wrt /run/uuidd/request, do you think it makes sense to file a bug for an interface request? I see it's currently available in openvswitch-support [14:20] roadmr ^ [14:20] mvo: you are correct. Needs https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1814355 tasks for initramfs-tools done. [14:20] let me comment on the PR [14:31] xnox: thank you [14:45] ackk: last we talked in Lyons, blake, et al all felt that the snap should be adjusted to use /proc/sys/kernel/random/uuid, which is standard and in the default template). there is nothing that will guarantee that /run/uuidd/request will be there on all distros. that said, it seems systemd is providing it and snapd requires systemd (and 14.04 is no longer supported by snapd) [14:48] jdstrand maybe we can use snap layouts for that [14:48] jdstrand, we don't explicitly use that socket, libuuid does that [14:49] ackk: well, sure, but why use libuuid? :) [14:49] jdstrand, actually we can't use templates as the kernel one is just a file you cat [14:50] jdstrand, because it's somewhat standard? IDK we don't use it explicitly :) [14:50] I mean it's not a direct dependency [14:50] a lot of stuff use that [14:51] * jdstrand wonders why it doesn't use the kernel interface rather than uuidd [14:51] * cachio afk [14:51] (when the kernel supports it) [14:51] jdstrand, dunno, portability? [14:51] I mean, sure libuuid could use that file if it's there [14:51] right [14:51] anyway, I've taken a todo to investigate this for the default template [14:52] I suspect systemd will be looking at that file [14:52] jdstrand, the python uuid uses os.urandom() afaics [15:01] ogra: that should be released already [15:01] ogra: the 'master' branch is out-of-date and not used [15:01] ogra: I actually need to just remove it [15:01] ijohnson: just fyi, I'm poking at the tests in your netplan-apply pr right now [15:01] ogra: there are 16 and 18 branches, and both have the latest uboot bumps for the pi3 [15:01] sil2100, well, none of the images i tried on the CM3+ booted [15:02] sil2100, this is not u-boot ... this is the raspberry pi firmware [15:02] sil2100, it doesnt even get to a point where it would load u-boot, hangs in teh step before [15:03] (on the ranbow colored splash screen (which shoudl really be disabled, not sure why thats on in the official gadgets) [15:03] ogra: ok, I see what's the problem - you are right, the core16 gadgets didn't get the firmware bump [15:04] ogra: since we only bumped the uboot version there [15:04] right [15:04] ogra: let me do that then [15:04] thanks !! [15:04] ogra: but the 18 ones are bumped if anything [15:04] ETOOMANYGADGETREPOS [15:04] wried [15:04] Sorry about that! [15:04] *weird even [15:04] sil2100, i definitely tried the rpi3 UC18 images from /current [15:05] (as well as the (totally pointless) cm3 ones ... why do we build them ?) [15:06] sil2100, could it be that the fix hasnt made it to /current yet ? (i can surely try /pending too if needed) [15:10] hm hm hm, it looks correct, manifest shows having the latest pi gadget, and from what I see in the gadget's build log the right boot-firmware tag has been cloned [15:10] And I thought waveform checked if it's the right firmware version before submitting the PR [15:10] mvo: thanks [15:14] ijohnson: I pushed some bits, if you mind I can also push to a separate PR [15:14] no, it's fine you can push to the same PR, thanks! [15:14] ijohnson: mostly suggestions, probably need a critical eye from samuele as well but should make things a bit easier [15:14] mvo: ack, thanks for that [15:14] ijohnson: its mostly there now, something in the TestYesNetplanApply test is unhappy still, it hangs here [15:15] ijohnson: not sure why, probably some mocking missing for something [15:15] yeah I've seen it hang a few times I think it has something to do with locks [15:15] err I've seen it hang in a few iterations I've done [15:15] ijohnson: yeah, its probably not my changes :) [15:15] ijohnson: aha, locks! good catch, let me check those [15:15] the version I pushed up shouldn't hang, but maybe your changes are moving back to what I tried originally that was hanging [15:16] * mvo nods [15:20] sil2100, looking at https://github.com/snapcore/pi-gadget/blob/18-armhf/snapcraft.yaml i see a tag from feb 2019 ... perhaps thats still not new enough [15:22] ogra: maybe, but waveform selected that tag exactly because of cm3+ support [15:22] Let me poke him [15:25] ogra: anyway I'd expect it to work as we use the exact same firmware tag for disco and eoan classic raspi3 images, and I'm sure those work on the cm3+ [15:25] (also, per Dave's original commit message: https://github.com/snapcore/pi-gadget/commit/b8a16cb1e1d9a2c0419da8fef0bffc12e3c914fb ) [15:25] very weird ... [15:25] I asked him to test again [15:25] Maybe there's something weird going on somewhere [15:26] I'll wait for this to be resolved before I update the 16 gadgets, since I'd like it all to use the same firmware [15:26] When a new hook is added to a snap package (e.g. interface connection/disconnection hook), does that need to be approved by the store guys ? [15:27] om26er: I'm not sure that the store review tools have been updated to take into account interface connection/disconnection plugs [15:27] jdstrand and roadmr should be able to help you with that when they get time [15:27] sil2100, well https://github.com/snapcore/pi-gadget/commits/18-armhf shows that all travis checks have failed ... not sure if that tree is gated by travis though [15:27] (for both last commits) [15:28] ogra: well, it passed on the PRs [15:30] ogra: (when you check the failed travis checks, those are issues with git clone and I also experienced that during the snap builds) [15:30] And even locally [15:31] There was a period of time where the default git repo url of uboot that we use was not usable [15:31] Anyway, basically the travis checks are build-checks, so if LP was able to build the end snap, that means it passed all the tests [15:32] ijohnson thanks, I'll wait [15:32] ok ... as long as the sync to LP isnt blocked by these travis checks i guess it is fine [15:33] just looked like a potential reason for not having the change in these images [15:34] ogra: I checked the snap build logs, and I saw that the git clone of the firmware branch printed "Note: checking out '98997f363e3683ead4f50c37902169248628303a'.", which is a commit from feb 2019 [15:34] ok [15:34] very strange then [15:35] cwayne: hey! Do you guys have cm3+'s in the lab? [15:35] they do [15:35] om26er: can you show me (snapcraft.yaml) what you added? [15:35] They're on the way sil2100 [15:35] plars has one at home [15:35] cwayne: awesome \o/ [15:36] sil2100: have a new image I need to try? [15:38] roadmr https://github.com/deskconn/deskconn-server/blob/master/snap/snapcraft.yaml [15:41] om26er: hm this looks reasonable; I think the best way would be for you to change what you need, upload the snap, and the review tools will tell you if it needs manual approval [15:41] plars: could you check the current one? Since Dave seems to be AFK - just so that someone else also tries booting it http://cdimage.ubuntu.com/ubuntu-core/18/current/ <- [15:41] om26er: typically what we do is approve that in general, and subsequent uploads won't need the same thing [15:41] sil2100: will do [15:41] plars: thank you! [15:41] plars, i couldnt get that one to boot on my CM3+ ... [15:42] roadmr https://dashboard.snapcraft.io/snaps/deskconn/revisions/84/ [15:42] would be good to have another datapoint [15:42] I think revision 82 is the one that needs approval [15:43] This will be some good input, since I at least hope he did test that gadget change before submitting the PR [15:43] (Dave that is) [15:47] sil2100: we had tried an image with a gadget using Dave's pr and were able to boot core18 but not core16 [15:47] cwayne: yeah, the core16 is expected, because of uh, my mistake [15:47] Too many repos of those gadgets [15:48] cwayne: but the 18 gadget booted fine on the cm3+? [15:48] plars: ^ [15:48] Yeah that was my understanding [15:48] yes, still setting up to try this one though - I'm also trying to boot it with a regular cm3 in the lab [15:49] The regular ones should be good, since I saw those being tested in the automated testing [15:52] yeah, very unlikely that they could have regressed [15:52] ogra: did that rpiboot from edge work for you at least? seems to be working on my cm3+ [15:53] it's writing the image now [15:54] sil2100, oh ... note that i tested http://cdimage.ubuntu.com/ubuntu-core/18/stable/current/ is that different from http://cdimage.ubuntu.com/ubuntu-core/18/current/ ? [15:54] ogra: yes! [15:54] oh man [15:54] can we clean up that directory mess ? [15:54] ogra: we don't daily-build stable images, only during point releases [15:54] ogra: you need to check the timestamps, we only daily build edge and beta images right now [15:55] oh my ... ok [15:55] ogra: stable is like the stable ones we want to brand and 'release', which only happens on every point release [15:55] ;) [15:55] then sorry for the noise ... i followed some download link (that i have lost now :( ... ) [15:55] from our main website somewhere [15:56] http://cdimage.ubuntu.com/ubuntu-core/18/current/ are for edge and http://cdimage.ubuntu.com/ubuntu-core/18/beta/current/ are for beta - and I guess for a daily stable image you'd have to build one locally with ubuntu-image --channel=stable [15:56] It might be a decent idea to build some stable images before point releases [15:56] But doing it daily wouldn't make much sense, there's not enough velocity there [15:56] well, yeah, dail stable (or daily in general) is nonsense anyway ... the images shoudl simply be built change-triggered [15:57] So we'd probably switch some stable images built on every stable snap change, but that's still not there [15:57] Yeah, that's the plan [15:57] But we still don't have that [15:58] plars, cwayne: issue has been resolved, sorry for the commotion o/ [15:58] http://cdimage.ubuntu.com/ubuntu-core/18/stable/current/ doesnt really suggest it is anything thats daily though ... if i navigate from the toplevel of cdimage i indeed want the "stable " release ... and then go to the most current image of that by using the "current" link [15:59] so even if i had not follwed a download link originally,m i would have ended up in the exact same place to get the latest stable image [15:59] Well, the header of the stable images mentions 'Ubuntu Core 18', where others explicitly say 'Daily Builds' - but indeed, we might want to make this a bit more intuitive [16:00] ogra: well, this is the 'latest stable image' ;) [16:00] right ... but there are no headers at http://cdimage.ubuntu.com/ubuntu-core/18/ :) [16:00] ogra: as mentioned, we will only promote to 'current stable' on point releases, so people following download links will always get only the image we blessed during point release preparation [16:00] True [16:01] cdimage is a confusing place [16:01] ;p [16:01] and when i finally enter the site i have to search for the right image between the gazillio pi images on that page (is there any reason why we do not have one pi image per arch /armhf/amd64 ?) [16:01] sil2100: this boots on both my cm3 and cm3+, but it has edge snaps for everything. Is that expected? [16:02] ah, just saw the backlog [16:02] * ogra hands plars some https://www.youtube.com/watch?v=7nqcL0mjMjw [16:02] :) [16:03] hah [16:03] sil2100: so we are waiting on a different image to test now I guess? [16:04] plars: no no, it's all good - I mean, I'll be pushing new cm3 16 gadgets out soon, but other than that we're good [16:05] ack [16:07] sil2100, but in all seriousness, why do we have pi2, pi3 and cm3 images there ? supposedly all using the universal pi gadget and the same kernel, so in the end they are all the same image [16:08] this smells like a massive waste of built/test time and server space [16:09] (i originally developed the universal pi gadget to exactly not have three images anymore :) ) [16:10] ogra: well, it is a waste, yes, that was some leftover of the 16 times, no reason for it - guess when the first 18 models have been created and signed we simply cloned what we had for 16 [16:10] So we got pi2, pi3 and cm3 models, thus a separate image for each model [16:10] oh well [16:10] sil2100, yeah, for 16 we couldnt change ... i had hoped for 18 it would though [16:11] i guess i'll send some expensive tea to bribe xnox to make sure core20 gets better here ;) [16:11] ogra: a missed opportunity I guess! There was a bit too much going on so I also didn't notice it's unneeded until it was already all released ;) [16:11] yeah [16:12] ijohnson: please check netplan-apply - should be mostly good now, feel free to fix anything I missed and there is also a spread test missing iirc [16:12] mvo: thanks I just saw your push I was waiting to push up my spread test until you were done [16:12] well, who knows ... i'm not sure we can actually support the pi4 with the same kernel snaps in the end ... perhaps it was a clever thing to keep them separate [16:13] until we can go to a new enough kernel that supports all of them in core20 [16:13] ijohnson: yeah, I think I'm finished with what I wanted to do :) feel free to tweak/adjust as needed, its your PR afterall, I feel slightly bad for being so pushy in it :/ [16:14] ijohnson: but it was also fun so I couldn't stop myself [16:14] mvo: no problem, I appreciate the help in the tests that was what I was blocked on so you have unblocked me :-) [16:15] ijohnson: great! happy to help and to be fair the testing was a bit delicate (but hopefulyl we can make things better/easier :) [16:15] +1 [16:18] 7083 needs a bit more work, there or in a follow up [16:18] it has 2 +1 but I should probably look at it as well [16:18] I commented about the work [16:21] hmm, om26er is gone. ijohnson> fyi, so long as the hook is known to the review-tools at all, then there is no problem. if a new hook is added to snapd, then the review-tools need to learn it [16:22] ijohnson: I was discussing with mvo, a smallish thing that we might need sooner rather than later, related to remodeling is this: https://forum.snapcraft.io/t/remodeling-to-a-new-model/10477/2 [16:23] jdstrand: does the store tools know how to handle interface hooks? IIUC, those hooks could be named anything in the patterns shown on https://docs.snapcraft.io/interface-hooks [16:23] pedronis: looking [16:25] pedronis: I'm not up on remodeling and I don't see anything in that forum that references hooks (am I blind?) [16:26] jdstrand: not related to the discussion, but ijohnson has progress quite a bit on his current task, so I was discussing with mvo what he can pick up next [16:26] *progressed [16:26] *can or could [16:26] oh, heh, I copied the wrong link. durr [16:27] pedronis: okay this seems straight forward, but actually like jdstrand I'm not familiar with the current plan on remodeling :-) [16:27] pedronis: is that forum post all we have for the plan for remodeling ? I see you also have quite a few PR's open with remodelling label [16:28] ijohnson: we progressed quite a bit [16:28] that forum topic is mostly concerened with reregistration, switching model [16:28] ijohnson: I'm close to eod but we can chat more tomorrow [16:28] ijohnson: the tools already handle those, yes [16:29] ijohnson: (https://docs.snapcraft.io/interface-hooks) [16:29] pedronis: okay sure I will try to schedule a meeting tomorrow to discuss more [16:30] jdstrand: hmm okay well it looked like using an interface hook was the problem that om26er was running into [16:32] ijohnson: ah, the tools know about connect-*, prepare-* but not disconnect-* [16:33] I see that r82 uses disconnect-plug-... [16:33] * jdstrand adjusts tools [17:01] roadmr: hi! can you pull 20190716-1700UTC? not urgent. fixes om2er's issue (cc ijohnson) [18:52] jdstrand: sure thing, I'll queue it up === kyrofa_ is now known as kyrofa