[05:22] * zyga-x240 is tired, very bad night [06:19] morning [06:28] morning [07:00] morning [07:00] pstolowski: good morning! just fyi, I'm about to push some support thing around the import PR [07:00] good morning guys [07:00] pstolowski: so probably best to not also work on it right now or we risk conflicts [07:00] good morning zyga-x240 [07:01] mvo: hey [07:01] hey mborzecki ! good morning to you as well :) [07:01] how are you all? [07:02] mvo: hey, ack. i'll get back to it in the afternoon then, i'll focus on tests for cleanup on interrupted import [07:02] * zyga-x240 is pretty sleepy, lucy gave us a beating at night [07:03] pstolowski: we should probably have a quick chat, we discussed with samuele last night, there is this problem that the current import approach is racy, we don't need many changes but a little bit, give me 5min and I can push a sketch of what we discussed [07:03] mvo: ok [07:04] pstolowski: there is 9461,9462 and 9467 (all relatively small) in the meantime if you feel like it :) [07:08] sure [07:09] pstolowski: I wrote the idea down in https://github.com/snapcore/snapd/pull/9036#discussion_r499816703 now - hope it makes sense(?) [07:14] mvo: interesting points there. should we also delay our own restarts, or not start import if restart is pending? [07:15] mvo: i think that the comments i made in 9461 can be addressed in #9467 [07:15] and perhaps have the former closed actually [07:16] mborzecki: sure, probably too granular all of this, sorry for that :/ [07:16] ijohnson xnox I'm trying to build the 20-armhf (or 20-arm64) pi-gadget snap, fails both on a Focal amd64 machine as well as remote-build with arm* --build-on flag. Anything I'm missing for the 20 build, since README instructions seem to be for uc18? [07:16] mborzecki, pstolowski thanks for the reviews! [07:17] mvo: I commented on that idea [07:17] just drive by comment [07:17] mvo: pstolowski: fwiw, can we actually detect the real snapshot size when reading the body? what if someone appends just some random content at the end after the legitimate zip data is finished? [07:21] mborzecki: we have the content-length and the limited reader now, that should help or am I misunderstanding? [07:25] mvo: hm maybe i'm complicating this a bit, what if i have a 1k snapshot, send content-length: 1G, and send the fill the rest with garbage? [07:29] mborzecki: i believe you need to be root to import. we do check if zip is valid, plus we will have disk space check there [07:30] or am i misunderstanding the concern here? [07:30] mborzecki: it will error when reading the tar content but yeah, in general we can not protect much about bad actors here. the content-length is mostly to ensure we can do size checks and that we protect against accidents [07:31] mvo: otoh as pstolowski mentioned, you need to be root, so there's easier ways to break stuff [07:32] brb [07:39] mborzecki: yeah [07:51] pstolowski, pedronis how does https://github.com/snapcore/snapd/compare/master...mvo5:snapshot-import-3?expand=1 look ? this should make the import race-free (at least on the filesystem level) [07:51] pstolowski, pedronis also (to me but I'm biased) slightly easier to read(?) [07:52] (tests need updating still but integration works) [07:52] mvo: the url seems broken, wants me to open a PR [07:53] pstolowski: sorry! is this one better? https://github.com/snapcore/snapd/compare/master...mvo5:snapshot-import-3 [07:54] https://github.com/snapcore/snapd/compare/master...mvo5:snapshot-import-3 [07:54] mvo: did you see zyga comment, not that I understand it [07:56] pedronis: I did, not sure how this helps against reboots, but once he is back I will ask him [07:56] yes, that's the same question I have, also not clear what we would lock [07:57] pedronis: yeah, we could lock the $setid_importing file but not sure about the gain [07:59] anyway checking for hashes will add more questions as well [08:01] pedronis: right, which ones to be exact? if we have hashes in export.json we can validate them after line 582. or am I missing something? [08:02] mvo: we'll need to do some locking and put some data in importing to avoid importing the same thing twice if asked in quick succession [08:02] mvo: not sure, it shows all the changes against master i think [08:03] pedronis: oh, interessting. yes! [08:03] anyway we are not there yet [08:03] pstolowski: oh, right, let me try to compare against just the old pr if that helps? [08:04] mvo: nah, i think i'll just look at entire thing again [08:04] pstolowski: might actually be easier [08:26] mvo: i think i'll hold off from touching import PR for now, it's moving too much [08:28] 2020-10-06 08:03:25 Cannot allocate google:centos-8-64: cannot allocate new Google server google:centos-8-64 hmm (oct060800-322222): timeout waiting for google:centos-8-64 (oct060800-322222) to provision [08:37] pstolowski: sorry for that, it should calm down very soon I think [08:37] zyga-x240: https://developers.redhat.com/blog/2020/09/24/new-c-features-in-gcc-10/?sc_cid=7013a00000260vRAAQ [08:39] mvo: no worries [08:39] mvo: i've fun with services now and that issue from the forum ;) [08:40] mborzecki: nothing really interesting [08:40] mborzecki: C++ is a sprawling mess [08:40] pstolowski: ha! great. [08:40] * zyga-x240 keeps hacking [08:41] there was a bit of worry as city-wide alarm sirens were on for like 10 minutes [08:41] fortunately it's just a test [08:41] but man, a bit nervous for a moment [08:41] *those are LOUD* [08:41] Is cloud-init still unsupported/buggy when building a core20 image, or does ubuntu-image need an update after https://github.com/CanonicalLtd/ubuntu-image/pull/186? I have some additional setup for my images in a cloud.conf that can't be reliably replicated through gadget or other supported mechanisms. [08:41] bombing run alarm test [08:42] 9462 is an easy win [08:42] very small [08:43] mvo: pedronis: https://github.com/snapcore/pc-amd64-gadget/pull/50 [08:44] +1 [08:45] dot-tobias: depending what kind of model grade you are building, you'll have to put the cloud-init config in the gadget itself [08:50] pedronis: Thanks, re-read https://core.docs.ubuntu.com/en/reference/gadget at Setup files. Don't remember why I didn't go that route initially, but it's been > 1 year so why not try again. Could you please elaborate what you mean by “model grade”? 😊 [08:52] dot-tobias: https://core.docs.ubuntu.com/en/releases/uc20 [08:52] under Model definition [08:54] pedronis: Thank you very much! Overlooked this page, I knew there was a new way to specify the channel for required snaps now … [12:17] ijohnson: Re https://bugs.launchpad.net/snapd/+bug/1898622 – can you give me a hint where/which logs to check to see if this happens in my case as well? I have a local uc20 gadget snap (i.e. model grade set to “dangerous” + passing local snap via --snaps) and the image boots on a Pi 4, but only with a blinking cursor like the thread opener in https://forum.snapcraft.io/t/ubuntu-core-20-hangs-with-added-snap/20342. [12:21] hi dot-tobias [12:22] I see you had a couple questions earlier, did you get your cloud-init question answered firstly? [12:22] and secondly, re building the 20 pi-gadget, what architecture are you trying to build for and what's the host system where you are trying to build the gadget? [12:24] hi ijohnson, sorry for brevity without proper thanks and please 😊 First: [12:25] haha no worries [12:25] Re cloud-init: Kind of, I placed it in the gadget but can't test if it works because I first need to have it boot properly 😄 Like https://forum.snapcraft.io/t/how-to-preconfigure-custom-image/4154/15?u=tobias describes, at some point way back, cloud.conf in the gadget wasn't applied – at least in my testing. Feeding it to ubuntu-image worked. [12:26] dot-tobias: for uc20 specifically the inverse is now true, feeding to ubuntu-image no longer works, but putting it in your gadget should work, we have tests for this, but in all honesty I haven't tested it for a couple weeks now so maybe something has broken again on this front [12:27] Second, the pi20-gadget: I'm trying to build for armhf, I applied your patch from https://github.com/snapcore/pi-gadget/pull/43 and it builds just fine on an amd64 host. Then I build the image with ubuntu-image and flash to an SD. The Pi 4 seems to boot, but only to a blinking cursor on the screen. [12:29] ijohnson: Will let you know if cloud.conf inside gadget snaps works for me as soon as I have the system booting properly 😄 [12:31] dot-tobias: ok, great happy to hear that patch works, I need to go back and address the comments on that pr before it can be merged but still good to know it works :-) [12:31] ijohnson: Unfortunately, I have to hurry to pick up my daughter at kindergarden. May I DM you on the snapcraft forum, because my IRC bouncer isn't as reliable as I would like it to be? [12:31] dot-tobias: yes of course [12:31] I'll wait to hear from you there then [12:31] s/at/from/i 😄 Great, thanks in advance ijohnson! [13:04] ijohnson: standup or are you at the roadmap review? [13:04] mborzecki: I'm watching the roadmap [13:04] * zyga-x240 is on roadmap review [14:30] cachio: have you seen the journal-state test fail like this: https://pastebin.ubuntu.com/p/HgxDDBNXgd/ [14:30] ? [14:31] the code looks rather odd `test 0 -eq 1` will never be true no matter how many iterations are run ... [14:31] ijohnson, no, weird [14:31] let me check if I can reproduce it [14:32] ijohnson, just happening in ubuntu 20.10? [14:32] cachio: yes that's where I just saw it fail [14:32] cachio: also did you see my update in the su doc about uc20-recovery ? [14:33] I can reproduce it now like 80+% of the time with your qemu cmdline but 0% with my qemu cmdline [14:33] yes [14:33] it's very odd and confusing and frustrating [14:33] could be associated to "no secure boot" [14:33] ? [14:34] I don't think so because I changed my cmdline to boot without secure boot and still couldn't reproduce it [14:44] ijohnson, I could reproduce the error on jounal-state test [14:45] cachio: interesting, what's the issue? [14:47] ijohnson, don't know yet [14:47] just failed [14:47] well good that you can reproduce it, that's always better than not being able to reproduce it :-) [14:49] ijohnson, well, seems to be that I used incorrectly the retry [14:50] ijohnson, I'll fix it and create a PR [14:58] nice thanks [15:04] * zyga goes for PT [15:33] * cachio lunch [15:51] cachio: I just saw the journal-state test fail again this time on uc20: https://pastebin.ubuntu.com/p/xKPMcypjDq/ is it the same thing ? [16:11] pedronis: 9469 is the snapshot work as discussed last night, still some gaps in the tests but should be without races === King_InuYasha is now known as Conan_Kudo === Conan_Kudo is now known as King_InuYasha [17:38] cachio: I think I finally figured out the uc20-recovery problem with your qemu command line args, or at least some kind of explanation [17:39] cachio: your command line has this: `-net nic,model=virtio -net user,hostfwd=tcp::8022-:22` while mine has `-netdev user,id=mynet0,hostfwd=tcp::8026-:22 -device virtio-net-pci,netdev=mynet0` [17:39] I don't know precisely what the difference in behavior is, but if I use your qemu command line net args the test fails, but with mine it passes [18:03] ijohnson, nice [18:03] I'll update my docs to start using yours [18:04] thanks for the research [18:20] back from pT [18:26] hey zyga [18:26] hey! [18:26] I gave a +1 to your notifications pr last night, it looks good! [18:27] ijohnson thank you, I will iterate to add some tests and then ask macieck for a re-review [18:27] I think he and mvo asked for tests on IRC [18:27] I started and am about halfway through the biggest file on your udev pr too [18:27] ijohnson woot! [18:27] it's a bit tedious [18:27] but I tried to structure the boring parts to be readable [18:27] it is a bit tedious and takes some close eyes for sure [18:27] thank you so much! that's my oldest really useful PR that has yet to land [18:27] (well, others are older but this is ready IMO) [18:28] I'm working on spread tests for inhibition locks now, just got back and noticed the test failed, but it worked interactively in spread -debug shell [18:28] must be missing something simple :) [18:28] simple missing things are the best missing things imho [18:35] I got tea and am digging into things :) [18:43] weird [18:43] my other PC disconnected [18:43] hmm [18:44] back [18:44] :) [19:01] hmm, nothing looking out of the ordinary, trying -debug [19:14] DOH [19:14] ijohnson: + snap install --classic SNAP_NAME [19:14] what could be wrong ;D [19:14] haha [19:14] SNAP_NAME [19:14] classic [19:14] not enough $ [19:14] ;D [19:14] haha good one [19:15] I was trying to come up with a way to get some measurements done while a compound operation like "snap refresh" runs [19:15] and I ended up using inotify in tests [19:15] I think that's a first for us [19:16] yeah all I see is a comment from Maciej about arch linux in prepare.sh about d-bus needing to inotify on some path [19:16] I will break out a few helpers [19:17] yeah, that's related to the fact that dbus uses inotify to detect .service files [19:17] but this only works if the relevant directory exists at the time dbus itself starts [19:17] so we created it so that it works on first install IIRC [19:17] inotify is a pretty poor API [19:17] yep that's exactly what the comment says :-) [19:17] good memory! [19:18] haha, this time [19:18] I forget many many things now [19:19] ijohnson: did you see the nvidia ai video conferencing thing? [19:19] zyga-x240: no I don't think so [19:20] https://www.youtube.com/watch?v=NqmMnjJ6GEg [19:20] pretty interesting IMO, would love to see this in practice [19:22] yeah that's really cool [19:24] I wonder how well it works with a non-static background though [19:24] I suspect it needs more keyframes with real image [19:24] yeah [19:25] but it's an interesting evolution of "motion vectors" and other classic video tricks [19:25] cannot wait for voice equivalent [19:25] bi-directional audio to text to audio codec [19:25] with emotions :) [19:26] maybe irc and mumble could merge [19:26] ah that's interesting, but I imagine very difficult to get right but without lots of data on individual users which is a bit of a privacy concern [19:27] like imagine if this service can impersonate the voice of someone you know / trust [19:27] with arbitrary text [19:27] yeah [19:27] I think in the age of deep fakes we are really there already [19:27] this just makes it mass market [19:28] true I guess I just haven't seen that for audio yet [19:28] (or so the AI overlord told me to type ;-) [19:28] deepfake video is definitely a thing which I guess implies that audio is already there and maybe it just wasn't scary enough to make headlines the way video did [19:28] ijohnson: there's a bunch of "obama speaks anything" videos on YT, given that he's got lots of public domain recordings available [19:29] no need to do trump variant though, he really speaks anything just like that ;-) [19:29] * zyga-x240 refrains from more political joes [19:29] *jokes [19:30] look you just made one! [19:30] while trying to refrain from making them [19:30] * ijohnson goes away and tries to debug pi things [19:30] haha [19:58] so the test was useful [19:59] I broke something in snapstate, checking [19:59] I mean, in my branch [19:59] it's just refactoring to make it fit the model [20:18] DOh [20:18] nothing is broken [20:18] I patched the test in the wrong spot [20:18] * zyga-x240 adjusts [20:20] zyga-x240: could you +1 9378? I addressed your feedback and you said your review was a +0.9 :-) [20:20] looking [20:20] cachio: also could you take a look again at #9378 ? [20:20] would be great to land this finally [20:21] done, thanks for changing @HOLD... [20:21] thank you! [20:22] cachio: btw, I realized that my comment about prepare/restore on nested helpers may have been confusing [20:22] I didn't imply that we should not have prepare/restore [20:23] but that they are unclear in the context of explicit resource management performed by other commands [20:23] I'm sure all those functions do something useful [20:23] ijohnson, sure [20:23] I was trying to understand their function and see if any of that functionality should shift from one command to another [20:23] zyga-x240, ah, ok [20:24] zyga-x240, I'll go back to that PR once I finish the current one [20:24] ok [20:26] zyga-x240, you are talking about the nested one, right? [20:26] yes [20:27] cachio: btw, did you know about $'foo' in bash? [20:27] bo [20:27] no [20:27] zyga-x240, what? [20:27] I found out today [20:27] echo $'hello\n $world' [20:28] what does that do [20:28] handles specific \ sequences [20:28] acts as ' ' otherwise [20:28] I also found [20:28] * ijohnson is ready to run for the hills at mention of more bashisms [20:28] ${foo@Q} [20:28] which is genuinely very useful [20:28] it expands foo as variable and quotes correctly for bash to read again [20:28] the @xxx syntax has a few more things it can do [20:29] but this one is most practical for our stuff [20:29] as it would let us quote things and pass to nested bash easily [20:29] anyway [20:29] oooh that is really cool [20:29] just more ladmines [20:29] in the no-mans bash land [20:29] just those pop candy [20:29] not your leg off [20:30] zyga-x240, cool [20:30] zyga-x240, how did you discover that? [20:30] reading docs? [20:31] cachio: by reading twitter and reading bash to correct an almost-useful reference [20:31] and finding more [20:31] "marry has a little lamb" [20:31] !:q [20:31] run that in a bash [20:31] it's another quoting mechanism [20:32] but only for interactive job [20:32] ${foo@Q} is generic [20:33] zyga-x240, nice [20:33] learning bash in twitter hehehe [20:34] it is cool [20:35] * zyga-x240 should get green pass now [20:37] * zyga-x240 goes to brush his teeth [20:37] cachio: I wrote a small add-on to tests.cleanup [20:37] tests.cleanup pop [20:37] it's useful when you do something [20:37] install cleanup via defer [20:38] want to clean up on failure (which runs in restore) [20:38] zyga-x240, awesome [20:38] but also want to clean up early on the happy code path [20:38] so that you don't have to repeat the "undo" logic in any way [20:38] anyway, I'll clean that up (add tests) and propose tomorrow [20:39] today I'm just going to see this inhibition lock test pass on 18.04 and 20.04 and call it a day [20:54] OUCH [20:54] I was cleaning the kitchen [20:55] and we routinely cut milk boxes in half to clean them (those are recycled and get collected twice a month) [20:55] and cut right through my thumb nail and good part of thumb [20:55] so... [20:55] * zyga-x240 EODs [20:55] but I got x3 pass [20:55] so some good in the ene [20:55] *end [20:56] * zyga-x240 pushed handlers changes, more tomorrow [20:57] need to find some bandage [20:57] good that I don't use my right thumb for typing that often [20:57] my left thumb is space bar [20:57] right.... dunno [20:57] maybe sometimes space