[00:07] uh why are my uploads to the store from launchpad failing? [00:07] eg. https://code.launchpad.net/~mwhudson/+snap/go110/+build/405061 [00:23] PR snapcraft#2427 closed: lifecycle: query for multipass install on darwin [06:21] morning [06:32] PR snapd#6299 closed: travis: short circuit failures in static and unit tests travis job

[06:54] PR snapd#6307 opened: systemd/systemd.go: add missing tests for systemd.IsActive [07:02] PR snapd#6308 opened: release: 2.36.3 [07:03] zyga: 6308 has some interessting bits, would love to understand where the code diff came from [07:16] Hello [07:17] This is because of different approach to fix a bug in master and in the release branch [07:17] I will review it carefully, the master version should be used most likely [07:28] zyga: mvo: hi === pstolowski|afk is now known as pstolowski [08:02] morning [08:16] zyga: ok, just push to the PR if you are happy [08:16] zyga: I remember now I think :) [08:16] mborzecki: hey, good morning [08:17] mvo: oh, still working today? [08:17] pstolowski: fire is not fully over :( [08:18] pstolowski: cloud-init on pi and dragon is acting up, i.e. it runs when it should not run and console-conf does not work correctly too [08:19] pstolowski: hopefully sil2100 (and foundations) can help with these issues [08:21] mvo: :( [08:22] pstolowski: it is what it is, I will just take 2 days off in jan [08:22] Yeah, looking into that as much as I can [08:22] sil2100: \o/ [08:23] sil2100: all the details are in the mail I send out ~2h ago or so (I hope, if anything is missing, please do let me know) [08:32] Good morning [08:33] zyga good morning :) [08:33] Some patches required [08:34] mvo: as per my e-mail, could you attach ds-identify.log from the cloud-init logs? [08:41] mvo: anyway, I'm building an image for my pi3 now and I'll test it here as well [08:45] sil2100: will do, mail goes out in 2min [08:46] mvo: hello [08:49] sil2100: you have mail [08:49] sil2100: the pi has booted now so things should be quicker now [08:49] pedronis: good morning [08:50] mvo: so cloud-init on arm devices thinks to be on a cloud, that is annyoing/strange, strange because it wasn't doing that with 16 [08:50] otoh is probably a newer cloud-init [08:50] pedronis: yeah, different version and +100 for annoying [08:51] pedronis: also console-conf is acting up and the IP keeps changing for me so not a great experience [08:51] ogra: hey, do you happen to know why the pi3 with 4.15 might switch IP addresses on reboot? (context is UC18 image) [08:52] mvo: so it think maybe it's an OpenStack [08:53] pedronis: can you elaborate? what makes it think that? is that in the ds-identify log? [08:53] yes [08:53] pedronis: heh, I see [08:54] pedronis: it does not have more details about the why it seems [08:54] yea, we need to stare at the code [08:56] pedronis: I just did and had a heart attack [08:56] pedronis: let me find a link [08:56] pedronis: it looks like we get the maybe because https://github.com/number5/cloud-init/blob/master/tools/ds-identify#L960 [08:58] wonderful [08:59] check for 'OpenStack' returned maybe [08:59] Oh yeah [09:00] yes, just verfied that code path is taken [09:00] so I guess cloud init does not run much on non-x86 ;) [09:00] * mvo weeps a bit in the corner [09:00] mvo: but we had the same thing in core16, right? [09:00] Is cloud-init in xenial different? [09:01] ppisati: hey, do you happen to know why the pi3 with 4.15 might switch IP addresses on reboot? (context is UC18 image) [09:03] no set -e in ds-identify, thats an interessting choice [09:03] anyway [09:04] mvo: sil2100: does ubuntu-image write some cloud-init conf [09:04] ? [09:05] pedronis: not if it's not asked to, you can explicitly add one if you need to though [09:05] mvo: sil2000: it seems quickest fix not touching cloud-init would be to configure ds-identify with maybe=none if arch is not x86 [09:07] pedronis: yeah, forcing cloud-init to be disabled via ubuntu-image is probably the quickest path forward. iirc --cloud-init can already be passed to u-i so we just need to craft a no-cloud config for it [09:07] mvo: not disabled [09:07] pedronis: aha, just maybe=none ? ok [09:07] ds-identify [09:07] has a different config [09:07] yes, maybe=none [09:07] either everywhere or only arm [09:07] ta, I was not aware of this [09:08] zyga: can you do another pass on https://github.com/snapcore/snapd/pull/6265 ? [09:08] PR #6265: cmd/snap: attempt to restore SELinux context of snap user directories [09:09] mvo: there's some doc at the top of it, it mentions /etc/cloud/ds-identify.cfg and the kernel cmdline [09:13] pedronis: thanks, playing with this now [09:13] pedronis: I also added a trello card UC18 release [09:13] pedronis: that lists the bugs [09:13] pedronis: or open issues [09:13] thx [09:19] $ cat /etc/cloud/ds-identify.cfg [09:19] policy: search,found=all,maybe=none,notfound=disabled [09:19] pedronis, sil2100 ^- this has helped and no cloud-init anymore [09:19] pedronis, sil2100 with that I also have a stable IP it seems [09:19] mborzecki: hey, if you get to reviewing hotplug PRs, then #6100 and #6114 would be the first to look at; thanks in advance! [09:19] PR #6100: overlord/ifacestate: hotplug-remove-slot task handler [09:19] PR #6114: overlord/ifacestate: handler for hotplug-disconnect task [09:19] pstolowski: ack [09:20] and i've just de-conflicted them [09:20] i'm doing 6114 now [09:20] thanks [09:21] hm need to grab some coffee [09:22] mvo: do you have cloud-init logs from when it was causing problems? [09:22] mvo: btw. from the failed run you had, could you upload the cloud-init log tarball somewhere? [09:23] mwhudson: cloud init logs for the console-conf setup? or just where it thinks its in a cloud? [09:23] mvo: one where set-name ended up in places it shouldn't [09:24] mwhudson: I can probably do that but need to rewrite the sd card and start from scratch, this is also tricky because cloud-init keeps the logs in /run but I cannot login with the broken network [09:25] mvo: ugh [09:25] mwhudson: so far I can share with you mvo's cloud-init-generator and ds-identify logs [09:25] debug shell time? [09:25] mwhudson: yes, just annoying to add in uboot, I struggle with this everytime because I never remember how to do it correctly :( [09:26] everyone loves uboot [09:26] mwhudson: but I give it a poke in some minutes, just trying to debug another failure on this pi [09:26] mwhudson: hahaha [09:26] ok [09:26] mvo: basically cloud-init does have code that will write netplan with set-name in it [09:26] mborzecki: reviewed [09:26] i don't at all understand how it could be triggered in this context [09:27] but i also have no other ideas how else it could get there [09:27] wait [09:27] "cloud-init keeps the logs in /run" [09:27] is that a core specific thing? [09:27] break in the initramfs and mount /run as a real partition? :) [09:28] (systemd might get angry if you do that i guess) [09:28] mwhudson: to break I need to add a kernel comandline and then I might as well add a debug shell. oh well, I think I will do that [09:28] oh yeah [09:28] mwhudson: maybe its in /var/log as well, I will check [09:29] it certainly is in classic [09:29] mwhudson: pretty sure that the generator log is only in /run but I guess thats not relevant for you, right? [09:29] yeah no i want cloud-init.log and cloud-init-output.log [09:29] Didn't see anything in /var/log [09:29] mwhudson: cool, if thats the case then I can help sooner [09:30] I had to do a cloud-init collect-logs previously [09:30] /var/log/cloud-init.log is empty for me [09:31] damn gnome-shell crashing on each unlock [09:32] sil2100: hmm [09:32] sil2100: syslog? [09:32] zyga: thanks for the review, i didn't think about adding a selinux backend, but maybe we should once the cleanups are in [09:34] mwhudson: no syslog + eek, persistent journal not enabled by default [09:35] sil2100: i am not surprised :/ [09:35] (persistent journal vs sd cards?) [09:35] Let me re-flash and enable persistent journal [09:38] mvo: I pushed to https://github.com/snapcore/snapd/pull/6308 [09:38] PR #6308: release: 2.36.3 [09:38] the diff isn't 0 but it is as close as it should [09:38] yes, the release branch had one more patch that was in the chain of fixing system key issue [09:38] it was because my initial branch got closed and reshuffled and I lost track of that [09:39] anyone interested in 2nd review on https://github.com/snapcore/snapd/pull/6288 ? [09:40] PR #6288: cmd/snap-confine: refactor call to snap-update-ns --user-mounts [09:41] mvo: do you know what is the status of SRUs into bionic? [09:41] 2.35.5 is in proposed [09:41] 2.34.2 is in main/updates [09:42] zyga I think there were issues with autopkgtest [09:42] aha [09:43] real or just noise? [09:43] zyga: I did not dig into them, iirc autopkgtest killed our tests because they ran too long [09:43] uhhh [09:43] anyway, it's probably not super urgent but it would be good to update the classic package [09:43] zyga I'm not sure what the latest status is though [09:43] and as soon as 2.36.x is out, do that as well [09:43] zyga absolutely [09:43] mvo: no worries, one thing at a time [09:43] mvo: :) [09:43] zyga its a mess - I think we need to disable adt again, its just not working in our cloud :( [09:44] for context, I'm asking because of this bug last evening: https://bugs.launchpad.net/snapd/+bug/1806070 [09:44] Bug #1806070: snapd.seeded.service never completes preventing full boot to default target

[09:44] the package in the archive is preventing snapd seeding and refreshing in some cases [09:44] zyga thanks! I remember we talked about this [09:44] * zyga looks at the snow outside [09:44] brr [09:44] I miss my old computer [09:44] the one that doubled as room heater [09:45] mvo: does the ethernet mac change? [09:45] ppisati: it's just the IP, I guess it's all good from the kernel side, it's more like a cloud-init thing [09:46] ppisati: yeah, sorry for the noise, it stopped for me AFAICT once I disabled cloud-init [09:46] ppisati: which should not have run in the first place but that is a different story. [09:46] ack [09:47] mvo: i think i have more or less convinced myself that the set-name comes from cloud-init [09:47] mvo: and now i am going to bed [09:47] * mwhudson zzzz [09:47] Indeed, mwhudson thanks! [09:48] mwhudson: sleep well [09:48] mwhudson: and thank you! [09:52] mvo: we should be careful with the notfound value in that config, it was enabled I think in your original log, not sure where that comes from [09:55] pedronis: I don't know what enabled it originally, iirc /etc/cloud is empty [09:56] mvo: I mean, I'm not sure what happens if it's set to disabled and there's cloud init config [09:56] or somebody attaches a disk some of that [09:56] it was working on 16 [10:00] pedronis: oh, ok. I suspect that 16 has a different ds-identify [10:02] Ugh, indeed [10:02] ds-identify-behavior-xenial.patch [10:03] sil2100: oh, interessting - can you pastebin that one? [10:03] Wait, one moment, this doesn't look right [10:07] mvo: 16 had also this: https://github.com/snapcore/core/blob/master/live-build/hooks/40-cloud-init-run-after-networkd-wait-online.chroot [10:08] probably not relevant given that in 18 networkd is used in all server images? [10:08] sil2100: one more interessting observation - even when I run console-conf I get a prompt to run it again instead of the usual "here is the machine IP and how you log in". but could be an artifcat [10:08] pedronis: that is the default now unless I very much misremember [10:09] pedronis: I remember looking at this patch and if we still need it [10:09] (I might be wrong though and can double check in a wee bit) [10:10] mborzecki https://bugzilla.redhat.com/show_bug.cgi?id=1648733 [10:11] heh [10:13] mvo: I read dsidentify, nocloud is a kind of cloud, either the things ubuntu-image writes or a properly labeled fs shoild make dsidentify detect that [10:13] so that's not a notfound case afaict [10:13] pedronis: hm, ok [10:14] see dscheck_NoCloud [10:14] Chipaca: heh, this is fun https://bugs.launchpad.net/snapd/+bug/1808450 [10:14] still wondering where the notfound=enabled comes from [10:14] Chipaca: %q needs to be localised :) [10:14] shouldn't be the default [10:14] Bug #1808450: Strings with placeholders don’t include quotation marks so they can’t be changed per locale [10:14] zyga: hrm? [10:15] pedronis: interessting [10:15] zyga: yeah, no [10:15] zyga: somebody's being very ornery there [10:15] well, I kind of agree [10:15] pedronis: I think it is the default, when you check ds-identify's DI_DEFAULT_POLICY_NO_DMI I guess? [10:15] snapd would not get a passing grade from my polish tearcher [10:15] teacher [10:16] sil2100: ah [10:16] pedronis: left some benchmarks in https://github.com/snapcore/snapd/pull/6265#discussion_r241703113 it's a vm running as a snapshot, though i took care to drop the caches each time, i'll see if i can get some measurements where the are actual writes to a spin drive [10:17] PR #6265: cmd/snap: attempt to restore SELinux context of snap user directories [10:17] fwiw the numbers aren't terrible, 27s for 600k small files, better than fontconfig [10:20] mborzecki: this sound like we don't want to do it inconditionally, no? [10:21] mvo: anyway afaik we can just set maybe=none and the parsing will use the defaults for the rest [10:21] s/afaik/afaict/ [10:21] pedronis: cool, I can try in a bit but right now I am running the testsuite [10:22] pedronis: the pr was already updated to do it only when the ~/snap (the dir) is not using the default label [10:22] pedronis: it looks like all the issues cachio had with running the arm tests are from cloud-init being enabled, so far I ran the core18 specific tests and all green, main is running now as well [10:22] good [10:23] pedronis: I'm sure sil2100 will find a good config and a way to get it into the system and then this fire is hopefully also done [10:25] zyga: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1808450/comments/1 [10:25] Bug #1808450: Strings with placeholders don’t include quotation marks so they can’t be changed per locale [10:25] mvo: sil2100: we can have that file in core18 itself? [10:25] that file = ds-identify.cfg [10:26] zyga: I mean, I agree, except in Spanish guillemots are very much optional / recommended-but-not-required [10:26] I agree about the low priority [10:26] zyga: and the 2nd preferred quote symbols are … “” [10:26] pedronis: so I see that in core 16 we had /etc/cloud/ds-identify.cfg [10:26] With policy: search,found=first,maybe=none,notfound=disabled [10:26] zyga: to which " is a close enough approximation for now [10:26] Trying to figure out where this file is coming from [10:26] * zyga is happy that he can at least read loads of spanish [10:27] sil2100: where did this come from? did u-i put it there ? [10:29] mvo: that's a good question, trying to figure it out - u-i doesn't really have mechanisms for arbitrary files in /etc [10:29] spread jobs hitting travis timeouts? [10:29] mvo: huh [10:29] mvo: it's in the core snap [10:30] mvo: not sure what's adding it, but if you unsquashfs core you can see the file there - I didn't see it created in any hooks so far, cloud-init itself also doesn't seem to ship it (at least as a file, maybe through maintainer scripts?) [10:32] sil2100: oh, run [10:32] sil2100: eh, fun [10:32] mvo: sil2100: fascinating [10:32] sil2100: ok, we can put it back if that fixes all the issues [10:32] anyway means doing the same for 18 wouldn't be too crazy :) [10:32] pedronis: :) [10:33] Yeah, sounds like the best way ;) I'll try to figure out where it's created out of curiosity anyway [10:33] But this would fix "all the issues" [10:33] sil2100: I think ubuntu-core-config created this [10:33] sil2100: give me a sec [10:33] mvo: oh, that's something new! [10:33] I saw it in the image PPA once but didn't know it was used [10:33] Sweet [10:33] sil2100: we no longer use it [10:34] sil2100: its a pain [10:34] core build is so byzantine [10:34] sil2100: I mean, the configs are split between gits [10:34] sil2100: so we fixed this in the core18 build but sometimes (like now) we still suffer [10:34] pedronis: indeed [10:34] sil2100: fwiw https://github.com/snapcore/core-build/tree/master/config/etc/cloud [10:35] ah [10:35] we have two core related repos? [10:36] pedronis: yes, sadly - core-build and core [10:37] At least it's all known now \o/ [10:37] PR core18#105 opened: static: add ds-identify.cfg back [10:37] mborzecki: when you a little bit of time could you look at https://github.com/snapcore/snapd/pull/5712 , it's long lived but now I finally got to make it green, had to adjust for parallel installs changes [10:37] PR #5712: overlord: make InstallMany work like UpdateMany, issuing a single request to get candidates [10:37] pedronis: ack [10:38] mvo: that config tree seems a bit in the wrong repo to me [10:38] pedronis: you mean core-build? [10:38] yea [10:38] if it's basically content of the thing [10:38] pedronis: this is why I created https://github.com/snapcore/core/pull/83 [10:38] it should go into core somehow [10:39] PR core#83: move most of the ubuntu-core config deb into the snap snap build [10:39] ah [10:39] pedronis: and then it did not get merged /o\ [10:39] eeek [10:39] * mvo weeps harder in the corner [10:40] it seems definitely a good idea [10:42] pedronis: I added a card for myself [10:42] mvo: SHIP IT! [10:43] mvo: sil2100: there's some other bits of cloud/ that might be relevant there btw [10:43] I suppose the resize bit is done differently now though [10:44] or is that done for clouds? [10:44] and we need it? [10:46] pedronis: we ship https://github.com/snapcore/core18/blob/master/static/etc/cloud/cloud.cfg.d/10_snappy.cfg (ironically) [10:47] very [10:47] pedronis: which is iirc the same as in uc16 - or am I misunderstanding [10:47] yes [10:47] it's ironic indeed because without cloud-init it would have done nothing [10:48] pedronis: :( [10:48] pedronis: so this is why the maybe become a yes? [10:49] ? [10:49] pedronis: what I meant its ironic that we only had half the config. but you indicate that if we had no config at all cloud-init would not have bothered to run? [10:53] mvo: sil2100: so it seems we need to port more bits from 16 to static/etc/cloud [10:54] pedronis: what else is missing? [10:54] there's a bit about azure it seems [10:55] pedronis: pretty sure that is obsolete but worth checking with someone from the cloud team [10:55] https://github.com/snapcore/core-build/blob/master/config/etc/cloud/cloud.cfg.d/99-snappy-azure.cfg [10:55] sil2100: can you check if we need this ^ or not anymore ? [10:56] mvo: ok, probably easiest for sil2100 to do that [10:56] PR core18#105 closed: static: add ds-identify.cfg back [10:56] building a new core18 with the right ds-identify.cfg now [10:59] * mvo also builds 2.36.3 for regular core beta [11:01] PR snapd#6296 closed: tests: new backend used to run upgrade test suite [11:04] sil2100: are you looking into why core18 CI is red? seems related to the checks for package removal [11:09] mvo: is new snapd okay but we need changes to core18 or will you do another snapd release to fix the cloud issues? [11:10] zyga: the last changes are all just in core18 afaiu [11:10] great [11:12] 2.36.3 is now in beta core everything but arm64 [11:12] (the later is still building) [11:13] pedronis: I didn't get to fixing them, but I confirmed that they're not real issues - the test env needs fixing, it's on my plate for ASAP [11:19] PR snapd#6309 opened: overlord/ifacestate: addHotplugSeqWaitTask helper [11:22] zyga: ^ if you have a moment [11:22] yep saw it [11:30] +1 [11:30] zyga: +1 on 6288, but plese see the comment [11:32] pstolowski: applied! [11:33] ty [11:39] * pstolowski lunch [11:42] mborzecki: something to garden https://bugzilla.redhat.com/buglist.cgi?component=snapd&product=Fedora [11:43] make sure to select *all* bugs [11:44] PR snapd#6310 opened: interfaces: return security setup errors [11:44] mvo: found the patch that didn't make it into master https://github.com/snapcore/snapd/pull/6310 [11:44] PR #6310: interfaces: return security setup errors [11:54] zyga: ta [11:56] I need to go to the boys' school for 2pm, meaning I'm not going to be here for the standup (although I should be back by 2.30 so I might see y'all if it extends a bit) (but, traffic, so dunno) [11:57] I'm still working on the epochs rerefresh thing [12:00] k [12:00] Chipaca: ok, let me know if you have questions or something [12:03] pedronis: another? [12:03] * Chipaca looks [12:03] Chipaca: ? [12:04] pedronis: I split one out from Epochs yesterday [12:04] Chipaca: forgot to assign it to you? [12:04] pedronis: " make sure to check and trigger new refreshes if there are further transitions available" [12:04] ah, drat, yes [12:04] pedronis: thanks [12:05] removing mine, and tweaking yours [12:05] thanks [12:05] and I didn't notice but replied in public to a private message [12:05] hope nobody's too confused :-D [12:05] or at most as confused as i clearly was [12:07] I'm going to have lunch and see if that makes things better :-) [12:10] mvo, 509 and 510 are the right versions? [12:11] for arm devices [12:16] pedronis: left one suggestion under https://github.com/snapcore/snapd/pull/5712 otherwise lgtm [12:16] PR #5712: overlord: make InstallMany work like UpdateMany, issuing a single request to get candidates