/srv/irclogs.ubuntu.com/2018/12/14/#snappy.txt

mwhudsonuh why are my uploads to the store from launchpad failing?00:07
mwhudsoneg. https://code.launchpad.net/~mwhudson/+snap/go110/+build/40506100:07
mupPR snapcraft#2427 closed: lifecycle: query for multipass install on darwin <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2427>00:23
mborzeckimorning06:21
mupPR snapd#6299 closed: travis: short circuit failures in static and unit tests travis job <Simple πŸ˜ƒ> <Squash-merge> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/6299>06:32
mupPR snapd#6307 opened: systemd/systemd.go: add missing tests for systemd.IsActive <Created by mvo5> <https://github.com/snapcore/snapd/pull/6307>06:54
mupPR snapd#6308 opened: release: 2.36.3 <Created by mvo5> <https://github.com/snapcore/snapd/pull/6308>07:02
mvozyga: 6308 has some interessting bits, would love to understand where the code diff came from07:03
zygaHello07:16
zygaThis is because of different approach to fix a bug in master and in the release branch07:17
zygaI will review it carefully, the master version should be used most likely07:17
mborzeckizyga: mvo: hi07:28
=== pstolowski|afk is now known as pstolowski
pstolowskimorning08:02
mvozyga: ok, just push to the PR if you are happy08:16
mvozyga: I remember now I think :)08:16
mvomborzecki: hey, good morning08:16
pstolowskimvo: oh, still working today?08:17
mvopstolowski: fire is not fully over :(08:17
mvopstolowski: cloud-init on pi and dragon is acting up, i.e. it runs when it should not run and console-conf does not work correctly too08:18
mvopstolowski: hopefully sil2100 (and foundations) can help with these issues08:19
pstolowskimvo: :(08:21
mvopstolowski: it is what it is, I will just take 2 days off in jan08:22
sil2100Yeah, looking into that as much as I can08:22
mvosil2100: \o/08:22
mvosil2100: all the details are in the mail I send out ~2h ago or so (I hope, if anything is missing, please do let me know)08:23
zygaGood morning08:32
mvozyga good morning :)08:33
zygaSome patches required08:33
sil2100mvo: as per my e-mail, could you attach ds-identify.log from the cloud-init logs?08:34
sil2100mvo: anyway, I'm building an image for my pi3 now and I'll test it here as well08:41
mvosil2100: will do, mail goes out in 2min08:45
pedronismvo: hello08:46
mvosil2100: you have mail08:49
mvosil2100: the pi has booted now so things should be quicker now08:49
mvopedronis: good morning08:49
pedronismvo: so  cloud-init on arm devices thinks to be on a cloud, that is annyoing/strange, strange because it wasn't doing that with 1608:50
pedronisotoh is probably a newer cloud-init08:50
mvopedronis: yeah, different version and +100 for annoying08:50
mvopedronis: also console-conf is acting up and the IP keeps changing for me so not a great experience08:51
mvoogra: hey, do you happen to know why the pi3 with 4.15 might switch IP addresses on reboot? (context is UC18 image)08:51
pedronismvo: so it think maybe it's an OpenStack08:52
mvopedronis: can you elaborate? what makes it think that? is that in the ds-identify log?08:53
pedronisyes08:53
mvopedronis: heh, I see08:53
mvopedronis: it does not have more details about the why it seems08:54
pedronisyea, we need to stare at the code08:54
mvopedronis: I just did and had a heart attack08:56
mvopedronis: let me find a link08:56
mvopedronis: it looks like we get the maybe because https://github.com/number5/cloud-init/blob/master/tools/ds-identify#L96008:56
pedroniswonderful08:58
sil2100check for 'OpenStack' returned maybe08:59
sil2100Oh yeah08:59
mvoyes, just verfied that code path is taken09:00
mvoso I guess cloud init does not run much on non-x86 ;)09:00
* mvo weeps a bit in the corner09:00
sil2100mvo: but we had the same thing in core16, right?09:00
sil2100Is cloud-init in xenial different?09:00
mvoppisati:  hey, do you happen to know why the pi3 with 4.15 might switch IP addresses on reboot? (context is UC18 image)09:01
mvono set -e in ds-identify, thats an interessting choice09:03
mvoanyway09:03
pedronismvo: sil2100:  does ubuntu-image write some cloud-init conf09:04
pedronis?09:04
sil2100pedronis: not if it's not asked to, you can explicitly add one if you need to though09:05
pedronismvo: sil2000: it seems quickest fix not touching cloud-init would be to configure ds-identify with maybe=none if arch is not x8609:05
mvopedronis: yeah, forcing cloud-init to be disabled via ubuntu-image is probably the quickest path forward. iirc --cloud-init can already be passed to u-i so we just need to craft a no-cloud config for it09:07
pedronismvo: not disabled09:07
mvopedronis: aha, just maybe=none ? ok09:07
pedronisds-identify09:07
pedronishas a different config09:07
pedronisyes, maybe=none09:07
pedroniseither everywhere or only arm09:07
mvota, I was not aware of this09:07
mborzeckizyga: can you do another pass on https://github.com/snapcore/snapd/pull/6265 ?09:08
mupPR #6265: cmd/snap: attempt to restore SELinux context of snap user directories <SELinux> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6265>09:08
pedronismvo: there's some doc at the top of it, it mentions /etc/cloud/ds-identify.cfg and the kernel cmdline09:09
mvopedronis: thanks, playing with this now09:13
mvopedronis: I also added a trello card UC18 release09:13
mvopedronis: that lists the bugs09:13
mvopedronis: or open issues09:13
pedronisthx09:13
mvo$ cat /etc/cloud/ds-identify.cfg09:19
mvopolicy: search,found=all,maybe=none,notfound=disabled09:19
mvopedronis, sil2100 ^- this has helped and no cloud-init anymore09:19
mvopedronis, sil2100 with that I also have a stable IP it seems09:19
pstolowskimborzecki: hey, if you get to reviewing hotplug PRs, then #6100 and #6114 would be the first to look at; thanks in advance!09:19
mupPR #6100: overlord/ifacestate: hotplug-remove-slot task handler <Hotplug πŸ”Œ> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6100>09:19
mupPR #6114: overlord/ifacestate: handler for hotplug-disconnect task <Hotplug πŸ”Œ> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6114>09:19
mborzeckipstolowski: ack09:19
pstolowskiand i've just de-conflicted them09:20
mborzeckii'm doing 6114 now09:20
pstolowskithanks09:20
mborzeckihm need to grab some coffee09:21
mwhudsonmvo: do you have cloud-init logs from when it was causing problems?09:22
sil2100mvo: btw. from the failed run you had, could you upload the cloud-init log tarball somewhere?09:22
mvomwhudson: cloud init logs for the console-conf setup? or just where it thinks its in a cloud?09:23
mwhudsonmvo: one where set-name ended up in places it shouldn't09:23
mvomwhudson: I can probably do that but need to rewrite the sd card and start from scratch, this is also tricky because cloud-init keeps the logs in /run but I cannot login with the broken network09:24
mwhudsonmvo: ugh09:25
sil2100mwhudson: so far I can share with you mvo's cloud-init-generator and ds-identify logs09:25
mwhudsondebug shell time?09:25
mvomwhudson: yes, just annoying to add in uboot, I struggle with this everytime because I never remember how to do it correctly :(09:25
mwhudsoneveryone loves uboot09:26
mvomwhudson: but I give it a poke in some minutes, just trying to debug another failure on this pi09:26
mvomwhudson: hahaha09:26
mwhudsonok09:26
mwhudsonmvo: basically cloud-init does have code that will write netplan with set-name in it09:26
zygamborzecki: reviewed09:26
mwhudsoni don't at all understand how it could be triggered in this context09:26
mwhudsonbut i also have no other ideas how else it could get there09:27
mwhudsonwait09:27
mwhudson"cloud-init keeps the logs in /run"09:27
mwhudsonis that a core specific thing?09:27
mwhudsonbreak in the initramfs and mount /run as a real partition? :)09:27
mwhudson(systemd might get angry if you do that i guess)09:28
mvomwhudson: to break I need to add a kernel comandline and then I might as well add a debug shell. oh well, I think I will do that09:28
mwhudsonoh yeah09:28
mvomwhudson: maybe its in /var/log as well, I will check09:28
mwhudsonit certainly is in classic09:29
mvomwhudson: pretty sure that the generator log is only in /run but I guess thats not relevant for you, right?09:29
mwhudsonyeah no i want cloud-init.log and cloud-init-output.log09:29
sil2100Didn't see anything in /var/log09:29
mvomwhudson: cool, if thats the case then I can help sooner09:29
sil2100I had to do a cloud-init collect-logs previously09:30
sil2100/var/log/cloud-init.log is empty for me09:30
mborzeckidamn gnome-shell crashing on each unlock09:31
mwhudsonsil2100: hmm09:32
mwhudsonsil2100: syslog?09:32
mborzeckizyga: thanks for the review, i didn't think about adding a selinux backend, but maybe we should once the cleanups are in09:32
sil2100mwhudson: no syslog + eek, persistent journal not enabled by default09:34
mwhudsonsil2100: i am not surprised :/09:35
mwhudson(persistent journal vs sd cards?)09:35
sil2100Let me re-flash and enable persistent journal09:35
zygamvo: I pushed to https://github.com/snapcore/snapd/pull/630809:38
mupPR #6308: release: 2.36.3 <Created by mvo5> <https://github.com/snapcore/snapd/pull/6308>09:38
zygathe diff isn't 0 but it is as close as it should09:38
zygayes, the release branch had one more patch that was in the chain of fixing system key issue09:38
zygait was because my initial branch got closed and reshuffled and I lost track of that09:38
zygaanyone interested in 2nd review on https://github.com/snapcore/snapd/pull/6288 ?09:39
mupPR #6288: cmd/snap-confine: refactor call to snap-update-ns --user-mounts <Per-user mount ns  πŸŽ> <Created by zyga> <https://github.com/snapcore/snapd/pull/6288>09:40
zygamvo: do you know what is the status of SRUs into bionic?09:41
zyga2.35.5 is in proposed09:41
zyga2.34.2 is in main/updates09:41
mvozyga I think there were issues with autopkgtest09:42
zygaaha09:42
zygareal or just noise?09:43
mvozyga: I did not dig into them, iirc autopkgtest killed our tests because they ran too long09:43
zygauhhh09:43
zygaanyway, it's probably not super urgent but it would be good to update the classic package09:43
mvozyga I'm not sure what the latest status is though09:43
zygaand as soon as 2.36.x is out, do that as well09:43
mvozyga absolutely09:43
zygamvo: no worries, one thing at a time09:43
zygamvo: :)09:43
mvozyga its a mess - I think we need to disable adt again, its just not working in our cloud :(09:43
zygafor context, I'm asking because of this bug last evening: https://bugs.launchpad.net/snapd/+bug/180607009:44
mupBug #1806070: snapd.seeded.service never completes preventing full boot to default target <amd64> <apport-bug> <bionic> <snapd:Fix Released> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1806070>09:44
zygathe package in the archive is preventing snapd seeding and refreshing in some cases09:44
mvozyga thanks! I remember we talked about this09:44
* zyga looks at the snow outside09:44
zygabrr09:44
zygaI miss my old computer09:44
zygathe one that doubled as room heater09:44
ppisatimvo: does the ethernet mac change?09:45
sil2100ppisati: it's just the IP, I guess it's all good from the kernel side, it's more like a cloud-init thing09:45
mvoppisati: yeah, sorry for the noise, it stopped for me AFAICT once I disabled cloud-init09:46
mvoppisati: which should not have run in the first place but that is a different story.09:46
ppisatiack09:46
mwhudsonmvo: i think i have more or less convinced myself that the set-name comes from cloud-init09:47
mwhudsonmvo: and now i am going to bed09:47
* mwhudson zzzz09:47
sil2100Indeed, mwhudson thanks!09:47
mvomwhudson: sleep well09:48
mvomwhudson: and thank you!09:48
pedronismvo: we should be careful with the notfound value in that config, it was enabled I think in your original log, not sure where that comes from09:52
mvopedronis: I don't know what enabled it originally, iirc /etc/cloud is empty09:55
pedronismvo: I mean, I'm not sure what happens if it's set to disabled and there's cloud init config09:56
pedronisor somebody attaches a disk some of that09:56
pedronisit was working on 1609:56
mvopedronis: oh, ok. I suspect that 16 has a different ds-identify10:00
sil2100Ugh, indeed10:02
sil2100ds-identify-behavior-xenial.patch10:02
mvosil2100: oh, interessting - can you pastebin that one?10:03
sil2100Wait, one moment, this doesn't look right10:03
pedronismvo: 16 had also this:  https://github.com/snapcore/core/blob/master/live-build/hooks/40-cloud-init-run-after-networkd-wait-online.chroot10:07
pedronisprobably not relevant given that in 18 networkd is used in all server images?10:08
mvosil2100: one more interessting observation - even when I run console-conf I get a prompt to run it again instead of the usual "here is the machine IP and how you log in". but could be an artifcat10:08
mvopedronis: that is the default now unless I very much misremember10:08
mvopedronis: I remember looking at this patch and if we still need it10:09
mvo(I might be wrong though and can double check in a wee bit)10:09
zyga mborzecki https://bugzilla.redhat.com/show_bug.cgi?id=164873310:10
mborzeckiheh10:11
pedronismvo: I read dsidentify, nocloud is a kind of cloud,  either the things ubuntu-image writes or a properly labeled fs shoild make dsidentify detect that10:13
pedronisso that's not a notfound case afaict10:13
mvopedronis: hm, ok10:13
pedronissee dscheck_NoCloud10:14
zygaChipaca: heh, this is fun https://bugs.launchpad.net/snapd/+bug/180845010:14
pedronisstill wondering where the notfound=enabled comes from10:14
zygaChipaca: %q needs to be localised :)10:14
pedronisshouldn't be the default10:14
mupBug #1808450: Strings with placeholders don’t include quotation marks so they can’t be changed per locale <snapd:Triaged> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1808450>10:14
Chipacazyga: hrm?10:14
mvopedronis: interessting10:15
Chipacazyga: yeah, no10:15
Chipacazyga: somebody's being very ornery there10:15
zygawell, I kind of agree10:15
sil2100pedronis: I think it is the default, when you check ds-identify's DI_DEFAULT_POLICY_NO_DMI I guess?10:15
zygasnapd would not get a passing grade from my polish tearcher10:15
zygateacher10:15
pedronissil2100: ah10:16
mborzeckipedronis: left some benchmarks in https://github.com/snapcore/snapd/pull/6265#discussion_r241703113 it's a vm running as a snapshot, though i took care to drop the caches each time, i'll see if i can get some measurements where the are actual writes to a spin drive10:16
mupPR #6265: cmd/snap: attempt to restore SELinux context of snap user directories <SELinux> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6265>10:17
mborzeckifwiw the numbers aren't terrible, 27s for 600k small files, better than fontconfig10:17
pedronismborzecki: this sound like we don't want to do it inconditionally, no?10:20
pedronismvo: anyway afaik  we can just set  maybe=none and the parsing will use the defaults for the rest10:21
pedroniss/afaik/afaict/10:21
mvopedronis: cool, I can try in a bit but right now I am running the testsuite10:21
mborzeckipedronis: the pr was already updated to do it only when the ~/snap (the dir) is not using the default label10:22
mvopedronis: it looks like all the issues cachio had with running the arm tests are from cloud-init being enabled, so far I ran the core18 specific tests and all green, main is running now as well10:22
pedronisgood10:22
mvopedronis: I'm sure sil2100 will find a good config and a way to get it into the system and then this fire is hopefully also done10:23
Chipacazyga: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1808450/comments/110:25
mupBug #1808450: Strings with placeholders don’t include quotation marks so they can’t be changed per locale <snapd:Triaged> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1808450>10:25
pedronismvo: sil2100: we can have that file in core18 itself?10:25
pedronisthat file = ds-identify.cfg10:25
Chipacazyga: I mean, I agree, except in Spanish guillemots are very much optional / recommended-but-not-required10:26
zygaI agree about the low priority10:26
Chipacazyga: and the 2nd preferred quote symbols are … β€œβ€10:26
sil2100pedronis: so I see that in core 16 we had /etc/cloud/ds-identify.cfg10:26
sil2100With policy: search,found=first,maybe=none,notfound=disabled10:26
Chipacazyga: to which " is a close enough approximation for now10:26
sil2100Trying to figure out where this file is coming from10:26
* zyga is happy that he can at least read loads of spanish 10:26
mvosil2100: where did this come from? did u-i put it there ?10:27
sil2100mvo: that's a good question, trying to figure it out - u-i doesn't really have mechanisms for arbitrary files in /etc10:29
mborzeckispread jobs hitting travis timeouts?10:29
sil2100mvo: huh10:29
sil2100mvo: it's in the core snap10:29
sil2100mvo: not sure what's adding it, but if you unsquashfs core you can see the file there - I didn't see it created in any hooks so far, cloud-init itself also doesn't seem to ship it (at least as a file, maybe through maintainer scripts?)10:30
mvosil2100: oh, run10:32
mvosil2100: eh, fun10:32
pedronismvo: sil2100: fascinating10:32
mvosil2100: ok, we can put it back if that fixes all the issues10:32
pedronisanyway means doing the same for 18 wouldn't be too crazy :)10:32
mvopedronis: :)10:32
sil2100Yeah, sounds like the best way ;) I'll try to figure out where it's created out of curiosity anyway10:33
sil2100But this would fix "all the issues"10:33
mvosil2100: I think ubuntu-core-config created this10:33
mvosil2100: give me a sec10:33
sil2100mvo: oh, that's something new!10:33
sil2100I saw it in the image PPA once but didn't know it was used10:33
sil2100Sweet10:33
mvosil2100: we no longer use it10:33
mvosil2100: its a pain10:34
pedroniscore build is so byzantine10:34
mvosil2100: I mean, the configs are split between gits10:34
mvosil2100: so we fixed this in the core18 build but sometimes (like now) we still suffer10:34
mvopedronis: indeed10:34
mvosil2100: fwiw https://github.com/snapcore/core-build/tree/master/config/etc/cloud10:34
pedronisah10:35
pedroniswe have two core related repos?10:35
mvopedronis: yes, sadly - core-build and core10:36
sil2100At least it's all known now \o/10:37
mupPR core18#105 opened: static: add ds-identify.cfg back <Created by mvo5> <https://github.com/snapcore/core18/pull/105>10:37
pedronismborzecki: when you a little bit of time could you look at https://github.com/snapcore/snapd/pull/5712 , it's long lived but now I finally got to make it green, had to adjust for parallel installs changes10:37
mupPR #5712: overlord: make InstallMany work like UpdateMany, issuing a single request to get candidates <Created by pedronis> <https://github.com/snapcore/snapd/pull/5712>10:37
mborzeckipedronis: ack10:37
pedronismvo: that config tree seems a bit in the wrong repo to me10:38
mvopedronis: you mean core-build?10:38
pedronisyea10:38
pedronisif it's basically content of the thing10:38
mvopedronis: this is why I created https://github.com/snapcore/core/pull/8310:38
pedronisit should go into core somehow10:38
mupPR core#83: move most of the ubuntu-core config deb into the snap snap build <Created by mvo5> <https://github.com/snapcore/core/pull/83>10:39
pedronisah10:39
mvopedronis: and then it did not get merged /o\10:39
sil2100eeek10:39
* mvo weeps harder in the corner10:39
pedronisit seems definitely a good idea10:40
mvopedronis: I added a card for myself10:42
sil2100mvo: SHIP IT!10:42
pedronismvo: sil2100: there's some other bits of cloud/ that might be relevant there btw10:43
pedronisI suppose the resize bit is done differently now though10:43
pedronisor is that done for clouds?10:44
pedronisand we need it?10:44
mvopedronis: we ship https://github.com/snapcore/core18/blob/master/static/etc/cloud/cloud.cfg.d/10_snappy.cfg (ironically)10:46
pedronisvery10:47
mvopedronis: which is iirc the same as in uc16 - or am I misunderstanding10:47
pedronisyes10:47
pedronisit's ironic indeed because without cloud-init it would have done nothing10:47
mvopedronis: :(10:48
mvopedronis: so this is why the maybe become a yes?10:48
pedronis?10:49
mvopedronis: what I meant its ironic that we only had half the config. but you indicate that if we had no config at all cloud-init would not have bothered to run?10:49
pedronismvo: sil2100: so it seems we need to port more bits from 16 to static/etc/cloud10:53
mvopedronis: what else is missing?10:54
pedronisthere's a bit about azure it seems10:54
mvopedronis: pretty sure that is obsolete but worth checking with someone from the cloud team10:55
pedronishttps://github.com/snapcore/core-build/blob/master/config/etc/cloud/cloud.cfg.d/99-snappy-azure.cfg10:55
pedronissil2100: can you check if we need this ^ or not anymore ?10:55
pedronismvo: ok, probably easiest for sil2100 to do that10:56
mupPR core18#105 closed: static: add ds-identify.cfg back <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/core18/pull/105>10:56
mvobuilding a new core18 with the right ds-identify.cfg now10:56
* mvo also builds 2.36.3 for regular core beta10:59
mupPR snapd#6296 closed: tests: new backend used to run upgrade test suite <Created by sergiocazzolato> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6296>11:01
pedronissil2100: are you looking into why core18 CI is red? seems related to the checks for package removal11:04
zygamvo: is new snapd okay but we need changes to core18 or will you do another snapd release to fix the cloud issues?11:09
pedroniszyga: the last changes are all just in core18 afaiu11:10
zygagreat11:10
mvo2.36.3 is now in beta core everything but arm6411:12
mvo(the later is still building)11:12
sil2100pedronis: I didn't get to fixing them, but I confirmed that they're not real issues - the test env needs fixing, it's on my plate for ASAP11:13
mupPR snapd#6309 opened: overlord/ifacestate: addHotplugSeqWaitTask helper <Hotplug πŸ”Œ> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6309>11:19
pstolowskizyga: ^ if you have a moment11:22
zygayep saw it11:22
zyga+111:30
pstolowskizyga: +1 on 6288, but plese see the comment11:30
zygapstolowski: applied!11:32
pstolowskity11:33
* pstolowski lunch11:39
zygamborzecki: something to garden https://bugzilla.redhat.com/buglist.cgi?component=snapd&product=Fedora11:42
zygamake sure to select *all* bugs11:43
mupPR snapd#6310 opened: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6310>11:44
zygamvo: found the patch that didn't make it into master https://github.com/snapcore/snapd/pull/631011:44
mupPR #6310: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6310>11:44
mvozyga: ta11:54
ChipacaI need to go to the boys' school for 2pm, meaning I'm not going to be here for the standup (although I should be back by 2.30 so I might see y'all if it extends a bit) (but, traffic, so dunno)11:56
ChipacaI'm still working on the epochs rerefresh thing11:57
zygak12:00
pedronisChipaca: ok,  let me know if you have questions or something12:00
Chipacapedronis: another?12:03
* Chipaca looks12:03
pedronisChipaca: ?12:03
Chipacapedronis: I split one out from Epochs yesterday12:04
pedronisChipaca: forgot to assign it to you?12:04
Chipacapedronis: " make sure to check and trigger new refreshes if there are further transitions available"12:04
Chipacaah, drat, yes12:04
Chipacapedronis: thanks12:04
pedronisremoving mine, and tweaking yours12:05
Chipacathanks12:05
Chipacaand I didn't notice but replied in public to a private message12:05
Chipacahope nobody's too confused :-D12:05
Chipacaor at most as confused as i clearly was12:05
ChipacaI'm going to have lunch and see if that makes things better :-)12:07
cachiomvo, 509 and 510 are the right versions?12:10
cachiofor arm devices12:11
mborzeckipedronis: left one suggestion under https://github.com/snapcore/snapd/pull/5712 otherwise lgtm12:16
mupPR #5712: overlord: make InstallMany work like UpdateMany, issuing a single request to get candidates <Created by pedronis> <https://github.com/snapcore/snapd/pull/5712>12:16
pedronismborzecki: thx12:17
* cachio afk12:18
zygacoffee12:21
mborzeckiwtf is going on with fedora repos?12:28
zygamborzecki: they have holidays too?12:43
mborzeckiheh12:43
zygaI need to go run an errand, brb in 30 minutes12:43
Son_Gokuoh god, I'm so happy12:43
Son_Gokuthis is my last day of work for 201812:43
zygaI wished :)12:44
zygaSon_Goku: I'm iterating on https://github.com/snapcore/snapd/pull/611112:44
mupPR #6111: packaging/opensuse: move most logic to snapd.mk <Created by zyga> <https://github.com/snapcore/snapd/pull/6111>12:44
Son_Gokuzyga, I didn't take much vacation this year, since I only had the one trip for Flock12:44
zygaI need to run that errand but the last part is the with/bcond work12:45
zygaand then I need to figure out why go is broken in leap again :)12:45
zygaafk :)12:45
* Son_Goku waves12:45
mborzeckizyga: the problem with refreshing fedora repos seem to happen only in gcp12:46
mborzeckii can refresh just fine here12:46
zygacloudscale problem ;)12:48
zygamborzecki: idea12:48
zygamborzecki: spin up our own mirror12:48
zygatry against that12:48
mborzeckizyga: heh, hardly sounds like the thing we ought to be focusing on :)12:51
mvocachio: hey, yeah, 508,9 should be the right ones13:23
mvocachio: I will be afk for a bit but looking at tg13:24
cachiomvo, good13:25
zygare13:35
zygathis took forever, sorry13:35
zygais it just me or is travis stuck?13:36
pedroniszyga: we might have hit some limit13:40
cachiozyga, hey13:44
zygayes?13:44
cachiofedora 29 is taking more time than other systems13:45
cachioI saw it is taking time to build snapd13:45
zygaany idea why? download speed? more updates than usual?13:45
cachiocould tha because because it is using go 11?13:45
zygadid go 11 got introduced?13:46
zygaas otherwise I don't see how go version is a factor13:47
cachionot sure when I was introduced13:49
cachiobut after last update it is taking more time to run fedora2913:49
zygaafter last update of what?13:49
cachiofedora 2913:49
zygaof the image?13:49
cachiothere was a kernel update13:49
cachiozyga, yes13:49
zygacan you diff the package manifest in the old and the new image?13:49
zygaI don't know what changed13:50
pedroniszyga: is the description of 6310 up-to-date? didn't we learn that the SIGTERM was coming from systemctl stop snapd ?13:50
zygaI also don't have an suspicion what could be a factor13:50
cachiozyga, sure13:50
zygapedronis: it's the original patch that got lost in various changes, this is to match the release merge PR that mvo noticed has some delta13:51
pedroniszyga: this for master though, right?  so an up-to-date description would be best13:51
zygayes, it is the same patch as before, just proposed in a branch that got closed13:52
zygaI can edit the description, sure13:52
pedronisthank you13:52
cachiopedronis, I'll join in 2 minutes14:00
=== ricab is now known as ricab|lunch
threshhow many snaps in the store are already using core18?14:26
mvocachio: core18 is still not good, it looks like /etc/cloud is not populated (cc sil2100)14:37
pstolowskimborzecki: updated #611414:38
mupPR #6114: overlord/ifacestate: handler for hotplug-disconnect task <Hotplug πŸ”Œ> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6114>14:38
pstolowskimborzecki: could you also take a look at the #6309 (simple)14:38
pstolowski?14:38
mupPR #6309: overlord/ifacestate: addHotplugSeqWaitTask helper <Hotplug πŸ”Œ> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6309>14:38
mborzeckimhm14:38
pstolowskithanks14:38
cachiomvo, ouch14:39
cachiobad news14:39
cachiomvo, I'll run beta validation for 2.36.314:39
cachiountil core18 is ready14:40
sil2100mvo: huh? Something wipes out the /etc/cloud contents?14:40
sil2100mvo: hm, looking into core18 snap's squashfs I see /etc/cloud/ds-identify.cfg14:41
sil2100mvo: you don't have it?14:41
sil2100mvo: are you sure you're testing the right image/core18 snap?14:42
sil2100Let me build a test image and check on my pi314:42
mvosil2100: if you boot your VM, do you see a populated /etc/cloud ?14:42
mvosil2100: x86 should be enough14:42
sil2100Building14:43
sil2100mvo: ah, I see what's happening, /etc/cloud is in writable-paths14:47
sil2100hmmm14:47
mvosil2100: I have a theory14:47
sil2100But it should work14:48
mupPR core18#106 opened: static,hooks: make /etc/cloud a "synced" dir for now <Created by mvo5> <https://github.com/snapcore/core18/pull/106>14:49
sil2100AH14:49
sil2100Holy moly14:49
mvosil2100: the rabbit hole is deeper14:50
mvosil2100: I just added another paragraph14:50
mvosil2100: anyway, I think this is the best we can do for now but we may need to deprecate ubuntu-images --cloud-config option14:50
mvosil2100: or push it into cloud.cfg.d or something14:50
mvosil2100: but even that has config :/14:51
mvosil2100: well, actually I think we need to simply support cloud.cfg and ds-identify.cfg in the gadget14:53
mvosil2100: anyway, the above PR will unblock the world14:54
zygamvo: fun friday, it seems14:58
sil2100mvo: ok, this looks good, I'm just wondering - what is creating the empty /etc/cloud directory? Is it part of snap prepare?14:59
sil2100Since I don't remember we have anything in ubuntu-image that does that explicitly15:00
zygahttps://github.com/snapcore/core18/search?q=%2Fetc%2Fcloud&unscoped_q=%2Fetc%2Fcloud15:00
zygasil2100: it is 020-extra-files.chroot15:00
mupPR snapd#6311 opened: image: do not write empty etc/cloud <Created by mvo5> <https://github.com/snapcore/snapd/pull/6311>15:01
mvosil2100: yeah, see 6311 (coming in a sec if mup is attentive)15:01
sil2100mvo: so maybe it'd be just better to remove that one?15:01
sil2100I mean, synced is fine I guess, but maybe we don't have to do it, all we need is to not create that empty dir?15:01
mvosil2100: we could remove it its a good question, removing requires building the images with a snap command build with 631115:02
sil2100ubuntu-image also won't put any files there, the only cloud-init stuff it puts in is the user-data15:02
sil2100hm15:03
mvosil2100: snap prepare-image creaets the dir even if its empty15:03
mvosil2100: thats a bug and then u-i puts the empty dir into the writable dir15:03
mvosil2100: of course if we can teach u-i to not copy the empty dir that would be even nicer15:03
sil2100Ah, ok15:03
mvosil2100: if you are comfortable with chaning u-i that works for me as well, slightly preferable15:04
sil2100mvo: ok, I guess it's just safest to go with synced then, since I see there's multiple places where the dir can pop up15:04
mvosil2100: we still have a problem with gadgets that ship a cloud.conf because we no longer "merge" the configs but I think this is a feature :)15:04
sil2100mvo: since as per what zyga pasted we also create the empty /etc/cloud dir in core18 itself15:04
mvosil2100: its fine if its in the core18 snap15:05
mvosil2100: the problem is really that its also in /writable15:05
mvosil2100: and snap prepare-image creates it and then u-i puts it there15:05
sil2100Ah, right!15:05
sil2100Stupid me15:06
mvosil2100: I can poke a u-i to see if there is a single place to check15:06
mvosil2100: no worries a) its friday b) its complicated15:06
mvosil2100: (not necessarily in this order ;)15:06
sil2100mvo: it's certainly possible to do in u-i, no problem15:06
sil2100Problem is in releasing u-i again ;p15:06
sil2100i.e. not something to be done on Friday15:06
mvosil2100: heh15:06
* zyga hugs sil2100 and mvo15:06
mvosil2100: as long as cachio can create correct images we can release monday ;)15:07
* mvo hugs zyga15:07
mvozyga: yeah, fun week so far15:07
* sil2100 hugs zyga back15:09
sil2100mvo: ok, for now I approved the synced MP15:09
sil2100I'll prepare a hack for u-i in the meantime15:09
mvosil2100: does the u-i snap embedd the snap command it uses?15:12
mupPR core18#106 closed: static,hooks: make /etc/cloud a "synced" dir for now <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/core18/pull/106>15:13
zygaso leap 42.3 *does* have new go 1.1015:13
mvosil2100: I take it for now to unblock sergio - if we have something better we can revert without any harm15:13
zygaupdate from 1.915:14
sil2100mvo: I don't think the u-i snap ships snap, I think it uses the system one15:15
sil2100I'll double check15:15
=== ricab|lunch is now known as ricab
mvosil2100: can I become part of the people who can trigger a build in https://code.launchpad.net/~ubuntu-core-service/+snap/core18 ?15:16
mvosil2100: aha, nevermind - autobuild started15:16
sil2100mvo: of course you can o/15:17
mvosil2100: fwiw, with an image generated with #6311 cloud-init is disabled so this part is fine15:30
mupPR #6311: image: do not write empty etc/cloud <Created by mvo5> <https://github.com/snapcore/snapd/pull/6311>15:30
mvocachio: new coew18 is ready in beta15:34
cachiomvo, nice15:39
cachioI'll start after lunch15:39
cachiostill without iternet at home15:39
cachioI think I'll need to go to another place15:39
mvocachio: uh, good luck15:40
sil2100mvo: ok I have the u-i hack + tests in place, let me give it a spin in a moment15:43
mvosil2100: cool15:44
mvosil2100: \o/15:44
mvosil2100: the "synced" change also works, tests are running on my pi3 and all the stragness is gone (noe more property set-name error in netplan etc)15:44
sil2100mvo: it's best to have more options to choose from I suppose ;) And I also think it'd be easier to fast-track ubuntu-image to release pockets than for instance snapd, since our livefs builders use archive packages for building the images15:46
zygacachio: I solved the problem with opensuse leap15:51
zygacachio: I will push a small update soon15:51
zygacachio: but it might be best to refresh all opensuse images15:51
zygathough let's do this only after my fix is merged15:51
cachiozyga, sure15:52
cachiozyga, thanks for that fix15:52
mvosil2100: yeah, I like the idea of doing it in u-i15:58
* cachio lunch16:01
cyphermoxmvo: property error for set-name in netplan?16:02
sil2100cyphermox: yeah, due to cloud-init meddling with the netplan config in some cases the netplan config ends up with a set-name: without a match:16:05
sil2100Like, it's injecting set-name when it shouldn't16:06
cyphermoxyeah, ok16:06
cyphermoxsil2100: to me "when it shouldn't" for set-name is pretty much always16:06
cyphermoxI feel strongly like it's something that should be avoided at all costs, and kind of sad that it got in, because it's often a cause for confusion :(16:07
sil2100mvo: https://github.com/CanonicalLtd/ubuntu-image/pull/167 <- seems to work16:16
mupPR CanonicalLtd/ubuntu-image#167: Do not copy-over /etc/cloud to the rootfs if it's empty <Created by sil2100> <https://github.com/CanonicalLtd/ubuntu-image/pull/167>16:16
sil2100mvo: I'd like to wait for the autopkgtests to run for those16:24
sil2100s/those/this PR16:24
mvosil2100: nice one16:26
mvosil2100: btw, can you commit to https://github.com/snapcore/core18 now? if not we need to ask niemeyer  to give you commit access16:35
mvosil2100: I slightly prefer your u-i workaround, if thats ok with you and if we can give cachio a way to create the images I'm +1 on this one and we can revert the core18 commit that uses a synced dir16:36
niemeyermvo: sil2100 should have access, as long as he accepted the invitation to join the team16:36
mvosil2100: but no super strong opinion, synced is also easy to change later it won't do any harm16:36
sil2100mvo: so I'd still keep the synced writable-path for now until I push out a new ubuntu-image - I guess it might be enough for us to have it in disco, then I guess it should go relatively fast16:39
sil2100mvo: not sure if we built core18 images before on disco16:40
sil2100(like, on the disco livefs)16:40
mvosil2100: ok16:40
mvosil2100: I leave this to you then16:40
mvosil2100: thanks, fwiw, tests on the pi look good now still16:40
sil2100\o/16:44
zygacachio: on second thought17:02
zygacachio: please run "zypper refresh && zypper update" on all suse images17:02
zygacachio: the fix is really equivalent to doing that and rebooting :/17:02
zygaso we may just as well do that17:02
kyrofaHey zyga, did you have any other thoughts on https://forum.snapcraft.io/t/raspberry-pi-3-install-broken/8957/5 ?17:06
zygakyrofa: I bet a beer that it's either ENOSPC or EROFS after ext4 corruption on flaky sd card17:07
kyrofazyga, want me to suggest trying another SD card?17:09
zygaWe should first get full logs from systemd17:09
zygaI don't think we got an answer since yesterday17:09
kyrofaIndeed, we have not17:10
kyrofaI'll go ping him on another forum :P17:10
=== pstolowski is now known as pstolowski|afk
* zyga EODs17:20
zygaI'll check my PRs later17:20
zygaI love how this page looks like for communicating features to users https://github.com/github/VisualStudio/issues/187817:24
sil2100mvo: ok, autopkgtests were happy with the branch, let me push it out to the archive17:41
sil2100mvo: I'll make it an interim release, e.g. not 1.6 per-se17:42
sil2100Actually hmm, I anyway need to get those changes out17:57
sil2100mvo: ubuntu-image 1.6 pushed to disco, SRUs to other series will follow18:00
diddledancan I specify something in snap.yaml (using snapcraft's passthrough) that will match the arch-triplet of the current system so that I can move $SNAP/usr/lib/$ARCH_TRIPLET/webkit2gtk-4.0 to /usr/lib via bind mount without adding every arch separately?18:39
zyganot using passthrough18:39
diddledanI obviously mean bindmounting it to /usr/lib/$ARCH_TRIPLET/webkit2gtk-4.018:39
diddledanand I'm referring to using layouts18:40
zygasnapd doesn't support arch triplet because it's distro specific18:40
zygawe tried to and abandoned that18:40
zygaonly snapcraft does18:40
diddledanso how do I do this?18:40
zygamanually18:41
zyganot sure18:41
zygausing snapcraft?18:41
zygawhy are you using passthrough?18:41
diddledanfor layouts18:41
zygayou can use layouts now without that18:41
diddledanlayout isn't supported by snapcraft yet18:41
zygait is18:41
diddledannope18:41
zygayep18:41
zygasince 3.018:41
zygabut18:41
zygafor unknown reason you need to start using "base: core18"18:41
zygawhile technically not required18:41
zygait's a snapcraft bait and switch18:41
diddledanwell that's silly18:42
diddledan:-p18:42
zygaI don't disagree :)18:42
zygait would not be silly if we had shipped core1618:42
zygaand you could just say "base: core16"18:42
zyga"layouts: ..."18:42
zyganow it is silly18:42
zygamvo: if you are working by any chance18:44
zygacould we ship core16 base snap alongside core18?18:44
diddledanhah: "You need multipass installed to build snaps which use the base keyword" <-- I've already got it!18:44
diddledanoh!18:44
diddledanit's trying to install multipass inside a multipass vm because I accidentally left SNAPCRAFT_BUILD_ENVIRONMENT=multipass in the environment18:45
diddledanturtles all the way down18:45
pedroniszyga: we really don't want to push core16 much yet18:54
zygapedronis: then we should not marry core18 with layouts in snapcraft IMO18:55
pedroniszyga: did I say we should18:55
zygawe do :)18:55
zygaor did18:55
pedronisalso for clarity: core16 is not stable yet19:01
diddledanwell blow me down with a feather! you're right, using base:core18 does indeed enable layout: to work19:01
pedronisfedora tests are a mess atm19:09
diddledanyey?19:10
zygapedronis: shall I disable it?19:17
pedroniszyga: maybe, it's causing a lot of red atm19:18
zygaon it19:18
cachiosil2100, hey, still don't have internet at home19:22
cachiosil2100, I'll make the validation over the weekend and I'll send the results19:22
zygapedronis: done19:22
cachioso we can continue to stable19:22
mupPR snapd#6312 opened: spread: disable fedora-29-64 <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/6312>19:23
pedroniscachio: if you are sending out mails about the results, please cc also me19:24
pedronisthank you19:24
pedroniszyga: thx, let's see if that one gets green19:24
cachiopedronis, sure19:32
cachioI think I'll send the results tomorrow19:32
cachiopedronis, today I am using my phone internet and it goes so slow19:33
sil2100cachio: thanks!19:43
cachiozyga, opensuse update failed19:45
cachiozyga, https://paste.ubuntu.com/p/pC9bphSTgr/19:46
cachiousing master19:46
zygacachio: this error shows up when you update and not reboot20:01
mvozyga: we talked about this, ideally we would just make core an alias for core16 in snapd20:57
mvozyga: so that snapd understands that it just needs core for this20:57
zygammm20:57
zygamvo: anyway, not urgent20:58
zygamvo: I'm trying to land https://github.com/snapcore/snapd/pull/631220:58
zygaif you want to look20:58
zygait should help20:58
mupPR #6312: spread: disable fedora-29-64 <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/6312>20:58
mvozyga: sure20:59
zygaha21:10
zygayou were faster21:10
mupPR snapd#6312 closed: spread: disable fedora-29-64 <⚠ Critical> <Created by zyga> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/6312>21:11
pedronisit's merged21:11
zygathank you :)21:11
zygaI restarted a few PRs21:12
zygalet's see21:12
zygamvo: I left a few comments on https://github.com/snapcore/snapd/pull/584521:12
mupPR #5845: interface: add new `{personal,system}-files`  interface <Created by mvo5> <https://github.com/snapcore/snapd/pull/5845>21:13
zyganot sure if you want to do anything about it21:13
zygaor just merge it21:13
zygait's green21:13
roadmrohai zyga21:23
zygahey :)21:23
roadmrhow's it going? is the glue trick keeping you safe from spaghetti?21:23
zygahaha21:23
zygayes!21:23
zygait's a miracle21:23
zygaprinting on 1st go21:24
zygaI'm shocked21:24
roadmrgreat! how do you unglue the piece from the bed when it's done?21:24
zygashould have learned this months ago21:24
zygathe bed comes off and you just bend it slightly21:24
zygathings pop off without any issues21:24
roadmrzyga: how'd you find out about this?21:25
zygaabout the glue?21:25
zygaread that it helps21:25
zygafound some information on the internet by chance21:25
roadmrinteresting, sounds like it'd be a very well-known fact, as I think it's a common problem21:27
roadmrglad it's working well now :)21:27
roadmrzyga: hey a snapd-related question - you may know and save me hours of dredging through the code base21:27
zygait may be well known, I'm happy I know it too :)21:27
zygahow can I help?21:28
roadmrzyga: do you remember at some point, snap deltas were disabled for armhf devices because $REASONS?21:28
zygabecause CPU uber slow21:29
roadmrzyga: hm, and were they enabled at some point? or are armhf devices still delta-less?21:30
zygaI don't know that21:30
zygaI don't know how the delta was disabled21:30
roadmrzyga: ok.. I found something about enabling deltas for core devices, back in February, but that seems to be arch-agnostic21:31
roadmrzyga: ok, no problem - thanks! maybe I'll ask ogra whom I remember talked about this at some point21:31
ograroadmr, they were in fact not enabled in the beginning because nobody had measured the performance for single core arm, but Chipaca did some research at some point, i thought they were enabled later on21:33
roadmrhi ogra ! yes, sorry, as I'm here grilling you guys I'm also analyzing download logs and it looks like they were enabled at some point; I'm trying to determine when (basically which version of snapd did that). For instance, I see some armhf devices with  2.35 getting deltas, while others with 2.32 seem to not get deltas21:35
roadmrogra, zyga : if I can pinpoint the version that enabled this, I can tell the guy using that old 2.32 to upgrade to 2.3x so they can benefit from awesome bandwidth savings for free21:36
roadmrif Chipaca is the one who did it, I can ask him on Monday; I'm in no rush :)21:36
mupBug #1808593 opened: AppArmor denies ptrace for ufw snap in lxc after purging ufw <Snappy:New> <https://launchpad.net/bugs/1808593>21:45
mupBug #1808593 changed: AppArmor denies ptrace for ufw snap in lxc after purging ufw <snapd:New> <https://launchpad.net/bugs/1808593>21:48
mvozyga: I want to address the comments but not today :) thanks for those!22:08
pedronisroadmr: it was done in february, not by John22:08
roadmrpedronis: https://github.com/snapcore/snapd/pull/4639 ?22:09
mupPR #4639: store: enable deltas for core devices too <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4639>22:09
pedronisyes22:09
roadmrpedronis: and that went out in snapd 2.33.1?22:09
thirasI'm getting "ENOENT: no such file or directory, lstat '/snap/code'"22:10
thirasat the vscode22:10
thirasany idea what is this?22:10
pedronisroadmr: 2.33 already if I believe the changelog22:11
roadmrpedronis: yay thanks22:11
pedronisroadmr: yes, 2.3322:12
roadmrpedronis: great! thanks for helping confirm.22:13
roadmrthis matches what I see in logs, btw22:13
roadmrthanks for the help folks, I appreciate it, I know it's late for you22:13
mupPR snapd#6288 closed: cmd/snap-confine: refactor call to snap-update-ns --user-mounts <Per-user mount ns  πŸŽ> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6288>22:37
mupPR snapd#6307 closed: systemd/systemd.go: add missing tests for systemd.IsActive <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6307>22:58
mupPR snapd#6309 closed: overlord/ifacestate: addHotplugSeqWaitTask helper <Hotplug πŸ”Œ> <Created by stolowski> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6309>22:58
mupPR snapd#6263 closed: Retrieving all pages for 'find' command, not only one <Created by WiseRaccoon> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6263>22:59
thirasis there a way to check if a local snap file (not downloaded from Canonical) has right to access to disk?23:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!