/srv/irclogs.ubuntu.com/2020/02/20/#snappy.txt

=== NickZ3 is now known as NickZ
mupPR snapcraft#2950 opened: meta: Snap to_dict() cleanup <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/2950>01:52
mupPR snapd#8148 closed: overlord/configstate/configcore: add support for backlight service <⛔ Blocked> <Created by EthanHsieh> <Closed by EthanHsieh> <https://github.com/snapcore/snapd/pull/8148>05:14
mborzeckimorning06:08
mborzeckischool run06:35
mupPR snapd#8160 opened: overlord/configstate: add backlight option <Created by EthanHsieh> <https://github.com/snapcore/snapd/pull/8160>06:38
mborzeckire07:12
mborzeckimvo: hey07:33
mvomborzecki: good morning07:38
zygagood morning07:50
mborzeckihmm looks like my git filter-branch incantations on #8156 were not good enough07:50
mborzeckizyga: hey07:50
mupPR #8156: [RFC] cmd/snap-bootstrap: subcommand to detect if we want a chooser to run <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8156>07:50
zygahmm07:51
zygadidn't I review https://github.com/snapcore/snapd/pull/8160 already?07:51
mupPR #8160: overlord/configstate: add backlight option <Created by EthanHsieh> <https://github.com/snapcore/snapd/pull/8160>07:51
zygaor something just like it?07:51
mvozyga: I think the config var got tweaks07:53
zygameanwhile master is still red07:54
zygaIan sent a fix but it seems to be insufficient or also broken somehow07:54
zygahmm, what is curious is that the failure is only present on core1807:55
zygaI wonder if that's because of the extra logic on how snapd is started there07:56
mvozyga: I run a debug session now08:03
mvogood morning pstolowski08:03
pstolowskimorning08:03
zygahey Pawel, good morning08:03
mvozyga: it's a bit annoying, also it's super unclear why it's starting now08:03
mborzeckipstolowski: morning!08:03
mvozyga: actually - new systemd in bionic since 2020-02-1708:04
zygaohhh08:04
mvozyga: I think this is roughtly when the trouble started?08:04
zygathat's a good trail08:04
zygayeah, let's see the changelog08:04
mvozyga: we could run core18 tests with stable core18, that still has the old version08:05
zygawow, that's true, that's brilliant08:05
zygaif that passes we know for sure it's the base OS that changed08:05
mvozyga: the changelog has a smoking gun08:05
zygareading it now08:06
zygaog08:06
zygaOnFailure job something08:06
zygadon't trigger ONFailure08:06
mvo    - Only trigger OnFailure= if Restart= is not in effect (LP: #1849261)08:06
mupBug #1849261: Update systemd for ubuntu 18.04 with fix for interaction between OnFailure= and Restart= <architecture-s39064> <bionic> <bugnameltc-182011> <ddstreet> <id-5de127a738dabd05006e38e8> <severity-high> <systemd> <targetmilestone-inin1804> <verification-done> <verification-done-bionic>08:06
mup<Ubuntu on IBM z Systems:Fix Released by canonical-foundations> <systemd (Ubuntu):Fix Released> <systemd (Ubuntu Xenial):Won't Fix> <systemd (Ubuntu Bionic):Fix Released by ddstreet> <systemd (Ubuntu Disco):Fix Released> <systemd (Ubuntu Eoan):Fix Released> <https://launchpad.net/bugs/1849261>08:06
mvozyga: exactly, this looks super suspicious08:06
mvozyga: also wrong, I mean, if it fails to restart for n times we need to go into failure08:06
mvoanyway08:06
zygahmm hmm hmm08:07
zygamvo: but it passes in 20.0408:07
zygaso something changed there08:07
zygathe patch is https://github.com/systemd/systemd/pull/9158/files08:08
mupPR systemd/systemd#9158: trigger OnFailure= only if Restart= is not in effect <pid1> <Created by poettering> <Merged by keszybz> <https://github.com/systemd/systemd/pull/9158>08:08
zygaI'll check if that got changed further down the line08:08
mvozyga: cool, thank you08:08
mvozyga: I think this is it, kind of annoying08:08
mvozyga: but at least we know now what to do08:08
zygamvo: the test does run on 20.04, right?08:08
mvozyga: lol - no08:09
mvozyga: it has a TODO:UC20: "does not work for unknown reasons"08:09
zygapfff08:09
zygahahahaha08:09
zygaso it's broken08:09
zygaand now we need to live with it08:09
zygaok08:10
mvozyga: well08:10
mvozyga: let me try something08:10
zygathe code didn't change since that patch, systemd master still behaves the same way it does in 18.0408:10
zygaok08:10
zygaso08:10
zygaI think we need to change how snapd restarts08:11
zygaif we limit that to a fixed number08:11
mvozyga: I hope we just need to change the startlimitinterval08:11
zygathen on failure will trigger again08:11
zygano, not the interval08:11
mvozyga: then the failure will be triggered again08:11
mvozyga: no?08:11
zygawe need to allow snapd to really fail08:11
zygathat's how I understand this fragment:08:11
zygahttps://www.irccloud.com/pastebin/phqUSrfE/08:11
zygalet me read the diff again08:11
mvozyga: right, my understanding is that once we hit the restart interval it will actually go into failed state and then the OnFailure is run08:12
mvozyga: but I just infered this, did not read the patch08:12
zygawe need to check08:12
zygaI mean08:13
zygajust do08:13
ijohnsonzyga: mvo: what I think is the best fix to fix this with the new systemd is 2 things, 1 nfs-support test needs to also reset-failed on snapd.socket and 2 snap-failure needs to reset-failed before trying to start the socket unit08:13
zygahttps://github.com/systemd/systemd/pull/9158/files#diff-a89a9b6f80aada989d298b4c2c3a9d64R243508:13
mupPR systemd/systemd#9158: trigger OnFailure= only if Restart= is not in effect <pid1> <Created by poettering> <Merged by keszybz> <https://github.com/systemd/systemd/pull/9158>08:14
zygaso, as long as the service will auto restart08:14
zygawe don't get failed state08:14
zygaijohnson: good mornign!08:14
zyga(it's kind of early, wow :)08:14
ijohnsonYeah couldn't sleep but probably will try to go back to bed shortly08:14
mvoijohnson: woha, hey08:15
mvoijohnson: good evening :)08:15
* mvo hugs ijohnson 08:15
ijohnsonmvo: indeed :-)08:15
mborzeckiijohnson: hey! already adjusting to CET timezone?08:15
zygamvo: I need to handle an errand at home, I'll be back in 2008:16
mvozyga: no worries08:16
mvozyga: testing a patch now08:16
ijohnsonAnyways snap-failure needs to be a bit more robust about starting the snapd.socket service anyways08:16
mvoijohnson: thanks, that's good to know08:16
ijohnsonBecause cachio ran into another problem with this test that happens because the socket is still lying around08:17
mvoijohnson: I'm poking at it now, a bit annoying since it takes forever to run each test08:17
ijohnsonHaha yes that was most of my day yesterday waiting for spread and fixing other random things in the meantime08:17
ijohnsonmvo: but also see my comment about the nfs-support test, it calls reset-failed on _only_ snapd, it should also call that on snapd.socket too08:18
mborzeckistepping out for a bit to get the papers to my accountant, and then will probably work from some place in the city08:22
mupPR snapd#8161 opened: tests: set StartLimitInterval in snapd failover test <Created by mvo5> <https://github.com/snapcore/snapd/pull/8161>08:34
mvoijohnson: thanks, I check these area too but it seems the current issue is about the StartLimitInterval08:36
zygare08:36
zygathere's some chaos at home today08:41
zygaI'm heading out08:41
ijohnsonmvo: what I was trying to say is that the nfs-support test will start failing too when you make adjustments to the StartLimitInterval that are favorable to snapd-failover because that test is flaky due to not resetting the failed count of the snapd.socket service08:41
zygadon't want to be a part of this08:41
zyga(family life and living with parent-in-law)08:41
* mvo hugs zyga 08:43
mvoijohnson: aha, thanks. my current approach was to adjust the StartLimitInterval  only in this one test08:44
mvoijohnson: i.e. just enough to get us on our feed again, not at all against fixing more08:44
ijohnsonAh okay nvm me then08:45
mvoijohnson: let's talk some more later, I really don't want to keep you from sleep :)08:47
ijohnsonIt's fine no worries08:47
zygamvo: re09:08
zygamvo: small observation09:08
zygamvo: perhaps we want to consider a spread suite that does run in autopkgtest and gates classic systemd09:08
zygamvo: we would catch this if that test was a part of that set09:09
zygamvo: something to consider after this is resolved09:09
zygamvo: I'll work on my tasks from Jamie's review - please pull me into a review once you have the fix09:10
zygamvo: and it's also a good good catch about this test, if we had disabled that test and released a new core we could really have a bad day09:10
mvozyga: it's an interessting idea, we did have autopkgtest for snapd but it was terrile unreliable09:11
mvozyga: we would have caught this thought09:11
zygayeah, perhaps we can turn some knobs to make it better though09:11
zygadunno09:11
* mvo nods09:11
zygatotally offtopic, I noticed that systemd --user runs for gdm09:13
zygawe should probably not run snap services for gdm09:13
pedronismvo: hi, is it expected there's no standup in the calendar today?09:15
mvopedronis: not expected, uh, let me fix this09:16
mvopedronis: thanks for catching that09:16
pedronis#8130 needs a 2nd review09:23
mupPR #8130: overlord, state: don't abort changes if spawn time before StartOfOperationTime (2/2) <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8130>09:24
mupPR snapd#8162 opened: features: enable "parallel-installs" by default <Created by mvo5> <https://github.com/snapcore/snapd/pull/8162>09:27
mborzeckiyay09:29
mborzeckimvo: that's for 2.44?09:29
zygamvo: that's great to see, thank you!09:30
mvomborzecki: that's something to discuss09:32
mvomborzecki: 2.44 means it's ready for 20.04 and most likely on the CD09:32
mvomborzecki: which is great09:32
mvomborzecki: but also risky09:33
mborzeckiayy ;)09:34
mvomborzecki: let's talk at the standup09:34
mborzeckimvo: ok09:34
mvoalso - amazon linux is failing to install packages right now :(09:34
mborzeckimvo: duh, got logs?09:35
mvomborzecki: yes, on sec09:36
mvomborzecki: I move it to unstable for now, look like some repo issue09:36
mvomborzecki: probably transient09:36
mvomborzecki: https://travis-ci.org/snapcore/snapd/jobs/652883253?utm_medium=notification&utm_source=github_status09:37
mvoyay, 8161 unbreaks the snapd-failover test09:39
pedronismvo: we really need to do something about aliases before turning on parallel installs by default09:39
mborzeckimvo: yeah, looks like inconsistency in amazon's repos09:39
mborzeckimvo: and kinda basic, iproute, iptables :/09:40
mupPR snapd#8162 closed: features: enable "parallel-installs" by default <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8162>09:40
mvopedronis: ok, closing this again then until we have time for this09:40
mvomborzecki: yeah, that was my impression as well, hopefully transient09:41
mborzeckipedronis: thanks for the review09:41
pedronismvo: yea, sadly it needs more work, but I think the current experience is suboptimal09:41
pedronisand we turn it on we are stuck with the behavior forever09:41
mvoreviews for 8161 would be appreicated09:45
mvo(should be super simple)09:45
mvoand fixed the snapd-failover test - at least in the one run that happened there :)09:45
zygamvo: what's the context of amazon linux and unstable10:09
zygathe commit message has no other information10:09
pedronis<mvo> also - amazon linux is failing to install packages right now :(10:12
mupPR snapd#8163 opened: tests: enable snapd-failover on uc20 <Simple 😃> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8163>10:13
* pedronis reboots10:37
mupPR snapd#8164 opened: snap: use the actual staging snap-id for snapd <Simple 😃> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8164>11:26
mborzeckipedronis: got a question about https://github.com/snapcore/snapd/pull/8156#discussion_r381881462 you'd see the WaitTriggerKey() be moved to some other file right and just keep the interfaces and related structs in triggerwatch.go?11:26
mupPR #8156: [RFC] cmd/snap-bootstrap: subcommand to detect if we want a chooser to run <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8156>11:26
pedronismborzecki: not quite,  I think the structs should go where they are used as input or output to functions11:27
pedronismborzecki: this is go so the interfaces can be defined on the consuming side11:27
pedronismborzecki: I mean keyEvent etc should be in evdev.go11:29
mborzeckipedronis: hmm right, the only concern i have is that keyEvent is a concrete type, so it's kind of shared between the consumer and producer, not sure if i'm explaining this right :)11:35
pedronismborzecki: you are, but is not a concern in go11:35
pedronisthe interfaces exist mostly for testing, no?11:36
pedronismborzecki: it's very unsual for a consumer to define structs it wants11:36
mborzeckipedronis: yeah, well, maybe i'm trying to complicated it too har :)11:36
pedronismborzecki: let's put it this, way if you tried to stick evdev.go in its own package as is, it wouldn't work11:37
pedronismborzecki: does this ^ remark makes sense? I'm also happy to HO if I'm still confusing you11:42
mborzeckipedronis: it's ok, i'm overcomplicating this, thought about a scenario when evdev went to a separate pacakge and KeyEvent would either need to be an iface, or one package would need to import the other to get the definition, but it makes no sense to restructure it this way11:47
pstolowskicachio: hey! i've asked 2 questions under #815711:49
mupPR #8157: tests: using google storage when downloading ubuntu cloud images from gce <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8157>11:49
cachiopstolowski, hi, I'll take a look11:49
cmatsuokamborzecki: just out of curiosity, what's the time difference between running the chooser trigger check from initramfs or early in the normal system?11:54
mborzeckicmatsuoka: we can start the detection early, in my VMs it's like <1s earlier than early boot, but i guess it may be different on the actual devices11:56
mborzeckicmatsuoka: and there's also a question, whether we'd be able to execute the gadget-provided trigger hook in initramfs, which i suspect we won't11:56
mborzeckicmatsuoka: as i said during the standup, it's probably only to the benefit of a device that supports the trivial input method like keyboard or somesuch11:57
cmatsuokamborzecki:  I had this feeling that initramfs should execute really fast11:57
cmatsuokamborzecki: but yeah I don't know how slow real devices could be11:58
jdstrandzyga: hey, fyi, https://github.com/snapcore/snapd/pull/7980 is -><- close12:07
mupPR #7980: packaging,snap-confine: stop being setgid root <Created by zyga> <https://github.com/snapcore/snapd/pull/7980>12:07
zygajdstrand: cool12:15
zygajdstrand: I'll check soon12:16
zygajdstrand: going through coverity, last one to fix12:16
* pstolowski lunch12:20
mborzeckimoving back home12:20
mborzeckibbiab12:20
cmatsuokamvo: the enable failover tests failed for some reason12:25
zygajdstrand: https://github.com/snapcore/snapd/pull/816512:26
mupPR #8165: cmd/snap-confine: fix everything flagged by coverity <Created by zyga> <https://github.com/snapcore/snapd/pull/8165>12:27
mupPR snapd#8165 opened: cmd/snap-confine: fix everything flagged by coverity <Created by zyga> <https://github.com/snapcore/snapd/pull/8165>12:27
zygajdstrand: replied to setgid, I'll jump there immediately now as it's so low hanging and it could be done shortly12:30
zygajdstrand: the bigger branch is still in progress, I'm working on improving the testing setup and resolving the issue you highlighted in remote access to 18.04 system12:30
zygajdstrand: it is real but for reasons I need to dig deeper to understand12:30
zygamvo, jdstrand: FYI - I will make coverity scan automatic, this is just the first step12:31
zygajdstrand: btw, I think we got remarkably little things flagged by coeveity12:32
zygaCoverity12:32
zygaI was bracing for a huge list12:33
zygaIt is a really cool tool12:33
zygamvo: it seems core 20 needs more love for failover12:37
zygamvo: /bin/bash: line 89: /usr/bin/snap: No such file or directory12:37
* zyga small coffee & pączek break12:39
cmatsuokaPączki look delicious!12:41
zygacmatsuoka: it's fat Thursday here12:44
zygacmatsuoka: so pączki are everywhere12:44
pedronismvo: is it known that snapd snap ships the code in /snap ?12:52
cmatsuokapedronis: you mean the go source code? yes, but it's to be solved with ijohnson's patch to use base: core in snapcraft.yaml12:56
zygawe back up like real men12:59
zygaby shipping all code to each user in production12:59
zygaDear $NAME; This is not a scam. Can you send us your core snap back please? We lost all disks in a thunderstorm. kthxbye13:00
mborzeckire13:05
zygaI just jumped through initrd break=premount13:07
zygafun13:07
zygaI think I will head back13:15
zygabetter to have the standup in a private setting13:15
zygaI'll finish my coffee and walk home13:15
ijohnsonSpeaking of my branch it should be ready to land when the Snapcraft in candidate goes to stable13:19
mvocmatsuoka, zyga re core20 failover> it is strange, I ran this locally and it worked, oh well. needs investigation13:19
pstolowskimvo: your fix for master failed on preseed test during image download. Perhaps it would make sense to land cachio's 8157 first (and flag it with 'skip spread')13:21
sdhd-saschazyga: Can you explain what syntax/grammer of the mount command here has? And where receive the command then?13:29
sdhd-sascha```13:29
sdhd-saschaemit("  mount options=(bind, rw) %s -> %s,\n", bindFile, path)13:29
sdhd-saschaemit("  mount options=(rprivate) -> %s,\n", path)13:29
sdhd-saschaemit("  umount %s,\n", path)13:29
sdhd-sascha```13:29
sdhd-saschaI mean: who receives the message?13:29
sdhd-saschazyga: wait. it's in the comments. Thank you13:31
mvopstolowski: hm, I can cherry pick his fix into my PR13:34
pstolowskimvo: that works too13:36
zygare13:44
zygasdhd-sascha: that's apparmor mount specification13:44
zygasdhd-sascha: I'm glad you asked about grammar13:45
zygasdhd-sascha: that's about the only thing that is very thoroughly documented :)13:45
zygasdhd-sascha: go to http://manpages.ubuntu.com/manpages/bionic/man5/apparmor.d.5.html13:45
zygasdhd-sascha: and check the MOUNT RULE = part13:46
cmatsuokamvo: Thursdays SUs are in a different room now?13:55
sdhd-saschazyga: :-) thank you, again.13:56
sdhd-saschai'm looking for a way to mount when a classic snap service restarts. Think the install hook is not enough.13:56
zygasdhd-sascha: can you summarize as to what you want to detect and what you want to do in response please13:56
mvocmatsuoka: not really, sorry, silly calendar13:57
zygahmm13:59
zygashould I join what meet proposes13:59
zygaI'm in the standp13:59
zyganobody there13:59
cmatsuokazyga: use the regular one I guess13:59
zygacan you spare the link14:00
sdhd-saschazyga: the github-runner, needs write access in the same directory. so i create /var/lib/runner/$SNAP_NAME/_work directory on install-hook. Then i want to be sure, that this is mounted to $SNAP/usr/lib/runner/_work14:00
zygaI just go to meet.google.com14:00
zygamvo: can you share the correct link14:00
sdhd-saschazyga: it's classic, because i don't know what the github-action-runner-scripts can do14:01
sdhd-saschazyga: on classic, the layout in snapcraft.yaml is not executed...14:02
zygasdhd-sascha: layouts don't do anything during classic confinement14:02
* sdhd-sascha nod14:03
sdhd-saschaWould be nice if layouts in classic mode would also work.14:04
zygasdhd-sascha: what would you expect them to do?14:04
zygasdhd-sascha: change the host?14:04
zygasdhd-sascha: what if two snaps want to have $SNAP/foo in /usr/lib?14:05
zygasdhd-sascha: who wins?14:05
sdhd-saschamaybe, limit the mounts to `/var/lib/runner/$SNAP_NAME/$SNAP_REVISION/_work` like above14:05
zygasdhd-sascha: what is /var/lib/runner?14:06
sdhd-saschai mean, without runner14:06
mupPR snapd#8166 opened: cmd/snap-bootstrap: create a new parser instance <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8166>14:06
zygasdhd-sascha: I think this is weird, a classic snap can just mount stuff there14:06
zygasdhd-sascha: perhaps I'm missing something but I think layouts are not meant for this14:06
sdhd-saschaok14:07
sdhd-saschathank you14:07
zygasdhd-sascha: layouts take something from the snap and put it somewhere in the view of the snap14:07
zygasdhd-sascha: but none of that changes the host for real14:07
mborzeckicmatsuoka: btw. could we just mkfs rather than delete/remove partitions?14:24
cmatsuokamborzecki: right, I think we could just redeploy them instead of messing with the partition table14:27
mborzeckicmatsuoka: just look at the attributes to know which ones we can wipe14:27
cmatsuokamborzecki: let's try that14:29
ackkmvo, hi, sorry to ping again wrt #1817276, is there anything I can capture when this happens to help debugging? I see it quite frequently locally. it might be made worse by the fact that my laptop is a bit old14:36
mupBug #1817276: snapfuse use a lot of CPU inside containers <maas> <snapd:Fix Released by mvo> <https://launchpad.net/bugs/1817276>14:36
mvoackk: I need to appologize for this one, we still have not invstigated how this can happen14:39
cmatsuokamborzecki: I blocked the PR while I try those changes, but I'll work on the TPM cmdline measuring first14:40
mborzeckicmatsuoka: cool, added a note there too14:40
ackkmvo, np. I can't change the status of that bug. does it make sense to reopen or should I file a new one ?14:41
pedronismborzecki: ping me when the chooser PR is ready for re-review14:42
mvoackk: let me reopen it14:42
ackkmvo, thanks14:42
mvoackk: reopened and updated the description. let me try to investigate again14:43
mvoijohnson: do you think you could have a look at 8161 ? it's green and seems to fix the snapd failover, if it's not wrong I'm in favor of merging and then we can merge your improvements next?14:48
mvoijohnson: it should unblock all the other PRs we have open14:48
ijohnsonmvo:  sure14:50
ackkmvo, thanks14:50
ijohnsonmvo: approved14:53
mvopedronis: thanks for letting me know about the /snaps subdir in the snapd snap - I (strongly) suspect that snapcraft is acting silly here, will investigate14:54
ijohnsonmvo: it's fixed in modern snapcraft14:55
mvoijohnson: aha! so we just need to land your PR :) ?14:55
ijohnsonmvo: my PR open right now fixes that, but we just need to wait until snapcraft 3.10 on candidate channel is promoted to stable14:55
ijohnsonyes14:55
mvoijohnson: I guess I could change our build recipe to use snapcraft from candidate?14:55
mvoijohnson: let me try this14:56
ijohnsonmvo: yes that would let us land sooner, not sure if that's okay to build snapd that gets released with candidate? depends on how much you trust sergiusens I guess :-)14:56
ijohnsonmvo: the spread test we have is already using candidate14:56
ijohnson(but that doesn't release anywhere, just makes sure it builds)14:56
mvoijohnson: I trust sergiusens a lot - plus the extra testing snapcraft would get this way is probably good14:57
ijohnsonsure sounds good then :-)14:57
mupPR snapd#8161 closed: tests: set StartLimitInterval in snapd failover test <Test Robustness> <⚠ Critical> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8161>14:57
sergiusensijohnson: I am mostly waiting on feedback from field, but I probably won't be releasing this week14:57
mvosergiusens: I switched our default snapd builds to use the snapcraft from candidate now, I think this makes sense anyway for testing your stuff earlier14:58
mvoijohnson: so just to double check 7904 will work with snapcraft 3.10 in candidate? so I can remove the blocked label?14:59
ijohnsonmvo: yes14:59
mvoijohnson: awsome15:00
sergiusensthanks mvo, that will help!15:02
mborzeckizyga: https://github.com/snapcore/snapd/pull/8165/files#r38205730515:06
mupPR #8165: cmd/snap-confine: fix everything flagged by coverity <Created by zyga> <https://github.com/snapcore/snapd/pull/8165>15:06
mborzeckizyga: found an example of use in qemu: https://github.com/qemu/qemu/blob/master/scripts/coverity-model.c15:07
zygamborzecki: looking15:08
mborzeckicmatsuoka: can you take a quick look at https://github.com/snapcore/snapd/pull/8166 ?15:21
mupPR #8166: cmd/snap-bootstrap: create a new parser instance <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8166>15:21
mvoackk: in the bugreport you write this happens inside a bionic container, this is puzzling. however it seems like I can explain why it's happening inside a disco container. but it's a bionic container?15:22
mvoackk: anyway, I will dig a bit deeper, if you have access to the container the output of "apt list squashfsfuse" would be nice15:22
mvoackk: to the problematic container15:23
ackkmvo, well yeah I've seen it in bionic, but also on focal while testing maas upgrade from deb to snap15:27
mvoackk: ok, I keep diging15:28
ackkmvo, in this scenario I launch a bionic container, install maas from deb, do-release-upgrade to focal which updates to the transitional deb (which installs the snap). once services in the snap start, snapfuse eats all cpu (along with python for the services)15:28
mvoackk: in a meeting right now, will get back to you (and try this out)15:43
sdhd-saschaIs it possible to backup a snap installation and restore them on a new installation as blob ?15:52
sdhd-saschaIt could be faster on github-actions if the installation of "snap", "lxd", "snapcraft" could be shortend with a tar-package15:53
zygasdhd-sascha: ish, it's not trivial15:54
* zyga is busy and won't respond now, sorry15:54
cmatsuokamborzecki: thanks, that's very useful!15:56
* mborzecki figures it's better to rebuild initramfs before repacking the kernel snap /o\15:58
mvoackk: just one more question - what is your "host" os that you run lxd on ?16:03
mvoackk: just to make sure I get a realistic reproducer16:03
ackkmvo, currently eoan16:03
ackkmvo, lxd backend is btrfs (if that matters)16:03
ackk(lxd 3.20)16:04
mvoackk: thank you16:05
ackkmvo, np, let me know if you need any more info16:05
* cachio lunch16:08
mvoijohnson: 7904 is green :)16:09
mvocan someone do a second review on 7904 please?16:09
zygay16:10
ijohnsonmvo: pstolowski reviewed and approved it a while ago16:10
ijohnsonbut sure more reviews is never a bad idea16:10
ijohnson:-)16:10
zygamvo: what does adopt-info snapd do?16:11
zygaI remember the concept16:11
mupPR snapd#8163 closed: tests: enable snapd-failover on uc20 <Simple 😃> <UC20> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8163>16:11
zygabut what's the data source for "snapd"?16:11
mvozyga: it means to get info like version from the snapd part16:12
zygaaha, I see16:12
zygabut where is the part defining anything that is adopted?16:12
zygais that set-version snapcraftctl?16:12
ijohnsonzyga: the data source for snapd when not specified defaults to "."16:15
ijohnsonerr for any snapcraft part rather16:16
ijohnsonzyga: `adopt-info: snapd` means get metadata for the snap from the part named "snapd", which is a bit counter-intuitive since the whole snap is named snapd and we also have the magic `type: snapd` so it's not clear if it's a special value or just a part16:18
zygayeah16:18
ijohnsonzyga: if you like I could rename the part from snapd to `snapd-deb` as that's really what the part is building16:18
zygaa comment would be nice16:18
zygafollowup material16:18
ijohnsoncool, yes I would prefer a followup if that's okay16:18
* ijohnson is not feeling lucky16:19
* ijohnson about tests today16:19
zygahahaha16:19
mupPR snapd#8167 opened: o/standby: add SNAPD_STANDBY_WAIT to control standby in development <Created by pedronis> <https://github.com/snapcore/snapd/pull/8167>16:19
mupPR snapd#7904 closed: snapcraft.yaml: use build-base and adopt-info, rm builddeb plugin <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/7904>16:22
mvoackk: one more question - in the bugrpeort you say that you see squashfuse generating a lot of cpu, it's squashfuse (and not snapfuse), right?16:22
ackkmvo, let me confirm16:23
ackkmvo, it seems it's actually snapfuse16:26
mvoackk: that's interessting16:26
mvoackk: the bugreport says "after installing core and rebooting". this is installing core in the container, I suppose but then rebooting the host?16:27
ackkmvo, no, rebooting the container, but that might be a red herring, I tried reproducing it that way and didn't manage to. so it might have been something else16:27
ackkmvo, what I did right now, though is just to install the maas snap, and initialize it16:27
ackknot a clean container, just my dev onne16:27
ijohnsonpedronis: mvo: xnox mentioned to me a while back that we should consider upgrading the build-base of the snapd snap to build a newer libc/other tools, thoughts on this? I don't think we need to do this now, but I could add a TODO:UC20: to the snapcraft.yaml for the snapd snap so we don't lose this suggestion16:28
ackkmvo, I take the more I/O the application does, the more snapfuse has to work as well?16:28
mvoackk: yes16:28
xnoxijohnson:  if you do, you do need to check minumum kernel requirements from newer libc, and how it matches the platforms you support.16:29
mvoackk: I mean, snapfuse taking quite a bit of cpu is normal but we fixed using the wrong snapfuse binary a while ago and it puzzling that apparently this is not fully working for you16:29
ijohnsonalso xnox in other news, you should now be able to cleanly build snapd snap from git master with candidate snapcraft :-)16:29
ijohnsonxnox: hmm good point16:29
xnoxbut do note that trusty is no longer a support target for snapd16:30
ackkmvo, I haven't seen issues with other snaps in containers, but maas does quite a lot of IO even just when starting up / setting up, so maybe the issue shows there more prominently16:30
ackkmvo, plus, my laptop is old and non-nvme16:30
ackkmvo, btw, the original issue reported by BjornT was on zfs, mine is on btrfs, so I was wondering if COW filesystems might play a role16:31
mupPR snapcraft#2939 closed: pluginhandler: user directories scoped to partdir for snapcraftctl <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2939>16:31
ackkmvo, I have a focal machine, I can try spawning a container there and see if it it's different16:32
mvoackk: interessting, could be. I'm trying in a clean 19.10 VM, all freshly created in qemu and see snapfuse hoover around 40% cpu which seems not that unexpected16:32
ackkmvo, is that with maas?16:33
mvoackk: yes, with maas init16:33
mvoackk: I now see two snapfuse, one 65 the other 20%16:33
ijohnsonzyga: see ^ or v depending on how quick mup is16:33
mvoackk: I think you mentioned multiple ones with 100% ?16:33
mvoackk: but this is ext4, I need to try zfs/btrfs :/16:33
mupPR snapd#8168 opened: snapcraft.yaml: add comments, rename snapd part to snapd-deb <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8168>16:33
ijohnsonthere it is ^16:34
ackkmvo, no just one, plus multiple python processes also using quite a lot of cpu, but that's expected during init16:34
mvoackk: yeah, I see python too but was assuming that is maas :)16:34
mvoackk: just one> but 100% cpu?16:34
mupPR snapcraft#2947 closed: remote-build: pass through 'source-subdir' property <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2947>16:34
ackkmvo, correct16:34
mvoackk: ok, I think I need to re-do this with a different fs, ext4 seems to not show it (or I'm doing something wrong)16:35
pstolowskicachio: can you merge master into 8157?16:35
mvoackk: it runs for a long time, that is normal?16:37
ackkmvo, yeah, although snapfuse stealing cpu makes it worse16:37
mvoackk: yeah :(16:37
ackkmvo, fwiw in my container here maas init has finished but snapfuse is still running at 100%cpu16:39
mvoackk: will redo this now with a different fs16:39
mvoackk: uh, that's interessting16:39
ackkmvo, that's what always happens16:39
mvoackk: can you check where snapfuse comes from? it should be /usr/bin/snapfuse from inside the container16:39
ackkmvo, /usr/bin/snapfuse16:40
ackk(from /proc/pid/exec)16:40
mvoackk: yeah, that should be fine (well, "should")16:40
ackkerr, exe16:40
mupPR snapcraft#2943 closed: spread: capture developer debug information <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2943>16:40
* mvo add zfs or something to see if it makes a difference16:40
* ackk stops maas before laptop catches fire16:41
BjornTackk: let it burn and buy a new one already :)16:42
pedronismborzecki: 8136 needs a 2nd review16:42
ackkBjornT, because of free mentining it I'm now kinda waiting for gen816:43
mupPR snapd#8166 closed: cmd/snap-bootstrap: create a new parser instance <Simple 😃> <Created by bboozzoo> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/8166>16:45
mupPR snapcraft#2923 closed: requirements: Update PyYAML requirement to 5.3 <Created by NickZ> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2923>16:46
mupPR snapd#8155 closed: tests: mv ubuntu-core-snapd{,-failover} to core/ suite <Test Robustness> <⚠ Critical> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/8155>16:47
pedronismvo: #8153 needs more work16:51
mupPR #8153: [RFC] "snap run --explain" with different formating <Created by mvo5> <https://github.com/snapcore/snapd/pull/8153>16:51
mborzeckipedronis: cmatsuoka: i've updated #815616:52
mupPR #8156: cmd/snap-bootstrap: subcommand to detect UC chooser trigger <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8156>16:52
mborzeckipedronis: as for naming, snap-boostrap core-chooser-trigger and snapd.core-chooser-trigger.service ?16:54
ackkmvo, I started a bionic container and installed maas there on my other machine which is running focal, snapfuse is not using cpu there16:55
cachiopstolowski, done16:55
cachiopstolowski, thanks for the heads up16:55
pedronismborzecki: s/core/recovery/16:56
mborzeckipedronis: recovery-chooser-trigger?16:56
mvoackk: oh? so on focal things are working for you?16:56
ackkmvo, it seems so, I wonder what happens if I update my laptop16:56
pedronismborzecki: yes16:57
pstolowskicachio: yw16:59
mvoackk: but I can reproduce the "keeps hogging cpu" issue in bionic17:02
ackkmvo, oh, "good"17:02
pstolowskicachio: i think only 19.10 and 20.04 images need to be re-downloaded often17:04
mvoackk: but I see a ton of run-supervisorctl restart <lots-of-services> in my ps output, so the snapfuse might be legit traffice because of all this activity in the container?17:05
mvoackk: like literally >200 supervisorctl restart17:06
mvoackk: is that expected? also it seems like its not actually restarting :(17:06
mvoackk: and the number fluctuates17:06
mvoackk: like I had 206 some moments ago, now 211, then 214, 21517:06
mvoackk: now 21717:07
pedronis#8130 and #8136 could use a 2nd review17:07
ackkmvo, you mean there are that many processes in parallel running?17:08
mupPR #8130: overlord, state: don't abort changes if spawn time before StartOfOperationTime (2/2) <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8130>17:08
mupPR #8136: boot: write current_kernels in bootstate20, makebootable <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8136>17:08
mvoackk: correct17:09
mvoackk: I added that to the bugreport too17:09
ackkmvo, can you check /var/snap/maas/common/log/supervisor-run.log ?17:10
mvoackk: I see a lot of waiting/stopped/spawned17:11
mvoackk: I can try to scp but i'm at >1600 process right now and I think this will soon OOM17:11
mvoackk: I am using 2.7/edge maybe edge was not a good idea :)17:11
ackkmvo, you might wanna snap stop maas17:11
zygamaster is fixed, right?17:11
ackko17:11
pedroniszyga: I have seen green things17:12
ijohnsonzyga: snapd-failover should be good now with mvo's PR17:12
ackkmvo, oh, yeah although I don't think there's much difference with stable17:12
zygasuperb news17:12
ijohnsoncan't speak to other things17:12
zygathank you everyone who pushed towards fixing that bug :)17:12
mvoackk: http://paste.ubuntu.com/p/MkgJwgfhnf/17:12
mvozyga: yeah, master should be green17:13
mvozyga: thank ijohnson mostly17:13
zygaThank you ijohnson :)17:13
mupPR snapd#8164 closed: snap: use the actual staging snap-id for snapd <Simple 😃> <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/8164>17:13
mvoackk: should I attach the log to the bugreport too?17:13
mvoackk: I wonder if maybe this is causing this heat/cpu issue17:13
ackkmvo, if you could grab a tarball of the whole log/ dir it would be great17:14
ackkmvo, I can take a look at that17:14
mborzeckipedronis: cool, pushed the naming update to #8156, it'd be great if mvo, cmatsuoka or ijohnson could take a look17:15
mupPR #8156: cmd/snap-bootstrap: subcommand to detect UC chooser trigger <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8156>17:15
ijohnsonmborzecki: I'll try to take a look today, it was on my queue of things to look at17:15
cmatsuokamborzecki: will check asap, will finish a debug run here first17:16
mvoackk: sure, I will do and attach to the bug(?). it looks huge though17:16
mborzeckiijohnson: cmatsuoka: if there's anything super silly feel free to push a patch, i'll pick it up in the morning17:16
ackkmvo, yeah if you don't want to attach it just put it somewhere and I can download it17:17
ijohnsonmborzecki: sounds good17:17
mvoackk: the gzip version of this dir is ~1.2Gb17:19
ackkoh boy17:19
mvoackk: yeah, I will upload to some gdrive, not sure if LP will like it if I attach to the bugreport17:20
ackkheh17:22
mupPR snapd#8169 opened: [wip] tests/many: don't use StartLimitInterval, StartLimitBurst anymore <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8169>17:23
ackkmvo, thanks, I think you're probably now hitting some bug17:25
ackkmvo, how did you reproduce the issue btw? just installing in a bionic container?17:25
mvoackk: correct. it's outlined in the original bug. I did use a clean 19.10 eon VM with 8g ram17:25
mvoackk: then installed snapd and lxd17:25
mvoackk: then lxd init with btfs as backend17:26
ackkmvo, weird, on this focal machine I don't even see snapfuse using CPU while maas init is running17:26
mvoackk: then created an "lxc launch images:ubuntu/18.04 container" and "lxc exec <image> bash" and installed maas in there and ran "maas init"17:26
mvoackk: that's pretty cool, at least it means this problem will go away :)17:27
zygamvo: snap-confine bug?17:27
mvoackk: but yeah, frustrating17:27
mvozyga: do you have more details?17:27
ackkmvo, yeah I hope so. I'll try upgrading my laptop to focal, the installer was crashing on me before, have to try a more recent daily17:27
zygamvo: no, I'm asking what this is about but now I see this is the snapfuse perf bug17:27
mvozyga: aha, yes, that's correct17:27
mvozyga: trying to reproduce their issue but no luck and apparently hitting a different bug17:28
zygamvo: what did you hit?17:28
ackkmvo, it might be related, because I also usually see supervisord-managed process restarting a lot, but never seen so many running supervsiorctl calls17:29
mvozyga: I don't know but it's some sort of (slow) fork bomb17:29
mvoackk: interessting!17:29
ackkmvo, I thought the restarts would be beacuse of timeouts caused by processes being slowed down by snapfuse17:30
mvoackk: could be but my VM is pretty fast, it's an ssd and has enough ram and the cpu load is not that high. still a possiblity of course17:32
mvoackk: shared my dir with you, please let me know if you need anything else before I stop this vm again17:37
mupPR snapd#8170 opened: snap-preseed: support for preseeding of snapd and core18 <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8170>17:39
ackkmvo, can you try starting the maas snap again? just to confirm if it starts happening again17:48
ackkmvo, (downloading logs as we speak)17:50
mupPR snapd#8167 closed: o/standby: add SNAPD_STANDBY_WAIT to control standby in development <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/8167>17:50
ijohnsonpstolowski: reviewed 800317:52
pstolowskiijohnson: thank you! if it's green then i'd like to land it and address your suggestions in a followup; if it fails then i'll push to this PR17:57
ijohnsonpstolowski: sounds good17:57
ijohnsonseems travis is clogged up with jobs :-/18:07
=== ijohnson is now known as ijohnson|lunch
pedroniswe opened too many new PRs especially considering that master was red18:09
ijohnson|lunchyeah but this actually happens pretty regularly in the afternoon my TZ, not sure why, but my thinking is that everyone in EU submits things before they log off18:10
=== ijohnson|lunch is now known as ijohnson
* ijohnson -> dentist18:44
NickZcheck yo teef18:50
kenvandinejdstrand: it looks like firefox isn't auto-connecting the pulseaudio plug anymore  https://twitter.com/thecalmsprings/status/123054292448987545718:56
kenvandinelove getting bug reports via twitter :)18:56
jdstrandkenvandine: that's weird. I grandfathered that18:58
* jdstrand fixes and investigates18:58
kenvandinejdstrand: thanks18:58
mvoackk: sure, let me retry this18:59
mvoackk: seems to be happening again, I see proxy, ntp, syslog19:00
mvoackk: then proxy again, it's now at 1519:00
mvoackk: now at 4619:04
mupPR snapd#8149 closed: snapmgr, backends: maybe restart & security backend options <Preseeding 🍞> <Created by stolowski> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/8149>19:08
* ijohnson -> back19:50
ijohnsoncachio: do you have a reliable way to reproduce that issue with snapd-failover you had the other morning, i.e. where in the logs snap-failure couldn't start snapd.socket because the socket file already exists? I have a fix for that I'd like to confirm if it works but I have never been able to reliably reproduce that error condition20:24
mupPR snapd#8165 closed: cmd/snap-confine: fix everything flagged by coverity <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8165>20:25
zygathis PR title is a lie, there are three left that weren't "new"20:38
zygaoh well20:38
cachioijohnson, yes I have20:44
cachiobut I am leaving in 2 mins20:44
cachioI ll be back in 2 hours20:44
cachioor just telegram me20:45
ijohnsoncachio: no problem, it can wait til tomorrow20:45
ijohnsonI will mention you on the PR I open I think, so if you could comment there tomorrow that would be great20:45
cachioijohnson, sure20:46
cachiowhen I am back I'll comment20:46
cachiosorry but I was working with the new arch image20:47
* cachio afk20:48
ijohnsoncachio: no problem, ttyl20:48
cmatsuokadebug tpm unlocking inside the initrd is painful21:01
ijohnsonsounds quite painful21:07
raerHope this works now: Hey snap people. A question regarding an issue (https://github.com/keepassxreboot/keepassxc-browser/issues/439) with the KeepassXC browser plugin (Firefox, Chromium) talking to KeepassXC through NativeMessaging. Afaik this won't work with snap-packaged apps, because of sandboxing / security.21:07
raerAh. That looks better...21:07
mupPR snapd#8171 opened: cmd/snap-failure/snapd: also rm snapd.socket if it still exists <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8171>21:08
raerWhat would be the "correct" solution to connect e.g. a browser plugin to an application ouside of the sandbox?21:09
ijohnsonraer: it depends, I suggest starting a post on forum.snapcraft.io where more folks will be available to look at your issue21:10
raerI'm reluctant to open that box...21:11
ijohnsonraer: AFAIK it's a known problem and oSoMoN has a bug filed somewhere about this issue with the chromium snap specifically21:11
raerhttps://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1741074 ?21:12
ijohnson(and notable oSoMoN is offline right now)21:12
mupBug #1741074: [snap] chrome-gnome-shell extension fails to detect native host connector <snap> <chromium-browser (Ubuntu):Triaged> <https://launchpad.net/bugs/1741074>21:12
raerdat :)21:12
ijohnsonit seems so21:12
ijohnsonI'm not familiar enough with the chromium snap to answer myself without more details about how plugins work with chromium and you're sure to find the right folks on the forum so that would be my recommendation21:13
ijohnsonyou could also try asking in EU tz morning time21:13
raerIt is broken for 2 years now21:13
raerA workaround is to install from repo, which is kind of ridiculous.21:14
raerMight ask oSoMoN in the morning. Not keen on signing up for yet another forum...21:15
ijohnsonif you have a LP account I believe that the ubuntu forums at discourse.ubuntu.com are tied to your LP account so you don't have to create another account21:16
ijohnsonand oSoMoN is around there as well21:16
raerok. thanks.21:17
NickZijohnson: did you ever get multipass working on the raspberry pi 4?21:20
ijohnsonNickZ: no I haven't tried for some time, but I think the required kvm options were added to the kernel now21:21
ijohnsonNickZ: IIRC there was some other problem with multipass not qemu/kvm related21:21
NickZyeah, I was lookinga t your issue21:21
NickZthat issue seems resolved now, but im having another one21:21
NickZI was just wondering if anyone else had gotten this working21:22
ijohnsonwhat's the new issue ?21:22
SaviqNickZ: do you know if Pi3 kernel also got those?21:23
NickZcan't say, haven't tried, although I heard that hardware virtualization is only supported on arm64 hosts21:24
NickZijohnson: https://github.com/canonical/multipass/issues/137621:24
SaviqNickZ: thanks, we'll have a look - it's weird that -machine would be required, maybe the default can be set somewhere21:26
ijohnsonhmm yeah I can't say I know anything about that but I bet Saviq can be of more help :-)21:26
NickZyeah, it seems unique for raspi hosts, dunno why21:27
mupPR snapd#8157 closed: tests: using google storage when downloading ubuntu cloud images from gce <Simple 😃> <Created by sergiocazzolato> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/8157>21:39
mupPR snapd#8172 opened: snapcraft.yaml: add python3-apt, tzdata as build-deps for the snapd snap <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8172>22:06
cmatsuokaijohnson: found the problem23:19
ijohnsoncmatsuoka: what was the problem?23:27
cachioijohnson, hey23:41
cachioijohnson, I'll run the test in a loop23:42
cachiothe fix is in the code? or in the tests?23:43
ijohnsoncachio the fix in the PR I mentioned was in the code for snap-failure23:44
ijohnsonAt least what I think is a fix23:44
ijohnsonI don't entirely understand the root cause but this should at least let snap-failure get past the issue and still recover23:45
cachioijohnson, mmmm, in that case it is not so simple to test23:45
cachiobecause I need to create an image with the fix23:45
cachiois this PR still open?23:46
ijohnsonYes one sec let me get you the link23:46
ijohnsoncachio: https://github.com/snapcore/snapd/pull/817123:46
mupPR #8171: cmd/snap-failure/snapd: also rm snapd.socket if it still exists <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8171>23:46
cachioijohnson, I'll run it in google23:48
cachioijohnson, the other way I can reproduce it is difficul because I need an image23:48
ijohnsonGreat, the tests all passed for that PR in Travis which is a good sign23:48
ijohnsoncachio it can certainly wait til tomorrow to23:48
ijohnson*too23:49
cachioijohnson, yes, I'll test it tonight and add a note in the PR23:49
ijohnsonAwesome thanks23:51
cachioyaw23:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!