| === cpaelzer_ is now known as cpaelzer | ||
| mborzecki | morning | 06:05 |
|---|---|---|
| zyga | Good morning | 06:22 |
| zyga | offie is a bit cold so I'll start from ... bed :) | 06:26 |
| === chihchun_afk is now known as chihchun | ||
| mborzecki | zyga: hey | 06:33 |
| zyga | hi :) | 06:33 |
| mborzecki | zyga: did you look in 2.36 apparmor or should I? | 06:34 |
| zyga | I looked | 06:34 |
| zyga | I was unable to reproduce it for ~5 hours | 06:34 |
| zyga | ideas on what is the magic trigger are very much welcome | 06:35 |
| * zyga wondres what caues commands with leading space to not be recorded in bash history | 06:54 | |
| zyga | mborzecki: I have something to share | 07:24 |
| zyga | just need a moment | 07:24 |
| zyga | mborzecki: https://pastebin.ubuntu.com/p/DS2BFfgYGp/ | 07:32 |
| zyga | mborzecki: pair with https://github.com/zyga/spread/tree/slices (though it contains extra stuff) | 07:33 |
| zyga | mborzecki: in spread.yaml add measure-each: ./measure.sh | 07:34 |
| zyga | mborzecki: needs tweaking for running in spread.yaml from snapd | 07:34 |
| zyga | because I chose different paths | 07:34 |
| zyga | should yield useful fixes very quickly | 07:35 |
| zyga | and may yield some eye-opening mistakes | 07:35 |
| zyga | oddly it fails on bintfmt misc being mounted | 07:36 |
| zyga | perhaps due to automount somewhere? | 07:36 |
| zyga | good morning mvo | 07:42 |
| mvo | hey zyga | 07:42 |
| mup | PR snapd#6194 closed: snap: make description maximum in runes, not bytes <Created by chipaca> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6194> | 07:48 |
| zyga | need to take the dog out, brb | 07:59 |
| === pstolowski|afk is now known as pstolowski | ||
| pstolowski | mornings | 08:01 |
| mvo | hey pstolowski - good morning | 08:05 |
| zyga | Hey Paweł | 08:08 |
| zyga | back | 08:16 |
| mborzecki | mvo: pstolowski: hey | 08:18 |
| mborzecki | guys, how about we un-manual fedora-28 until we fix f29? | 08:20 |
| mvo | mborzecki: sounds good to me | 08:22 |
| ackk | hi, is there a way to get stdout/stderr from a snap hook even if it doesn't fail? | 08:25 |
| mup | PR snapd#6211 opened: spread: run tests on Fedora 28 again <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6211> | 08:26 |
| zyga | ackk: pstolowski would know but I don't believe so | 08:27 |
| ackk | zyga, not even in some log? | 08:28 |
| zyga | yep | 08:28 |
| zyga | not anywhere | 08:28 |
| zyga | maybe some things are logged in tasks in case errors occur | 08:28 |
| zyga | but not sure | 08:28 |
| ackk | yeah you get stderr if the hook fails | 08:29 |
| pstolowski | ackk: not really, apart from manually writing it from withing a hook to a place snap can write to | 08:29 |
| ackk | but I have a case where it doesn't but things don't work as expected. so far I've been forcing a failure to get the output | 08:29 |
| zyga | mvo: hey | 08:29 |
| ackk | pstolowski, I see | 08:29 |
| zyga | mvo: some experiments from last week | 08:30 |
| zyga | https://github.com/snapcore/snapd/pull/6212 | 08:30 |
| zyga | since nothing can land because everything is perpetually red | 08:30 |
| mup | PR #6212: tests: measure kernel properties across tests <Created by zyga> <https://github.com/snapcore/snapd/pull/6212> | 08:30 |
| mup | PR snapd#6212 opened: tests: measure kernel properties across tests <Created by zyga> <https://github.com/snapcore/snapd/pull/6212> | 08:30 |
| zyga | mvo: to use this for real you need patched spread | 08:30 |
| zyga | but I wanted to share it | 08:30 |
| mvo | zyga: thanks, looking | 08:31 |
| zyga | it's super simple | 08:31 |
| zyga | I played with it on a toy project | 08:31 |
| zyga | trying to tune some of the things I capture | 08:31 |
| mvo | zyga: do you think the kernel is a problem during the tests ? or is this just an example? | 08:31 |
| zyga | all of the "measurables" are working fine in my toy project | 08:31 |
| zyga | mvo: I do think it is a problem | 08:31 |
| zyga | as in | 08:31 |
| zyga | look what is measured | 08:31 |
| zyga | I think all of those fail in snapd | 08:31 |
| zyga | without exceptions | 08:32 |
| zyga | some will need tuning in spread.yaml | 08:32 |
| zyga | some in tests | 08:32 |
| zyga | some in test helper code | 08:32 |
| ackk | pstolowski, unrelated question: I have a clean cosmic install, it seems gnome doesn't find icons for desktop snaps (like gnome-calculator, gnome-system-monitor,...). when I search they all show the default icon. is that a known issue? | 08:32 |
| mvo | zyga: silly question - should/could we run this after each restore? | 08:32 |
| zyga | mvo: that's exactly when this runs :-) | 08:32 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/6212/files#diff-3c11e56e5f7f82b0f352d0fe1851a107R429 | 08:32 |
| mup | PR #6212: tests: measure kernel properties across tests <Created by zyga> <https://github.com/snapcore/snapd/pull/6212> | 08:33 |
| zyga | it runs when the system state is supposed to be invariant | 08:33 |
| zyga | mvo: realistically I think it will be a long while before we can really merge this | 08:33 |
| zyga | but I wanted to share what I have | 08:33 |
| mvo | zyga: spread.yaml has "measure-each" - this is not in the released spread, yes? | 08:33 |
| zyga | using this locally can yield real fixes | 08:33 |
| zyga | mvo: that's right but that's not the problem | 08:33 |
| zyga | the problem is that we have many tests and complex setup that is broken | 08:33 |
| zyga | it will be a while before this turns green | 08:33 |
| zyga | mvo: the upside is that unlike causual spread debugging | 08:34 |
| mvo | zyga: what I mean is, could we run this even without measure-each by running it last in restore-each? | 08:34 |
| zyga | this is failing fast | 08:34 |
| zyga | mvo: no, they run at the wrong order | 08:34 |
| mvo | zyga: and we can't put them in the right order without changing spread? | 08:35 |
| zyga | no | 08:35 |
| zyga | but that's irrelevant really | 08:35 |
| pstolowski | ackk: i don't know that, you may want to ask around on #ubuntu-desktop | 08:35 |
| zyga | spread ownership is a separate topic | 08:35 |
| zyga | this will not pass yet | 08:35 |
| zyga | note I only enabled kernekl | 08:35 |
| zyga | *kernel | 08:35 |
| zyga | out of all the set of measuremetns | 08:35 |
| zyga | I think kernel will show the biggest offenders | 08:35 |
| zyga | stray processes | 08:36 |
| zyga | and stray mounted stuff | 08:36 |
| ackk | pstolowski, ok. thanks. do you know what would be the right lp project to report the issue? | 08:36 |
| zyga | mvo: next we should look at files, specifically /var | 08:36 |
| zyga | mvo: anyway, anyone can run this trivially | 08:36 |
| zyga | and see what they can carve of the mess plate | 08:36 |
| mvo | zyga: why can't it run on restore? pardon my ignornce | 08:39 |
| pstolowski | ackk: probably one of the affected projects on LP - e.g. https://bugs.launchpad.net/ubuntu/+source/gnome-calculator - you may also ask on the forum https://forum.snapcraft.io/ | 08:39 |
| zyga | mvo: it must run before any prepare and after all restore | 08:40 |
| zyga | mvo: there's no wat do do that | 08:40 |
| zyga | *no way | 08:40 |
| mvo | zyga: because spread provides no way to order things? | 08:41 |
| mvo | zyga: I mean in the toplevel spread.yaml we have "prepare-each" - we cannot run it there before we run anything else? | 08:42 |
| zyga | mvo: if you hand tune a project you might run it at the right order | 08:42 |
| zyga | but it can be broken by simple changes to unrelated parts | 08:43 |
| ackk | pstolowski, thanks | 08:43 |
| zyga | mvo: but that's not the problem, really | 08:43 |
| zyga | mvo: the problem is we have a backlog of things to fix in our suite first | 08:44 |
| Chipaca | moin moin | 08:44 |
| Chipaca | the staticcheck pr refuses to go green :-( | 08:45 |
| pedronis | mvo: hello, bit of a high number of open PRs | 08:45 |
| mvo | zyga: I think I'm missing things here, the "but it can be broken by simple changes to unrelated parts" is something I don't grok, I would assume this only breaks when we change the toplevel prepeare-each and do something else before the "measure", or what am I missing? | 08:45 |
| mvo | pedronis: yes :/ | 08:45 |
| mvo | zyga: let me play with this real quick, maybe then I understand things better | 08:46 |
| mvo | pedronis: and good morning - hope you had a good trip back | 08:46 |
| zyga | look at spread patch | 08:46 |
| zyga | see what it doeas | 08:46 |
| zyga | *does | 08:46 |
| zyga | it is easier to discuss things | 08:46 |
| zyga | I added it to the PR | 08:46 |
| zyga | I pushed a typo fix as well, my local measure.sh was slightly different and some editing broke it, it should work now | 08:47 |
| mvo | pedronis: also 2.36 is not passing tests right now and tests are flaky in general, not great times | 08:47 |
| * Chipaca hugs zyga | 08:47 | |
| zyga | Chipaca: hey :) | 08:47 |
| zyga | mvo: the patch is https://github.com/snapcore/spread/pull/47 | 08:48 |
| mup | PR spread#47: Add support for project-wide measure-each stanza <Created by zyga> <https://github.com/snapcore/spread/pull/47> | 08:48 |
| Chipaca | zyga: really sorry I answered your feedback on 6205 so grumpily on friday | 08:48 |
| mvo | zyga: maybe we need to talk during the standup I still don't get why we can't do it in prepare-each | 08:48 |
| mvo | zyga: I read the diff | 08:48 |
| mvo | zyga: anyway, let me play with this | 08:48 |
| mvo | zyga: I'm sure my ignorance will fade then | 08:48 |
| pedronis | mvo: will it help something immediate, this? | 08:48 |
| zyga | pedronis: no but fixes from running it will | 08:48 |
| zyga | I'm trying to land my stuff | 08:49 |
| zyga | but since it's always red | 08:49 |
| zyga | and normal debugging doesn't yield results | 08:49 |
| zyga | I wrote this | 08:49 |
| pedronis | zyga: https://github.com/snapcore/snapd/pull/6147 this is green and without reviews | 08:50 |
| mup | PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns 🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6147> | 08:50 |
| zyga | pedronis: god knows how many times I retriggered test last week | 08:50 |
| zyga | it's utterly wasteful | 08:50 |
| pedronis | zyga: mvo: we should have a HO | 08:51 |
| mvo | zyga: does measure.sh already raise the red flag(s) you describe in your spread PR? | 08:51 |
| mvo | pedronis: sure, happy to | 08:51 |
| zyga | mvo: as in does it really fail? | 08:52 |
| mvo | zyga: aha, yes - I see it now | 08:52 |
| pedronis | pstolowski: hi, how it's going with hot plug and conflicts? | 08:52 |
| pedronis | zyga: this one fails very consistently in one test fwiw: https://api.travis-ci.org/v3/job/454933150/log.txt | 08:53 |
| pstolowski | pedronis: hey, i need to push the changes, will do in a moment | 08:53 |
| pedronis | zyga: mvo: can we do a HO now? | 08:53 |
| zyga | pedronis: what's the PR link if I may ask? | 08:53 |
| zyga | pedronis: yes | 08:53 |
| mvo | pedronis: sure | 08:53 |
| pedronis | zyga: #6149 | 08:53 |
| mup | PR #6149: cmd/snap-confine: capture initialized per-user mount ns <Per-user mount ns 🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6149> | 08:54 |
| zyga | pedronis: did you notice the comment on that PR? | 08:54 |
| zyga | pedronis, mvo: so shall we join the standup HO | 08:55 |
| pedronis | zyga: which comment? | 08:55 |
| zyga | https://github.com/snapcore/snapd/pull/6149#issuecomment-438771770 | 08:55 |
| mup | PR #6149: cmd/snap-confine: capture initialized per-user mount ns <Per-user mount ns 🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6149> | 08:55 |
| zyga | it depends on another PR going in first, I proposed it for full context | 08:56 |
| pedronis | zyga: ok, we do really need this HO | 08:56 |
| zyga | stuck in HO again | 08:57 |
| zyga | hmmm | 08:57 |
| zyga | can we please try meet, I don't know how to fix this | 08:57 |
| zyga | pedronis, mvo: ^ | 08:58 |
| zyga | tried logging out and back in | 08:59 |
| zyga | trying with my personal account | 08:59 |
| zyga | no luck | 08:59 |
| zyga | I gave up | 09:00 |
| zyga | pedronis: can we please try meet? | 09:01 |
| mvo | zyga: sure | 09:01 |
| zyga | thanks, sorry, I really don't know what's wrong with HO | 09:01 |
| pedronis | zyga: mvo: https://meet.google.com/bwv-hykn-thy | 09:01 |
| zyga | same browser, same network | 09:01 |
| zyga | thanks | 09:01 |
| mup | PR snapd#5792 closed: [RFC] {config,snap}state: add new refresh.metered=force option and flip default <⛔ Blocked> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/5792> | 09:15 |
| mup | PR snapd#6174 closed: many: add snap debug refresh-catalogs <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6174> | 09:15 |
| mup | PR snapd#6184 closed: perf: add performance helpers <Performance 🚀> <⛔ Blocked> <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6184> | 09:15 |
| mup | PR snapd#6110 closed: spread.yaml: add openSUSE Tumbleweed <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6110> | 09:16 |
| mup | PR snapd#6209 closed: run-checks: discard test good/bad banner <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6209> | 09:16 |
| mup | PR snapd#6212 closed: tests: measure kernel properties across tests <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6212> | 09:16 |
| mup | PR snapd#6111 closed: packaging/opensuse: move most logic to snapd.mk <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/6111> | 09:17 |
| mup | PR snapd#6163 closed: interfaces: tweak deny-auto-connect policy tests <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/6163> | 09:17 |
| mup | PR snapd#6164 closed: wrappers: rename the desktop file to their apps <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/6164> | 09:17 |
| === epod is now known as luk3yx | ||
| mup | PR snapd#6166 closed: tests: run gpio test on core18 <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/6166> | 09:18 |
| mup | PR snapd#6205 closed: many: staticcheck fixes <Created by chipaca> <Closed by chipaca> <https://github.com/snapcore/snapd/pull/6205> | 09:18 |
| Chipaca | zyga: any news on the 2.36 fix? | 09:19 |
| zyga | no | 09:19 |
| zyga | I cannot reproduce it! | 09:19 |
| zyga | but I was running main in a loop | 09:19 |
| zyga | out of 2.36 release branch | 09:19 |
| zyga | perhaps it is broken by other suites? | 09:19 |
| Chipaca | pstolowski: mborzecki: #6104 has half a review from both of you :-) | 09:20 |
| mup | PR #6104: snapctl: add "services" <Created by chipaca> <https://github.com/snapcore/snapd/pull/6104> | 09:20 |
| Chipaca | zyga: what is 'running main in a loop'? | 09:20 |
| zyga | running the main suite in a loop | 09:21 |
| Chipaca | zyga: how were you running it? | 09:23 |
| === bloodearnest_ is now known as bloodearnest | ||
| === pstolowski_ is now known as pstolowski | ||
| === davdunc_ is now known as davdunc | ||
| === clobrano_ is now known as clobrano | ||
| === marosg_ is now known as marosg | ||
| === benoitc_ is now known as benoitc | ||
| === diddledan_ is now known as diddledan | ||
| === mwhudson_ is now known as mwhudson | ||
| zyga | Chipaca: spread running xenial main suite out of the 2.36 release branch | 09:33 |
| zyga | I'm running again without just main (so all suites) | 09:33 |
| zyga | let's see if I can hit this | 09:33 |
| Chipaca | zyga: in the run I have here the failing tests didn't run non-main tests before failing | 09:33 |
| Chipaca | zyga: (search for /runs) | 09:34 |
| zyga | :/ | 09:34 |
| zyga | so why did it not fail for me | 09:34 |
| zyga | eh eh | 09:34 |
| Chipaca | zyga: ＭＡＧＩＣ | 09:34 |
| zyga | let's see if I'm more unlucky today | 09:35 |
| mvo | zyga: fwiw, I ran http://paste.ubuntu.com/p/FqXw8pMSVV/ but get tons of errors after the lxd test, it changes the module/sysctl state quite a bit | 09:36 |
| mvo | zyga: anyway, running 2.36 now to see what is going on there | 09:36 |
| zyga | mvo: I'm also looking at 2.36 now | 09:38 |
| zyga | we can chat after that one is fixed | 09:38 |
| pedronis | mborzecki: hi, so far the loop devices mount bug, if we have a reproducer and are waiting for answers from upstream? is that right? | 09:40 |
| pedronis | s/far/for/ | 09:40 |
| pedronis | s/if// | 09:40 |
| Chipaca | pedronis: https://github.com/systemd/systemd/issues/10872 | 09:45 |
| pedronis | Chipaca: thx | 09:47 |
| pedronis | it actually was also in the card | 09:47 |
| zyga | 2.36 tests running... so far green :( | 09:49 |
| mborzecki | pedronis: yup, in the card too | 09:52 |
| pedronis | mborzecki: saw that now, moved/updated the card to reflect status, thank you | 09:53 |
| mup | PR snapd#6196 closed: many: validate title <Created by chipaca> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/6196> | 09:54 |
| zyga | mvo: any luck with 2.36? can you trigger it? | 09:56 |
| Chipaca | mborzecki: I think #6183 is gtg (github only counts one of its +1s but it does have two) | 09:57 |
| mup | PR #6183: sanity, spread, tests: add CentOS (2.36) <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6183> | 09:57 |
| Chipaca | (as of just now) | 09:57 |
| mvo | zyga: I just got a failure this second | 09:57 |
| zyga | how did you run spread? | 09:58 |
| mvo | Disable one alias for aliases snap | 09:58 |
| mvo | error: pattern not found, got:\nRemoved: | 09:58 |
| mvo | - aliases.cmd2 as alias2 | 09:58 |
| mvo | in 2018-11-26 10:57:46 Debug output for google:ubuntu-16.04-64:tests/main/parallel-install-aliases : | 09:58 |
| zyga | hmmm | 09:58 |
| mborzecki | Chipaca: i was waiting for #6189 but no luck so far | 09:58 |
| mup | PR #6189: daemon, vendor: bump github.com/coreos/go-systemd/activation, handle API changes (2.36) <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6189> | 09:58 |
| mvo | zyga: looks very unrelated .( | 09:58 |
| zyga | that's not the issue that was een | 09:58 |
| zyga | yep | 09:58 |
| zyga | another random fluke | 09:58 |
| mvo | zyga: spread -v -debug google:ubnuntu-16.04-64 | 09:58 |
| Chipaca | mborzecki: ah drat | 09:58 |
| zyga | thanks | 09:58 |
| zyga | same here :/ | 09:58 |
| zyga | I'll get a coffee | 09:58 |
| mborzecki | Chipaca: that way Neal would be able to rebuild the package on his end | 09:59 |
| mvo | zyga: https://paste.ubuntu.com/p/FYqWkFmVbN/ <- looks also wrong at least at first glance | 09:59 |
| zyga | mvo: looks like stdout vs stderr | 10:00 |
| zyga | though | 10:00 |
| zyga | not sure | 10:00 |
| zyga | it looks like IO intertwined | 10:00 |
| zyga | no idea | 10:00 |
| mvo | zyga: yeah, I have not seen this I poke a bit more | 10:01 |
| mborzecki | wonder why auditd is not started in our fedora images on gcp, while the cloud image starts it by default | 10:03 |
| zyga | mvo: got it! | 10:04 |
| zyga | :D | 10:04 |
| zyga | https://www.irccloud.com/pastebin/jiEYTg2z/ | 10:04 |
| mvo | zyga: nice | 10:04 |
| zyga | Nov 26 10:02:44 nov260941-894159 audit[24801]: AVC apparmor="DENIED" operation="mkdir" profile="/snap/core/6013/usr/lib/snapd/snap-confine" name="/tmp/snapd.quirks_wlXkFF/" pid=24801 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0 | 10:05 |
| zyga | google:ubuntu-16.04-64 /var/lib/snapd/apparmor/profiles# pastebinit snap-confine.core.6013 | 10:06 |
| zyga | http://paste.ubuntu.com/p/xKvYgnBR2n/ | 10:06 |
| mvo | zyga: oh, we removed the quirks profile generation but not in 2.36 | 10:09 |
| zyga | yes | 10:09 |
| zyga | but we knew that | 10:09 |
| zyga | I said that last week when first reports came in | 10:09 |
| zyga | the question is why the profile is not matching 2.36 | 10:09 |
| zyga | mvo: the kernel profile != disk profile | 10:10 |
| mvo | zyga: yeah, exactly, why is there a mismatch | 10:10 |
| zyga | disk profile is correct | 10:10 |
| zyga | another cache bug? | 10:10 |
| mvo | zyga: uhhh | 10:10 |
| zyga | I'll get that coffee first | 10:10 |
| mvo | zyga: that sounds likely, oh boy | 10:10 |
| zyga | mvo: the measure state dumper I wrote can measure that too :) | 10:10 |
| zyga | brb | 10:10 |
| mvo | zyga: maybe we should enable that then instead of the kernel one first | 10:11 |
| mvo | zyga: can you get the timestamps of the kernel profile and the disk profile? | 10:12 |
| zyga | mvo: looking | 10:17 |
| zyga | kernel profile has no timestamp, looking at what is in the cache | 10:18 |
| zyga | http://paste.ubuntu.com/p/bjDm3gxXTz/ | 10:18 |
| zyga | that is /var/cache/apparmor/ | 10:19 |
| zyga | a bit odd | 10:19 |
| zyga | I have a hunch now, hold on | 10:20 |
| zyga | mvo: it's super interesting that we do skipReadCache | 10:21 |
| zyga | so | 10:21 |
| zyga | if we had actually got to the phase where snap-confine profile is loaded we would pass an argument to apparmor parser that instructs it to skip any cache reads | 10:21 |
| zyga | ah | 10:21 |
| zyga | sorr | 10:22 |
| zyga | ha | 10:22 |
| zyga | I see the bug now | 10:22 |
| zyga | one liner fix coming up | 10:22 |
| mvo | zyga: cool | 10:22 |
| mvo | zyga: curious to see what it is/was | 10:24 |
| pedronis | mvo: I put some comments in #6185 | 10:27 |
| mup | PR #6185: snap: add new `snap run --trace-exec` call <Performance 🚀> <Created by mvo5> <https://github.com/snapcore/snapd/pull/6185> | 10:27 |
| pedronis | Chipaca: we are getting failures of the funcarg completion tests with timeouts | 10:30 |
| Chipaca | pedronis: yes | 10:31 |
| pedronis | Chipaca: see for example https://api.travis-ci.org/v3/job/458809746/log.txt | 10:31 |
| pedronis | Chipaca: any idea what is happening? | 10:31 |
| zyga | mvo: 2nd depth to the problem | 10:33 |
| Chipaca | pedronis: no | 10:33 |
| zyga | mvo: it's not cache | 10:33 |
| * pstolowski is having network outage | 10:33 | |
| zyga | Nov 26 10:02:33 nov260941-894159 snapd[24445]: backend.go:312: cannot create host snap-confine apparmor configuration: cannot reload snap-confine apparmor profile: cannot load a | 10:33 |
| zyga | pparmor profiles: signal: terminated | 10:33 |
| pedronis | Chipaca: can you investigate when you have a bit of time? | 10:34 |
| Chipaca | pedronis: yes | 10:34 |
| Chipaca | pedronis: on it already | 10:34 |
| pedronis | thx | 10:34 |
| Chipaca | pedronis: I wish I'd get something as clean as that logfile though | 10:34 |
| Chipaca | :-) | 10:34 |
| pedronis | Chipaca: that zyga branch seems to be failing with that error or many of it | 10:35 |
| pedronis | all the time | 10:35 |
| Chipaca | ok, i'll try on that, thanks | 10:35 |
| zyga | mvo: so I though it was missing skipReadCache | 10:39 |
| zyga | mvo: but that's not it | 10:39 |
| zyga | the cache timestamp and profile timestamp warrant a cache skip | 10:39 |
| zyga | but apparmor parser was killed | 10:39 |
| zyga | and snapd ignores the error | 10:39 |
| zyga | why it gets killed is unclear yet, looking | 10:39 |
| zyga | mvo: I made the error fatal | 10:49 |
| zyga | but no clue as to what triggers it yet | 10:49 |
| zyga | more debugging in progress | 10:49 |
| zyga | another random failure: | 10:53 |
| zyga | panic in unit tests https://www.irccloud.com/pastebin/BaMV9MrJ/ | 10:53 |
| pedronis | that's definitely a bug or test flakiness, first time I see it tough | 10:57 |
| mborzecki | hmm, core snap pulled from the store does not have selinux labels in xattrs, but the tests repack the core snap and it picks up user_home_t in the process :/ | 11:00 |
| mvo | zyga: thanks, sorry, had a meeting reading backlog | 11:00 |
| zyga | k | 11:00 |
| zyga | backlog? | 11:00 |
| mvo | mborzecki: do we need to append -no-xattr to our mksquashfs when we repack? | 11:01 |
| mborzecki | mvo: iirc s-c was supposed to not use +s bur rely on xattr caps, not sure if that's true for the core snap now | 11:03 |
| Chipaca | mvo: you need to not append it for non-app snaps | 11:03 |
| Chipaca | (snap pack dtrt) | 11:04 |
| pedronis | mborzecki: yes, I remember was conversations around that | 11:04 |
| pedronis | not sure what was the outcome | 11:04 |
| Chipaca | https://github.com/snapcore/snapd/blob/master/snap/squashfs/squashfs.go#L325 fwiw | 11:05 |
| mvo | zyga: backscroll :) so it seems the issue is not entirely clear yet? | 11:05 |
| mborzecki | Chipaca: yup, that's the line | 11:05 |
| zyga | mvo: the origin is not, the mechanics are | 11:06 |
| zyga | mvo: we ignore the error from compiling the apparmor profile of snap-confine | 11:06 |
| mvo | zyga: uh, ok | 11:06 |
| zyga | the error is that apparmor_parser is killed with a signal | 11:06 |
| mvo | zyga: thanks, that sounds like a bug in itself | 11:06 |
| pstolowski | zyga: i'll take a look at that panic | 11:06 |
| zyga | mvo: the reason for the signal is unclear | 11:07 |
| Chipaca | WAT | 11:07 |
| zyga | I made the error not ignored | 11:07 |
| mvo | zyga: right, so no error output just a signal? | 11:07 |
| mvo | zyga: is the error reproducible? | 11:07 |
| pedronis | pstolowski: btw do ping me if there is something ready for me to review/re-review | 11:07 |
| Chipaca | https://imgur.com/ndV0USM | 11:08 |
| zyga | mvo: we collect output but there is none | 11:09 |
| zyga | mvo: ran it again and it passed | 11:09 |
| zyga | running it again until I hit the bug | 11:09 |
| pstolowski | pedronis: #6114 is ready; travis failure unrelated | 11:09 |
| mup | PR #6114: overlord/ifacestate: handler for hotplug-disconnect task <Hotplug 🔌> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6114> | 11:09 |
| Chipaca | sergiusens: ping | 11:09 |
| Chipaca | or was today a holiday in .ar | 11:10 |
| mvo | zyga: thank you! | 11:12 |
| zyga | mvo: one more observation from running measure.sh, we never remove any apparmor profiles | 11:15 |
| zyga | mvo: (running two loops) | 11:15 |
| zyga | as in, from the kernel | 11:15 |
| zyga | they stay there forever | 11:15 |
| pedronis | pstolowski: ok, thx, I'll look at it after lunch | 11:17 |
| mvo | zyga: aha, interessting, that sounds like a bug too | 11:17 |
| zyga | yes, fixed locally now | 11:17 |
| mup | PR snapd#6213 opened: interfaces: let NM access ifindex/ifupdown files <Created by alfonsosanchezbeato> <https://github.com/snapcore/snapd/pull/6213> | 11:20 |
| zyga | got the panic again | 11:23 |
| zyga | ... Panic: change 1 unexpectedly became unready (Do) (PC=0x45ECEE) | 11:23 |
| zyga | carrying on with debugging | 11:24 |
| pstolowski | zyga: running these tests in a loop, no panic (yet) | 11:25 |
| zyga | pstolowski: FYI, this is on 2.36 release branch | 11:25 |
| zyga | maybe fixed since | 11:25 |
| zyga | not sure | 11:25 |
| pstolowski | zyga: i doubt it, we haven't touched this area afaik | 11:25 |
| zyga | pstolowski: if you want to try spread it was google:ubuntu-16.04-64:tests/unit/go | 11:25 |
| pstolowski | zyga: thanks, will try (running net on my mobile atm, my isp is down) | 11:27 |
| zyga | try with -repeat or -count (forgot which one is real) | 11:27 |
| pstolowski | zyga: ok | 11:27 |
| zyga | mvo: reproduced again | 11:29 |
| zyga | let me inspect snapd | 11:29 |
| zyga | mvo: same Nov 26 11:27:30 nov261108-796129 snapd[9598]: helpers.go:187: cannot regenerate apparmor profile for snap "core": cannot create host snap-confine apparmor configuration: cannot | 11:30 |
| zyga | reload snap-confine apparmor profile: cannot load apparmor profiles: signal: terminated | 11:30 |
| zyga | Nov 26 11:27:30 nov261108-796129 snapd[9598]: apparmor_parser output: | 11:30 |
| pstolowski | zyga: i suspect task runner doesn't like the fact that we manually mess with task status in the test, and if i read change.go code correctly, it doesn't like that task status became out of sync with change status (a race provoked by SetStaus in the test?). we might have been lucky not to hit this earlier | 11:30 |
| zyga | (nothing, no output0 | 11:30 |
| zyga | mvo: I can load and compile the profile manually | 11:31 |
| zyga | then things work | 11:31 |
| zyga | mvo: my patch made us return an error but it just got logged still, something is also wrong | 11:31 |
| zyga | none of the changes recorded the error | 11:33 |
| zyga | mvo: again, we ignore setup errors in ifacestate | 11:34 |
| mvo | zyga: ok | 11:35 |
| zyga | fixed that too, trying again | 11:35 |
| mvo | zyga: thanks! | 11:35 |
| zyga | still no insight into what kills that process | 11:35 |
| zyga | my patch so far | 11:36 |
| zyga | http://paste.ubuntu.com/p/B688VzKY6S/ | 11:36 |
| zyga | mvo: hopefully with this run I will pinpoint the moment this breaks, maybe that will suggest something | 11:39 |
| mvo | zyga: ta | 11:41 |
| mvo | zyga: and fingers crossed that this yields some results | 11:41 |
| mvo | Chipaca: I get some :funcargs errors currently too, should we disable this test until this clearer what is going on or are you already close to a fix? | 11:42 |
| Chipaca | mvo: I can't yet see why it's failing | 11:43 |
| * zyga is happy to see people attacking test issues | 11:43 | |
| === tinwood_ is now known as tinwood | ||
| mup | PR snapd#6214 opened: spread.yaml: disable _/funcarg test for now <Created by mvo5> <https://github.com/snapcore/snapd/pull/6214> | 11:47 |
| zyga | pstolowski: I get the panic each time, wonder what upsets the timing now | 11:54 |
| pstolowski | zyga: haven't seen it yet. i see "main.go:158: Exiting on %!s(<nil>) signal." though, is that what you discussed above? | 11:55 |
| zyga | pstolowski: can you provide more logs please? | 11:55 |
| zyga | pstolowski: that's not the error I'm seeing though | 11:55 |
| zyga | but that is what Chipaca found last week | 11:55 |
| zyga | broken daemon shutdown | 11:55 |
| zyga | we should cherry pick the fix into stable | 11:56 |
| zyga | probably is proposed and cannot land | 11:56 |
| Chipaca | zyga: I closed that PR because it never got green | 11:56 |
| zyga | but I did not chcek | 11:56 |
| zyga | *check | 11:56 |
| zyga | heh :-) | 11:56 |
| Chipaca | restarted the tests all weekend :-( | 11:56 |
| Chipaca | but I'll open a new one with just that bit | 11:56 |
| Chipaca | or you can | 11:56 |
| Chipaca | :-) | 11:57 |
| zyga | Chipaca: my hands hurt after hitting restart for the billionth time | 11:57 |
| Chipaca | you monster | 11:57 |
| Chipaca | :-) | 11:57 |
| pstolowski | zyga: i don't think i can have more logs atm, running it on spread | 11:57 |
| zyga | k | 11:58 |
| zyga | mvo: when we restore in spread we are unpacking the tarball | 11:58 |
| zyga | so we are putting apparmor profiles on disk | 11:58 |
| pstolowski | https://www.irccloud.com/pastebin/nYunJ1EE/ | 11:58 |
| zyga | mvo: when we do that we may confuse the code that looks at the profiles to see what to do | 11:58 |
| pstolowski | zyga: ^ | 11:58 |
| zyga | pstolowski: if anything the log message needs improvements | 11:59 |
| zyga | mvo: we only restore the disk profiles, not the kernel state | 11:59 |
| === cpaelzer__ is now known as cpaelzer | ||
| zyga | mvo: no more progress yet, given that I see it very infrequently we may attempt to restart 2.36 tests on travis | 12:09 |
| zyga | mvo: no failure since introducing trivial changes not to ignore errros | 12:38 |
| sergiusens | Chipaca: pong? | 12:44 |
| pstolowski | mvo, pedronis interesting failure - i've just tried to test something on an old VM which was stuck on 2.25: https://pastebin.ubuntu.com/p/Rg4CTMrGDr | 12:44 |
| sergiusens | with no context 😉 | 12:44 |
| sergiusens | holiday was last week | 12:44 |
| sergiusens | today is only a major strike day 😛 | 12:45 |
| pedronis | pstolowski: interesting | 12:48 |
| pedronis | pstolowski: does installing hello-world works? | 12:49 |
| pedronis | Chipaca: I reviewed #6104 | 12:49 |
| zyga | pstolowski: report the bug or we'll forget soon | 12:49 |
| mup | PR #6104: snapctl: add "services" <Created by chipaca> <https://github.com/snapcore/snapd/pull/6104> | 12:49 |
| pstolowski | pedronis: yep, and afterwards it's all fine as expected | 12:50 |
| pstolowski | zyga: good point, doing | 12:50 |
| pedronis | ok | 12:50 |
| pedronis | pstolowski: it's a store bug btw, we cannot really change old versions | 12:51 |
| pedronis | mvo: could you review #6126 ? | 12:53 |
| mup | PR #6126: dirs, wrappers, overlord/snapstate: make completion + bases work <Created by chipaca> <https://github.com/snapcore/snapd/pull/6126> | 12:53 |
| pedronis | Chipaca: I added some questions in #6034 | 12:59 |
| mup | PR #6034: many: save media info when installing, show it when listing <Created by chipaca> <https://github.com/snapcore/snapd/pull/6034> | 12:59 |
| pstolowski | pedronis: reported https://bugs.launchpad.net/snapstore/+bug/1805140 | 13:06 |
| mup | Bug #1805140: 'snap find' on old version of snapd 2.25 produces error about base <Snap Store:New> <https://launchpad.net/bugs/1805140> | 13:06 |
| pedronis | pstolowski: thx | 13:11 |
| mup | PR snapd#6215 opened: tests: reset the snapd package to make the first test on the suite work properly <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6215> | 13:25 |
| pedronis | cachio: hi, what's the status of #5174 ? | 13:28 |
| mup | PR #5174: interfaces/default,process-control: miscellaneous signal policy fixes <Created by jdstrand> <Merged by jdstrand> <https://github.com/snapcore/snapd/pull/5174> | 13:28 |
| pedronis | gah, sorry | 13:28 |
| pedronis | cachio: I meant, #5714 | 13:28 |
| mup | PR #5714: tests: new test for cifs-mount interface <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5714> | 13:29 |
| mvo | pedronis: sure, I have a look now | 13:29 |
| zyga | cachio: so 6215 should the reset instead go to per-suite restore function? | 13:29 |
| mvo | zyga: hm, hm, no errors still? | 13:30 |
| cachio | pedronis, it was failing but not because of the test | 13:30 |
| mvo | pstolowski: looking | 13:30 |
| cachio | pedronis, I was waiting a fix for that, I'll update it and see the current status | 13:30 |
| pedronis | cachio: ok, it has a conflict now, thx | 13:30 |
| zyga | mvo: apart from the panic, no | 13:30 |
| zyga | mvo: I reproduced it twice in total today | 13:31 |
| zyga | "luck' | 13:31 |
| cachio | zyga, the problem is happening on the first completions test after run other test suite | 13:31 |
| cachio | zyga, because before we were calling the reset | 13:31 |
| zyga | cachio: and why should this not run in per-suite restore? | 13:32 |
| cachio | I zyga we need to split it and reorganize it a bit more so we dont need to do it at suite level | 13:33 |
| cachio | just on test level | 13:33 |
| zyga | not sure I follow | 13:33 |
| cachio | now I am trying to define the initial/end state at suite level, and test level | 13:33 |
| cachio | because we have a mix of things currently | 13:34 |
| cachio | zyga, conceptually, we should define a state for the whole environment when we run any suite | 13:34 |
| cachio | and a different one for the tests | 13:35 |
| zyga | mhm | 13:35 |
| zyga | yeah, I agree about that 100% | 13:35 |
| cachio | so then we make sure each suite and test leaves the same env on the restore | 13:35 |
| zyga | right | 13:35 |
| zyga | so what is wrong now? | 13:35 |
| cachio | but currently it is not happening 100% | 13:35 |
| zyga | what is the state we want to have at suite level | 13:35 |
| zyga | and at test level | 13:36 |
| cachio | now the first test in the suite is not getting the same env than the rest | 13:36 |
| pedronis | what changed? | 13:36 |
| zyga | I suspect https://github.com/snapcore/snapd/pull/6202/commits/8b174e8daec7b21f3091a0e99fb79467f56625ab | 13:37 |
| mup | PR #6202: tests: restore in restore, not in prepare <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6202> | 13:37 |
| cachio | yes | 13:37 |
| zyga | cachio: can you be specifc, do you know what is actually not the same now | 13:37 |
| cachio | when we moved the reset to the restore | 13:37 |
| pedronis | well, sounds like that needs reverting until we understand more | 13:37 |
| pedronis | for sure I'm seeing bizarre errors I have never seen | 13:38 |
| pedronis | before | 13:38 |
| cachio | the snapd state is restored on the reset | 13:38 |
| pedronis | yes | 13:38 |
| cachio | so | 13:38 |
| cachio | when we run the first test suite | 13:39 |
| cachio | the first test runs with the initial state with is ok | 13:39 |
| cachio | and as any test is calling the reset in its restore | 13:39 |
| cachio | all the tests receive the correct snapd state | 13:39 |
| pedronis | yes | 13:39 |
| pedronis | then next suite? | 13:40 |
| cachio | then when the suite calls its restore | 13:40 |
| cachio | it cleans snapd | 13:40 |
| cachio | then next suite is called | 13:40 |
| cachio | but the first test receives a different state | 13:40 |
| cachio | because the reset is not called | 13:40 |
| zyga | but ... ? | 13:40 |
| zyga | suite-a/restore => calls reset | 13:41 |
| zyga | suite-b/prepare => ... | 13:41 |
| pedronis | anyway I'm not sure why this is better than the old approach | 13:41 |
| zyga | suite-b?/test => runs in what was left by reset.sh | 13:41 |
| pedronis | even if the old approach was called confusing :) | 13:41 |
| cachio | zyga, suite-a/restore => is uninstalling snapd | 13:41 |
| Chipaca | I personally like the tests that do "snap try" in prepare, and "rm -rf underlying/dir" on restore | 13:42 |
| cachio | after the reset | 13:42 |
| pedronis | Chipaca: without remove ? | 13:42 |
| Chipaca | mhmm | 13:42 |
| cachio | zyga, so there we clean the state again | 13:43 |
| Chipaca | I don't know if snapd copes well enough with that to be in a sane state :-) | 13:43 |
| zyga | cachio: my point is that now each suite restore calls reset.sh, the change you are proposing is making the prepare.sh run the same thing | 13:43 |
| zyga | seems like a no-op | 13:43 |
| zyga | there must be a part missing | 13:43 |
| cachio | zyga, the change I did it is just a workaroud to fix the tests | 13:43 |
| pedronis | zyga: there is a prepare in suite | 13:43 |
| pedronis | between that restore | 13:43 |
| pedronis | and the suite | 13:43 |
| pedronis | in the old world | 13:43 |
| zyga | yes, I know | 13:44 |
| zyga | but that is all calling reset.sh | 13:44 |
| zyga | so? | 13:44 |
| zyga | we used to call it in prepare.sh | 13:44 |
| zyga | er | 13:44 |
| pedronis | yes, | 13:44 |
| zyga | sorry, we used to call it in suite prepare | 13:44 |
| pedronis | simpler I think | 13:44 |
| pedronis | if confusing | 13:44 |
| zyga | after suite prepre we run tasks, then suite restore | 13:44 |
| cachio | zyga, I think we need to reorganize a bit the prepare/restore at test suite level | 13:44 |
| zyga | I bet this will work | 13:44 |
| zyga | because we essentially revert that patch | 13:44 |
| zyga | but I think we don't understand why exactly (or perhaps just me) | 13:45 |
| pedronis | cachio: zyga: it's all good but given the state of things I would prefer we get back to a state that was known to work a bit more | 13:45 |
| zyga | because the sequence of calls to reset.sh seems to be the same | 13:45 |
| pedronis | before reorganizing again | 13:45 |
| cachio | zyga, the workaroud which I proposed is to run the reset at the begining of the test suite | 13:46 |
| pedronis | basically first revert https://github.com/snapcore/snapd/pull/6202 | 13:46 |
| mup | PR #6202: tests: restore in restore, not in prepare <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6202> | 13:46 |
| cachio | not for each test | 13:46 |
| zyga | cachio: I'd love to understand this better when time is not ticking | 13:46 |
| cachio | zyga, we have standup in 15 mins :) | 13:47 |
| zyga | oh, I need to take the dog out then | 13:47 |
| zyga | ttyl | 13:47 |
| cachio | zyga, let see that there | 13:47 |
| zyga | cachio: I would rather understand the details of what is wrong | 13:47 |
| zyga | revert the thing I pushed that broke master | 13:47 |
| zyga | and later on fix master so that we can clearly agree that the prepare/restore logic is ok | 13:48 |
| pedronis | yes, that's my preferred approach | 13:48 |
| pedronis | as well | 13:48 |
| mup | PR snapcraft#2411 closed: cli: snapcraft init with a base <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2411> | 13:48 |
| mup | PR snapcraft#2412 closed: schema: add support for title <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2412> | 13:48 |
| zyga | with perhaps revert even before the understand :) | 13:48 |
| pedronis | yes | 13:48 |
| zyga | sorry for breaking master for everyone | 13:48 |
| zyga | mvo: I will try with seed, still 0 reproducted cases | 13:49 |
| pedronis | cachio: can you prepare a revert of #6202 ? | 13:49 |
| mup | PR #6202: tests: restore in restore, not in prepare <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6202> | 13:50 |
| zyga | pedronis, cachio: perhaps just https://github.com/snapcore/snapd/pull/6202/commits/8b174e8daec7b21f3091a0e99fb79467f56625ab | 13:50 |
| zyga | the rest is not related | 13:50 |
| Chipaca | is the standup via hangouts still? | 13:52 |
| zyga | why? | 13:52 |
| zyga | are they broken for you? | 13:52 |
| Chipaca | zyga: weren't you unable to use it? | 13:52 |
| zyga | (asking for a friend :) | 13:52 |
| Chipaca | no, my browser works | 13:52 |
| Chipaca | :-) | 13:52 |
| zyga | yes, same thing today | 13:52 |
| zyga | chrome | 13:52 |
| zyga | video starts, then ... nothing | 13:52 |
| zyga | meet worked | 13:52 |
| zyga | I can do standup from phone | 13:52 |
| mvo | pedronis: thanks for your review on 6185 - do you want me to move some code to osutil/strace there ? or would you prefer to do that in a followup? | 13:53 |
| cachio | zyga, pedronis sure | 13:53 |
| pedronis | mvo: I think there, as I also said the code seems right but probably some better naming of thing/comment would help explaining but is doing | 13:54 |
| pedronis | s/but is/what is/ | 13:54 |
| mvo | pedronis: sure, happy to do that. thank yoU! | 13:54 |
| pedronis | there are probably also some assumptions around how exec are expected to happen | 13:54 |
| pedronis | that maybe need to be explicit | 13:54 |
| mvo | pedronis: I will move code around and use the opportunity to clean naming a bit more, I agree its not ideal | 13:54 |
| * mvo nods | 13:54 | |
| pedronis | thank you | 13:55 |
| pedronis | cachio: zyga: I made a note in 6125 about this | 13:58 |
| mup | PR snapd#6216 opened: tests: revert "tests: restore in restore, not prepare" <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6216> | 14:00 |
| geodb27 | People : hi ! | 14:02 |
| geodb27 | I suspect a problem on my kubuntu18.04 laptop with the "service" named "run-snapd-ns-lxd.mnt.mount" what are the common log-files or log-programs that can help diagnose a problem (or not) for snaps ? | 14:09 |
| zyga | geodb27: hey | 14:17 |
| zyga | in a call, please hold | 14:17 |
| zyga | that is not a service as you noticed | 14:18 |
| zyga | geodb27: can you tell me what the problems are | 14:19 |
| zyga | that file is a mount namespace | 14:19 |
| zyga | belonging to the lxd snap | 14:19 |
| geodb27 | of course I can zyga, thanks a lot for your help. | 14:19 |
| geodb27 | What I suspect is the following : my system boots, the /tmp is mounted. But, at one point, the snap lxd service is started and somehow bind-mounts something *OVER* the /tmp. | 14:20 |
| zyga | over the /tmp on the host? | 14:20 |
| geodb27 | yep. | 14:20 |
| zyga | geodb27: can you you please run cat /proc/self/mountinfo on the host where lxd is running | 14:20 |
| geodb27 | http://dpaste.com/0H48WSH here it is | 14:21 |
| zyga | geodb27: according to that log there is nothing mounted over /tmp | 14:22 |
| zyga | geodb27: perhaps you meant that something is in /tmp inside the containers started by lxd? | 14:22 |
| geodb27 | You will get some more infos here https://github.com/lxc/lxd/issues/5171 since I've already discussed this issue here. | 14:22 |
| zyga | geodb27: so can we get back to the problem, how is that file a problem? | 14:24 |
| geodb27 | The output I gave you is the one I got after having run "systemctl stop run-snapd-ns-lxd.mnt.mount" | 14:24 |
| zyga | sorry, I still don't understand, when you inspected mountinfo, was the system affected by this issue? | 14:25 |
| geodb27 | Well, here is what I've observed on my machine and the conclusions I came to : when my machine boots, the sddm service is broken (I can enter my password, but the "connexion" button does nothing). To access my desktop, I must go on a virtual console, log-in as root and issue systemctl restart sddm. | 14:25 |
| geodb27 | That's one point. And I came to suspect that the run-snapd-ns-lxd.mnt.mount was the problem. I suspect it to remount the /tmp after the sddm service is started, so that sddm loses it's tmp files). | 14:26 |
| zyga | why do you suspect that? | 14:27 |
| zyga | the .mnt file is not a service | 14:27 |
| zyga | it's a virtual unit created by systemd | 14:27 |
| zyga | for every entry in the mount table | 14:27 |
| zyga | you will see one | 14:27 |
| zyga | it is a specific mount namespace used by all the processes of the lxd snap | 14:27 |
| zyga | it cannot execute code so it cannot mount or otherwise over /tmp | 14:27 |
| geodb27 | Indeed, that's what I've observed. However, when I issue systemctl stop run-snapd-ns-lxd.mnt.mount the content of the /tmp folder is altered. | 14:28 |
| zyga | how? | 14:28 |
| zyga | can you show me mountinfo *before* that stop and *after* that stop | 14:28 |
| zyga | I suspect that what happens is that stop on that unit affects other units | 14:28 |
| zyga | that also stop | 14:28 |
| zyga | but that's not the unit per se, just something related to it | 14:28 |
| geodb27 | I can do that. Give me the time to do so :-) | 14:29 |
| zyga | sure, thank you | 14:29 |
| geodb27 | I'll have to reboot my machine. I'll be back when it is done. Thanks a lot for your help. | 14:29 |
| zyga | sure | 14:29 |
| craigulliott | Good morning everyone. Is there an appropriate channel where I can ask about manual review for a snap we have created. I believe it failed automatic review because it uses an X11 plug and slot (message said due to 'deny-connection' constraint for 'on-classic' from base declaration). | 14:50 |
| mup | PR snapd#6215 closed: tests: reset the snapd package to make the first test on the suite work properly <Created by sergiocazzolato> <Closed by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6215> | 14:52 |
| niemeyer | Weird.. | 14:55 |
| niemeyer | Server is not responding.. is it just me? | 14:55 |
| zyga | niemeyer: seems so | 14:55 |
| niemeyer | Back.. the tubes are broken | 14:55 |
| geodb27 | back again. Sorry for the time it took me zyga. Here are all the facts I've collected from boot : http://dpaste.com/2CFWT2S | 14:56 |
| zyga | geodb27: thanks | 14:56 |
| roadmr | craigulliott: you can ask in https://forum.snapcraft.io/c/store but you don't need to; the manual review queue is checked periodically by reviewers | 14:56 |
| zyga | geodb27: that's not quite what I asked for, I wanted to see the mount table alone (cat /proc/self/mountinfo) across the stop of that mount unit | 14:57 |
| craigulliott | @roadman Great, thank you! Do you know how long it usually takes? I'm not looking to expedite, just to plan according. | 14:57 |
| zyga | geodb27: one thing is sure, there's nothing mounted over /tmp in your log | 14:57 |
| craigulliott | @roadmr Great, thank you! Do you know how long it usually takes? I'm not looking to expedite, just to plan according. | 14:57 |
| zyga | geodb27: let me read it in detail though, maybe something interesting shows up | 14:58 |
| roadmr | craigulliott: there's no "standard of service" unfortunately :/ depends on the reviewers' workload really. | 14:58 |
| zyga | stgraber: hello | 14:59 |
| zyga | stgraber: one question | 14:59 |
| zyga | stgraber: does this mount entry look normal to you? | 14:59 |
| zyga | 860 844 0:3 mnt:[4026532713] /var/snap/lxd/common/ns/shmounts rw - nsfs nsfs rw | 14:59 |
| zyga | that's a mount namespace over shmounts | 14:59 |
| zyga | is that something that lxd does? | 14:59 |
| craigulliott | roadmr: thanks! | 14:59 |
| geodb27 | The last command I launched is indeed the systemctl stop run-snapd-ns-lxd.mnt.mount. I provided all this to show that before this command, the lxc profile edit throws an error related to /tmp, while after it runs fine. So, to me, there definetly is a problem related to this virtual service. | 14:59 |
| pedronis | zyga: can you check whether #6216 does what you expected from the revert? | 15:00 |
| pedronis | otherwise it can land I think, it's green | 15:00 |
| mup | PR #6216: tests: revert "tests: restore in restore, not prepare" <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6216> | 15:00 |
| stgraber | zyga: yeah, that's normal, it's the logic we have to be able to keep our mounts across snap mntns changes (when core gets refreshed) | 15:01 |
| geodb27 | Hi stgraber :-) | 15:02 |
| stgraber | zyga: on first startup we create a separate mount namespace, setup propagation both ways (rshared) with the snap-confine mntns and put a reference to it in the initial mount namespace so we can recover it even when our mntns is completely destroyed | 15:02 |
| mvo | 6216 needs a second review | 15:06 |
| mup | PR snapd#6214 closed: spread.yaml: disable _/funcarg test for now <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/6214> | 15:08 |
| zyga | stgraber: thank you for confirming | 15:10 |
| zyga | pedronis: looking | 15:10 |
| zyga | pedronis: yes, merged now | 15:10 |
| zyga | thank you | 15:10 |
| zyga | I'll grab quick lunch now | 15:10 |
| pedronis | np | 15:10 |
| zyga | mvo, pstolowski: still 100% reproducing that race in unit tests and 0% reproducing that 2.36 issue | 15:11 |
| stgraber | geodb27: still that weird /tmp behavior? :( | 15:11 |
| stgraber | geodb27: we had one other report of something similar so far, but that user was running tmpreaper, so that made sense and was easy enough for them to fix | 15:11 |
| mup | PR snapd#6216 closed: tests: revert "tests: restore in restore, not prepare" <Created by sergiocazzolato> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6216> | 15:11 |
| mvo | zyga: I'm running the branch with your diff too, but no success so far | 15:12 |
| pstolowski | zyga: wait, i thought you were reproducing it in 2.36 release? | 15:13 |
| geodb27 | indeed, still the same very problem. Well, for now, the solution is "simple" as you can see : issue first a "systemctl stop run-snapd-ns-lxd.mnt.mount" and then "systemctl restart sddm" to have my laptop behave as expected. The fact is that I really want to understand why it is so. | 15:13 |
| geodb27 | (I'm somehow stubborn and obstinate and really want to understand) :p | 15:14 |
| mvo | pstolowski: I get the "... Panic: change 1 unexpectedly became unready (Do) (PC=0x45ECEE) | 15:17 |
| mvo | " panic pretty reliable here currently when running google:ubuntu-16.04-64 from upstream/release/2.36 fwiw | 15:17 |
| pstolowski | mvo: thanks, interesting i couldn't repro locally with 2.36 (although kinda expected if it's a race) | 15:18 |
| kyrofa | zyga, interesting feedback from a snap user: https://help.nextcloud.com/t/nextcloud-snap-users-please-list-the-issues-youre-facing/1979/152 . Does a revert after a refresh that changes a channel not revert the followed channel as well? | 15:20 |
| mup | Bug #1805162 opened: Impossible to get newly settled props <configure> <props> <snapctl> <snapset> <validation> <Snappy:New> <https://launchpad.net/bugs/1805162> | 15:20 |
| zyga | back now | 15:30 |
| pedronis | pstolowski: probably speed of the hardware (virtual or not) is relevant | 15:30 |
| zyga | geodb27: we want to understand as well :) | 15:30 |
| geodb27 | I'm sure you want :-) | 15:31 |
| zyga | kyrofa: I think .. I don't know - perhaps not, we just revert the revision, not the tracking change operation | 15:31 |
| geodb27 | hang on, I've found something that might interest you, zyga. | 15:31 |
| zyga | pedronis, pstolowski: I was testing on google | 15:31 |
| * cachio lunch | 15:31 | |
| zyga | geodb27: cool | 15:31 |
| geodb27 | as @stgraber suggested and pointed out on the discussion we both had on github, look at this : http://dpaste.com/0C35VXQ and in particular line 71. | 15:33 |
| geodb27 | This is from my system, right now, so after the systemctl stop and when things work. However, this looks weird to me : the source of the mount point is indeed deleted. | 15:34 |
| geodb27 | oups, typo, line 73 | 15:34 |
| zyga | oh, nice | 15:34 |
| zyga | 661 751 259:2 /tmp/snap.0_lxd_M9HkFH/tmp//deleted /tmp rw,relatime - ext4 /dev/nvme0n1p2 rw,errors=remount-ro,data=ordered | 15:35 |
| geodb27 | yep, that one. I've checked, there is nothing instructed to wipe /tmp at boot time. | 15:35 |
| zyga | this says that /tmp/snap.0_lxd_M9HkFH/tmp/ of the ext4 filesystem at device /dev/nvme0n1p2 is mounted at /tmp and that the source has been removed | 15:36 |
| zyga | so | 15:36 |
| zyga | as I see it | 15:36 |
| zyga | when you boot you get /tmp from ext4 on nvme | 15:36 |
| zyga | then lxd snap starts up and creates that lxd.mnt file you know about in /run/snapd/ns/ | 15:36 |
| zyga | that file captures the mount namespace of the lxd process | 15:36 |
| zyga | though lxd is special because it has more permissions than most snaps, so it can do many mount changes by itself | 15:37 |
| zyga | then something cleans up /tmp | 15:37 |
| zyga | removing the snap.0_lxd_$random directory | 15:37 |
| zyga | so I have an ide | 15:37 |
| zyga | *idea | 15:37 |
| zyga | geodb27: what is your distribution? | 15:37 |
| geodb27 | kubuntu 18.04 lts | 15:38 |
| zyga | geodb27: at least on ubuntu there's a systemd unit that wipes /tmp | 15:38 |
| zyga | one sec, let me check something | 15:38 |
| geodb27 | take your time... I'm not in a hurry :-) | 15:40 |
| zyga | can you share the output of systemctl status systemd-tmpfiles-clean.service | 15:41 |
| geodb27 | Sure : here it is http://dpaste.com/2NMEPHJ | 15:42 |
| zyga | Active: inactive (dead) since Mon 2018-11-26 15:46:46 CET; 55min ago | 15:43 |
| zyga | can you try to corelate that to the snap.lxd.service please? | 15:43 |
| geodb27 | http://dpaste.com/0X9RW2S I think that you got it ! | 15:45 |
| zyga | Active: active (running) since Mon 2018-11-26 15:31:51 CET; 1h 12min ago | 15:45 |
| zyga | so | 15:45 |
| zyga | lxd starts before the tmpfsiles wipe job | 15:45 |
| geodb27 | But I guess that systemctl disable systemd-tmpfiles-clean.service will not be the right solution. | 15:45 |
| zyga | when you unmount that .mnt file you get a clean slate | 15:45 |
| zyga | but one suggestion | 15:45 |
| zyga | next time instead of doing that | 15:45 |
| zyga | call | 15:45 |
| zyga | sudo /usr/lib/snapd/snap-discard-ns lxd | 15:45 |
| zyga | this does more than that umount alone | 15:46 |
| zyga | geodb27: no, it's started by a timer | 15:46 |
| zyga | and it is a good idea anyway | 15:46 |
| zyga | but this feels like a systemd bug | 15:46 |
| zyga | it seems, unless I'm mistaken | 15:46 |
| zyga | that it can wipe /tmp at any time | 15:46 |
| zyga | CC: stgraber ^ | 15:46 |
| zyga | geodb27: I would appreciate if you could add this to the bug report | 15:46 |
| geodb27 | which bugreport ? The one on github I pointed to ? | 15:47 |
| zyga | yep | 15:47 |
| zyga | to capture that tmp is wiped with that service | 15:47 |
| geodb27 | I'll do it right now. Thanks a lot for your kind help ! | 15:47 |
| zyga | and this causes the tmp subdirectorty that is used by lxd as /tmp is gone | 15:47 |
| zyga | thank you! | 15:47 |
| zyga | *subdirectory | 15:48 |
| zyga | and I'm back to debugging other things | 15:48 |
| zyga | pstolowski: do you have any patches, even early on that I could try | 15:49 |
| zyga | I hit that error every time I try now | 15:49 |
| stgraber | zyga: right, so not specific to LXD at all, this would break any snap that's started on boot, LXD just happens to be a very common one of those | 15:49 |
| zyga | stgraber: exactly | 15:49 |
| stgraber | zyga: still odd that geodb27 is the only report we've seen of this | 15:50 |
| geodb27 | @stgraber not really... I don't think that lxd is much used on Kubuntu... on ubuntu there is no doubt at all... But on a desktop distro ? | 15:52 |
| pstolowski | zyga: no, nothing yet | 15:52 |
| pstolowski | zyga: will let you know as soon as i've something | 15:53 |
| mup | Bug #1805170 opened: Extend snapd REST API to get conf props from change transaction <Snappy:New> <https://launchpad.net/bugs/1805170> | 15:56 |
| stgraber | geodb27: sadly Kubuntu itself reports Ubuntu through lsb-release so we don't get any idea of number of users | 15:57 |
| stgraber | geodb27: I do however see about 500 users on KDE neon 18.04, so at least those are desktop users of LXD using KDE | 15:57 |
| geodb27 | I beleive you @stgraber :-) This was just a guess. Thenafter, my machine was updated from kubuntu 16.04 to 18.04 with a do-release-upgrade. I don't say that this is what caused the problem, but it could somehow be (though I don't know how at all). | 16:01 |
| zyga | mvo: can you reproduce the issue with 2.36 branches | 16:02 |
| zyga | I will re-trigger them now | 16:02 |
| zyga | I cannot reproduce the issue at all | 16:02 |
| mvo | zyga: I have not reproduced the permission denied issue yet, I run it again, I ran it ~2 times | 16:11 |
| mvo | zyga: or three maybe | 16:12 |
| zyga | ok | 16:12 |
| zyga | maybe not super critical | 16:12 |
| zyga | if it happens once in ~10 hours | 16:12 |
| pedronis | Chipaca: #6126 got +2 , don't know if I need to look at it as well, or if there is still some open Q there | 16:13 |
| mup | PR #6126: dirs, wrappers, overlord/snapstate: make completion + bases work <Created by chipaca> <https://github.com/snapcore/snapd/pull/6126> | 16:13 |
| Chipaca | pedronis: ah i missed the second +1 | 16:20 |
| Chipaca | pedronis: should be gtg | 16:20 |
| mup | PR snapd#6126 closed: dirs, wrappers, overlord/snapstate: make completion + bases work <Created by chipaca> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/6126> | 16:21 |
| geodb27 | zyga, with your kind help and some reading, I've added a /etc/tmpfiles.d/snap.conf file with this content : x /tmp/snap* X/tmp/snap* (on two lines). | 16:22 |
| zyga | ah | 16:22 |
| zyga | this won't unlink those directories | 16:22 |
| zyga | perhaps snaps should ship that? | 16:22 |
| zyga | geodb27: is there a bug report about this on launchpad.net? | 16:22 |
| zyga | specifically bugs.launchpad.net/snapd | 16:23 |
| zyga | if no, can you file one, while we cannot reproduce this straight away shipping those exclusion rules in snapd seems possible | 16:23 |
| geodb27 | I'll be heading for this when I'll be 100% sure that this solves the problem, indeed ! | 16:32 |
| zyga | geodb27: thank you! | 16:32 |
| zyga | good luck :) | 16:32 |
| pedronis | pstolowski: I reviewed the hotplug PR that you fixed, lgtm, I think the one I reviewed now need a pass by mvo. And as I said we should try to chat on wed about more global issues about event sequences for the same device | 16:37 |
| pstolowski | pedronis: thanks, wed sounds good | 16:38 |
| geodb27 | zyga : https://bugs.launchpad.net/snapd/+bug/1805182 job done. Thus, I'm sure not to forget about it :-) | 16:47 |
| mup | Bug #1805182: /tmp is sometime cleaned up after snaps are launched <snapd:New> <https://launchpad.net/bugs/1805182> | 16:47 |
| zyga | geodb27: thank you! | 16:47 |
| geodb27 | Time to go, my day's work's over. | 16:47 |
| geodb27 | Again, thanks a lot for your kind help ! | 16:47 |
| zyga | thank you for the detailed reports! | 16:47 |
| mvo | pedronis: I look at the hotplug PR tomorrow | 16:48 |
| pedronis | mvo: there's a couple where I requested your review | 16:48 |
| zyga | mvo: can you look at that bug please^ | 16:48 |
| pedronis | yes, tomorrow is fine | 16:48 |
| zyga | mvo: just triaged it to high | 16:48 |
| mvo | pedronis: thanks, will do those tomorrow as well | 16:48 |
| zyga | mvo: seems that we could ship a tmpiles removal rule that would exempt removal of /tmp/snap.* directories | 16:49 |
| zyga | mvo: all the details are in the bug | 16:49 |
| mvo | zyga: ok, I check that tomorrow as well | 16:49 |
| zyga | sure | 16:49 |
| zyga | 17:49 | 16:49 |
| mvo | zyga: ? | 16:49 |
| * zyga wonders if EODing on time is a good idea | 16:49 | |
| mvo | zyga: heh, I play hockey in some minutes | 16:49 |
| * mvo looks forward to this | 16:50 | |
| zyga | mvo: thatt's a great thing to do :) | 16:50 |
| zyga | enjoy! | 16:50 |
| mvo | ta | 16:50 |
| sacarde | hi | 17:00 |
| sacarde | I installed vlc 3.0.4 from snap in ubuntu-18.04, but I dont find binaries: cvlc nvlc... is right? | 17:00 |
| pedronis | cachio: does 5714 need a 2nd master merge? | 17:05 |
| pstolowski | zyga: puzzled by that panic, can't repro on spread either - (release/2.36*) $ spread -debug -repeat 10 google:ubuntu-16.04-64:tests/unit/go | 17:07 |
| zyga | hmm | 17:08 |
| zyga | wonder what I do | 17:08 |
| zyga | anyway, let me know if you figure out the error, may be some debug log can help | 17:08 |
| zyga | e.g. extra logging we do in that test case | 17:08 |
| roadmr | sacarde: looks like the vlc snap does not provide those commands. | 17:13 |
| pstolowski | zyga: can you try with this patch https://pastebin.ubuntu.com/p/sGzrRdbV6Y/ and show me the output? | 17:16 |
| zyga | sure! | 17:17 |
| pedronis | zyga: #6206 has a conflict now also probably needs master merge anyway | 17:18 |
| mup | PR #6206: many: fix composite literals with unkeyed fields <Created by zyga> <https://github.com/snapcore/snapd/pull/6206> | 17:19 |
| zyga | ta | 17:19 |
| pstolowski | zyga: also, let me know if you run it differently; nb i believe my 2.36 is up-to-date, last commit 84e5ab2ad44511cb287a6450f5be6074563ea535 | 17:19 |
| mup | PR snapd#6199 closed: Revert "cmd/snap, tests/main/snap-info: highlight the current channel" (2.36) <Created by chipaca> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/6199> | 17:20 |
| === chihchun is now known as chihchun_afk | ||
| zyga | pstolowski: I'm getting this error each time I run a snap | 17:58 |
| zyga | warning: cannot open cookie file /var/lib/snapd/cookie/snap.sublime-text | 17:58 |
| zyga | this particular snap that is | 17:58 |
| zyga | pstolowski: it is classic, is this expected? | 17:58 |
| pstolowski | zyga: no | 17:58 |
| zyga | pstolowski: and I have some good news | 17:58 |
| zyga | panic with debug messages https://www.irccloud.com/pastebin/HTj9vvG7/ | 17:59 |
| zyga | pstolowski: can you reproduce that quickly with sublime-text? | 17:59 |
| pstolowski | zyga: oh, re panic, we're not mocking something? | 17:59 |
| zyga | popey: well, we always paniced | 18:00 |
| zyga | ... Panic: change 1 unexpectedly became unready (Do) (PC=0x45ECEE) | 18:00 |
| zyga | *panicked | 18:00 |
| pstolowski | zyga: sublime installed fine, cookie created, no error when run | 18:03 |
| zyga | hmm | 18:03 |
| zyga | odd | 18:03 |
| * zyga wonders | 18:03 | |
| pstolowski | zyga: did you mess up with state or anything else could get out of sync? | 18:04 |
| zyga | ha | 18:04 |
| zyga | that's fun | 18:04 |
| zyga | I don't have sublime-text installed | 18:04 |
| zyga | something is veeery broken on that system | 18:05 |
| zyga | no worries, thanks for checking | 18:05 |
| zyga | https://www.irccloud.com/pastebin/uHJV0NSj/ | 18:05 |
| zyga | ^ this runs sublime-text from classic package | 18:05 |
| zyga | fun, eh? | 18:05 |
| zyga | stray alias entry can run any command on host | 18:05 |
| pstolowski | hmm | 18:06 |
| * Chipaca wanders off | 18:08 | |
| pstolowski | zyga: i wonder if we shouldn't mock snap-confine wrt to that setup-profiles error in the test? still, wondering why i don't see it | 18:19 |
| zyga | pstolowski: I'm happy to try more patches tomorrow, I'm wrapping up now | 18:20 |
| zyga | and working on something else still | 18:20 |
| zyga | sorry if I'm not more helpful | 18:20 |
| pstolowski | zyga: yeah, me too | 18:20 |
| pstolowski | zyga: you helped a lot, thanks! | 18:20 |
| === pstolowski is now known as pstolowski|afk | ||
| === zyga is now known as zyga|afk | ||
| * cachio afk | 19:07 | |
| sacarde | roadmr, who build package vlc(snap or deb), ubuntu or snap ? | 19:26 |
| mup | PR snapd#6206 closed: many: fix composite literals with unkeyed fields <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6206> | 19:34 |
| roadmr | jdstrand: are you around? | 20:42 |
| roadmr | sacarde: the vlc developers are responsible for maintaining that snap | 20:42 |
| roadmr | sacarde: you can find contact and support info here https://snapcraft.io/vlc | 20:43 |
| sacarde | roadmr, in #videolan reply to my question, that command are alias from vlc command | 20:44 |
| sacarde | thank you | 20:44 |
| roadmr | sacarde: glad you got it sorted out :) cheer | 20:44 |
| sacarde | thanks | 20:44 |
| ijohnson | roadmr: jdstrand is out today and tomorrow | 20:44 |
| roadmr | thanks ijohnson ! I'll poke him on Wednesday then | 20:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!