[06:05] morning [06:13] good morning :) [06:14] mborzecki: I pushed missing patches from master into https://github.com/snapcore/snapd/pull/6235 [06:14] PR #6235: overlord,apparmor: new syskey behaviour + non-ignored snap-confine profile errors (2.36) [06:14] the reference is https://github.com/snapcore/snapd/pull/6233 [06:14] PR #6233: overlord: don't write system key if security setup fails [06:14] oh, and it's green now [06:14] there are a few extra lines of diff [06:15] because that branch combines another small patch from master [06:15] namely https://github.com/snapcore/snapd/pull/6235/commits/cbc6e3e029e36e2d120a3dbb2edad39f689da479 [06:15] PR #6235: overlord,apparmor: new syskey behaviour + non-ignored snap-confine profile errors (2.36) [06:15] I think it's good to go :) [06:19] mborzecki: I think we can merge https://github.com/snapcore/snapd/pull/6248 [06:19] PR #6248: tests/lib/reset: restore context of removed snapd directories [06:20] PR snapd#6248 closed: tests/lib/reset: restore context of removed snapd directories [06:21] a pair of simple looking conflicts on https://github.com/snapcore/snapd/pull/6238 [06:21] PR #6238: [RFC] many: add minimal SELinux support, refactor the policy [06:22] mborzecki: did you pick that keyboard? :) [06:22] zyga: no, not yet [06:23] mborzecki: pawel has apparently moved his work onto the new machine [06:23] expectedly, because it's the quietest and fastest machine he has now [06:23] I'm curious if you will experiment with vmware as well :) [06:24] PR snapd#6256 closed: snap: add new `snap run --trace-exec` call (2.36) [06:34] mborzecki: I resolved conflicts on https://github.com/snapcore/snapd/pull/6193/files [06:34] PR #6193: spread: drop Fedora 27, add Fedora 29 <⛔ Blocked> [06:34] though I made F29 manual so this can land [06:34] it should pass with just F28 [06:52] Quick school run [07:28] Re [07:50] PR snapd#6258 opened: WIP: interfaces: support D-Bus activation [07:58] and back now [07:58] jamesh: hey :) [07:58] hi zyga [07:59] I will try to review all branches this week [07:59] but we have a backlog and limited capacity [08:00] zyga: no problem. I'm still working on the dbus activation one: the PR is more for CI feedback [08:00] (that's why I marked it WIP) [08:00] * zyga loves to see :: targets used in makefiles === pstolowski|afk is now known as pstolowski [08:06] mornings [08:08] pstolowski: hey [08:11] hey pawel :) [08:22] https://github.com/snapcore/snapd/pull/6257 is green [08:22] PR #6257: testutils: split checkers, tweak tests [08:22] and very long :-) [08:24] nvidia physix is BSD now [08:24] that's impressive [08:24] wonder if nvidia will have a change of strategy for their other software [08:25] mborzecki: hey [08:25] mborzecki: I think we should do something about that snapd.mk PR [08:26] did that land? [08:26] I would like to resurrect it, make minimal modifications as discussed (move variable generation back to .spec file and have snapd.mk include vars.mk) [08:26] and land it [08:26] looking at the PR from james, there will be new files, new things to look at [08:28] uh oh [08:28] The job exceeded the maximum log length, and has been terminated. [08:28] travis now kills jobs that log too much [08:28] perhaps we should tweak spread to log less stuff [08:28] and definitely make rpm builds quieter [08:29] zyga: yeah, 2.36 branch is failing atm because of this i think [08:30] one other thing I would silence is the package name selection [08:30] that is, mapping from ubuntu names to other names [08:30] zyga: or sort of, something else fails, but travis terminates the job :) [08:30] that's extremely verbose [08:30] set +too-much-gibberish-x [08:50] trying some simple changes to reduce the log size now [08:54] mborzecki: do you know if it is possible to query the state of "set" settings? [08:55] zyga: snap get ? [08:55] snap get -d [08:55] no no [08:55] I mean in shell [08:55] set -x vs set +x [08:55] aah [08:55] there's nothing I can find quickly [08:59] zyga: set -o should do the trick [08:59] ah, neat, thanks! [09:00] zyga: xtrace is apparently the option [09:00] yes [09:00] and even supported by zsh :) [09:02] quick patch for reduced verbosity https://www.irccloud.com/pastebin/o15U1EqK/ [09:06] PR snapd#6259 opened: tests: reduce verbosity around package installation <⚠ Critical> [09:06] mborzecki, pstolowski: ^ [09:07] ok [09:08] mborzecki: do you want to discuss the connections API? [09:08] just looking at some random part of trace log [09:08] I would love to have [09:08] snap debug is-connected [09:08] snap debug connected-slots SNAP:PLUG [09:08] snap debug connected-plugs SNAP:SLOT [09:09] snap debug connecte-snaps SNAP: [09:09] (typos aside, you get the idea) [09:09] mhm [09:09] ideally all fuelled by a new API call [09:10] perhaps those could also be visible via snapctl [09:10] pstolowski: ^ do we have something like that for snapctl today? [09:11] zyga: no, unfortunately not [09:11] we might expose it in both places eventually [09:11] feels like something useful to snap developers [09:12] mborzecki: the general idea feels like [09:12] mborzecki: connections [09:12] interface optionally limits to interface type [09:12] string optionally filters by name of plug/slot [09:13] snap:plug and snap:slot can be empty or partially empty [09:13] (perhaps string is useless actually) [09:13] flags can alter the set of returned data (e.g. attributes) or visible things (e.g. show disconnected things as well as connected, for hotplug) [09:13] it seems it could power all the commands mentioned above [09:14] all of the options would probably be packed into a query structure [09:14] we might also have a flag for controling core/gadget connections [09:14] (e.g. skip them or include them) [09:15] so that the client code can stop having to special case "system/snapd/core/ubuntu-core" [09:17] zyga: let's have a chat ~11, 1115? [09:17] sure [09:22] #6193 is green [09:22] PR #6193: spread: drop Fedora 27, add Fedora 29 <⛔ Blocked> [09:32] merge it [09:32] we will solve the 29 blocked aspect separately [09:34] zyga: looking at 6257 [09:34] zyga: about fontconfig again, desktop interface mount of /var/cache/fontconfig is probably a problem [09:34] mmm [09:34] but that only applies to strictly confined apps [09:35] isn't the problem you experienced only affecting classic confined apps? [09:38] zyga: i'm still bit worried that we're generating that cache in a location that does not match distro choice, with the way things are set up now it should work, we're going to make /var/cache/fontconfig ourselves after all [09:38] mborzecki: yes [09:39] maybe we should really build a cache in /var/cache/snapd/fontconfig [09:39] zyga: righ, but the the classic snaps that use libfontconfig from core will end up poking /var/cache/fontconfig ;) [09:40] yes [09:40] tradeoffs everywhere [09:40] but for those we have no way out [09:40] until we can attack mount namespaces for classic confined software [09:41] mhm [09:42] mborzecki: updated https://github.com/snapcore/snapd/pull/6251 [09:42] PR #6251: cmd/snap-confine: refactor calling snapd tools into helper module [09:42] #6180 is green and needs (re)reviews [09:42] PR #6180: snap/info: bind global plugs/slots to implicit hooks [09:47] zyga: pstolowski: can you take a look https://github.com/snapcore/snapd/pull/6193 and +1 ? [09:47] PR #6193: spread: drop Fedora 27, add Fedora 29 [09:48] k [09:50] mborzecki: aha [09:51] pstolowski: and this one needs a 2nd review too: https://github.com/snapcore/snapd/pull/6246 super simple :) [09:51] PR #6246: spread: show AVC audits when debugging, start auditd on Fedora [09:52] PR snapd#6193 closed: spread: drop Fedora 27, add Fedora 29 [09:52] mborzecki: replied on https://github.com/snapcore/snapd/pull/6251 [09:52] PR #6251: cmd/snap-confine: refactor calling snapd tools into helper module [09:53] PR snapd#6246 closed: spread: show AVC audits when debugging, start auditd on Fedora [09:53] anyone knowledgeable of assertions left around? https://forum.snapcraft.io/t/cant-install-or-refresh-snaps-on-arch-linux/8690 the device is seeded, however there is no model assertion even though data.auth.model is already set to generic-classic [09:53] hmmm [09:53] I guess we just have to man up and read that code [09:54] pstolowski: on https://github.com/snapcore/snapd/pull/6257, shall I update all headers to 2018? [09:54] PR #6257: testutils: split checkers, tweak tests [09:55] zyga: i'd use this opportunity [09:55] sure [10:00] pstolowski: done :) [10:00] ty ! [10:04] suprising how travis jobs are succeeding now [10:04] pstolowski: this is not super urgent but it needs a 2nd review https://github.com/snapcore/snapd/pull/6244 [10:04] PR #6244: release: detect too old apparmor_parser [10:05] a much simpler variant of this went into 2.36 [10:05] this one is just tidier and less noisy [10:08] PR snapd#6090 closed: Granted 'account_control' access to /home/** [10:10] pstolowski, mborzecki: what do you guys think about https://github.com/snapcore/snapd/pull/6250 [10:10] PR #6250: data: set KillMode=process for snapd [10:11] PR snapd#6200 closed: tests: fix for tests test-*-cgroup [10:11] perhaps cherry pick the detector I suggested and LGTM? [10:11] I'm sure mvo can iterate on top [10:11] but feels like the right thing to do [10:19] zyga: do apparmor_parser and snap-seccomp lock profiles? any risk of having profiles messed up if new instances start writing profiles while old ones are still running? [10:20] (but in general yes, i think that's the right step what this PR proposes) [10:20] they don't lock profiles because they don't have to AFAIK: [10:20] snap-seccomp writes a new file with atomic move [10:21] snapd writes source profiles with atomic move [10:21] ah, good then [10:21] apparmor_parser loads profiles into the kernel, kernel is the point of mutual exclusion [10:21] snap-confine loads profiles from disk only at startup [10:21] that holds the lock but that's not really a factor [10:21] seccomp profiles persist until process death [10:22] apparmor profiles are changed dynamically [10:22] uhh reached asserts db trying to find out what's going on [10:22] and? [10:25] got lost :) [10:25] back to devicemgr [10:31] i bet that device serial is set on that guy's machine, but it's unclear why that'd be the case and the model assertion would not be present [10:33] feels like we could use some commands to show stuff [10:33] more than "snap known" [10:33] which feels like a debug command, not an end-user thing [10:33] super unpleasant user command if anything :) [10:34] https://www.irccloud.com/pastebin/xNFfHNva/ [10:34] hmmm [10:34] something really broken there :) [10:34] looking [10:34] nice [10:35] note that it uses dpkg on centos [10:35] heh right [10:35] snapd-xdg-open? i don't recall this in rpm [10:36] tests/upgrade/snapd-xdg-open/task.yaml [10:36] it runs on centos [10:36] but is clearly specific to ubuntu [10:36] is that a new task? [10:36] nope [10:36] look at it [10:36] wonder why the error never surfaced? [10:36] haha so how did it not fail before? [10:36] exactly [10:36] set -e is easy to lose [10:37] well [10:37] it failed now [10:37] I really wonder what I changed [10:38] I don't quite understand some of the logic in the pkgdb.sh [10:46] mborzecki: removed cloexec handling in https://github.com/snapcore/snapd/pull/6251 [10:46] PR #6251: cmd/snap-confine: refactor calling snapd tools into helper module [10:47] zyga: what was it? [10:47] mborzecki: will to fix that centos issue or shall I? [10:47] mborzecki: with cloexec? [10:47] mborzecki: just my debug wrappers [10:47] mhm [10:47] aah [10:47] shell scripts? [10:47] I didn't realise this is needed for scripts but not for programs [10:47] yep [10:47] yup, they mention that in the manpage [10:47] I test this way locally [10:48] yeah, in the BUGS section below :) [10:48] I missed that [10:54] PR snapd#6235 closed: overlord,apparmor: new syskey behaviour + non-ignored snap-confine profile errors (2.36) [10:57] mborzecki, pstolowski: my main feature branch that can land https://github.com/snapcore/snapd/pull/6149 [10:57] PR #6149: cmd/snap-confine: capture initialized per-user mount ns [10:57] I will have follow ups and tests once the feature snapd side is ready [10:58] zyga: k, will take a look [10:58] thank you! [11:00] mborzecki: I have a debug shell on centos 7 with that dpkg-query script [11:00] no idea why it would ever pass before [11:01] heh [11:01] disabled it, pushed [11:01] magic [11:01] btw. i'm playing with spread on fedora 29 [11:01] DEBUG: running snap-device-helper add snap_test-snapd-udev-input-subsystem_slot /sys/class/mem/null 1:3 [11:01] execl failed: No such file or directory [11:01] do you know my main issues with spread on fedora 29 [11:01] right [11:01] snap-device-helper is a script, wtf? [11:01] I know about htis [11:01] *this [11:01] yes [11:01] but that's not the main problem [11:01] the problem is that we must never repackage core or snapd snaps [11:01] with stuff we've built on a random distro [11:01] that's a bad test [11:01] bad logic [11:02] and not representative of what ever happens in production [11:02] locally built snapd package gets injected into core snap [11:02] yes, wanted to reach that point and inspect what happens exactly [11:02] so libc from 2019 will be in a 2016 system [11:02] I mean, progs linked to 2019 will be in a 2016 system [11:02] what is wrong is that rpm helpers change the #!/ line [11:03] look at the shebang line in the source and in that system that is broken [11:03] the short term resolution: [11:03] stop repackaging core / snapd anywhere outside of 16.04 [11:03] we can perhaps repackage snap-exec [11:03] but nothing more [11:07] woot, rpm learned some new tricks [11:27] mborzecki: mkosi got a huge static typing boost [11:28] https://github.com/systemd/mkosi/pull/300 [11:28] PR systemd/mkosi#300: Modify so that `flake8` and `mypy` are happy [11:31] hm still got mixed feelings about python and type hints [11:38] really? that's like 99% of the error checking there :) [11:41] PR snapd#6260 opened: spread, tests: use audit checkpoints when dumping audit log [11:41] simple one ^^ [11:41] zyga: it's just the type enforce part is missing [11:42] mborzecki: no need [11:42] mypy checks that it is true :) [11:42] if mypy says it's ok, the types match [11:42] reality is somewhat different as sometimes there are bugs in mypy and sometimes there are cases that are valid but the type system cannot express it [11:42] but really, apart from changing python in py4 [11:43] this is the best way to write python now [11:52] eh [11:52] untangling the mess in config manager [11:52] sucks [11:53] cyclic import mess, that is [11:53] hah :) [11:54] I will never finish user mounts this way [12:03] I think I got it [12:03] I will propose a small branch with just the tweak to config [12:06] PR snapd#6261 opened: tests/lib/prepare: make sure that SELinux context of repacked core snap is controlled [12:06] another simple PR ^^ [12:06] this time not dropping xattrs [12:07] eh [12:07] more cyclic shit [12:07] I hate this part of golang [12:08] LGTM with capitalisation of sentences [12:10] mborzecki: hmm [12:10] does set -o [12:10] does that set something in practice [12:10] because i have a feeling that that patch actually uncovered real bugs [12:10] zyga, hey [12:10] error: Failed build dependencies: [12:10] compiler(go-compiler) is needed by snapd-1337.2.36.2-0.el7.x86_64 [12:10] I get this error on centos 7.6 [12:10] −o Write the current settings of the options to standard output in [12:10] an unspecified format. [12:11] cachio: ehey [12:11] cachio: hmm interesting [12:11] zyga, I made sure go is instlaled [12:11] mborzecki: https://www.irccloud.com/pastebin/3I0NnCO5/ [12:11] mborzecki, yes [12:11] cachio: I don't follow EPEL work so no idea [12:11] mborzecki, do you? [12:11] right [12:11] ? [12:12] cachio: any chance i could use the image? [12:12] I have a debug session already openned [12:12] mborzecki, do you want to use it? [12:12] mborzecki: my guess is that sourcing some of our packages nukes set -e [12:12] and this is now changed _somehow_ [12:13] mborzecki, otherwise the image name is test-centos-1 [12:13] sourcing vs executing [12:13] sourcing is evil [12:13] cachio: yes, i'll try to build snapd there, i recall chasing down something similar [12:14] zyga: hence, 'evil sourcerer' :) [12:14] haha [12:14] we should really stop doing that [12:15] or add sanity checks on 1st/last line of each file [12:15] this really sucks [12:15] I wonder what else is hidden by this [12:15] shell sucks [12:16] mborzecki, great, thanks [12:19] zyga, are you working with this error on snap-run right? [12:20] which error? [12:20] zyga, I have seen many times the error : [12:21] snap run --trace-exec test-snapd-tools.echo hello [12:21] I haven't touched that code [12:21] nor seen the error [12:21] that's mvo's new thing, if it fails perhaps we should revert for now [12:22] mborzecki: https://github.com/snapcore/snapd/pull/6190 should now pass and has the configuration untangled [12:22] zyga, ok, I'll collect the logs since yesterday to check it is failing on the same place [12:22] PR #6190: overlord/configstate,features: expose features to snapd tools <⛔ Blocked> [12:22] how is it failing? [12:24] zyga, well, it is failing on arm devices [12:24] _how_ [12:24] zyga, https://paste.ubuntu.com/p/4RGQB3VmCv/ [12:25] error: unknown flag `trace-exec' [12:25] that's the new thing [12:25] looks like combination of old binaries and new tests [12:25] so? [12:26] hi, I opened https://bugs.launchpad.net/snapcraft/+bug/1802345 a while ago. now, with snapcraft building by default in VMs, is there a way to export that env var? [12:26] Bug #1802345: can't build python part with C modules [12:26] ackk: that's a question for sergiusens [12:26] zyga, ahh, ok, in that case the core on edge did't get the last updates [12:29] zyga, sorry, I'll check what happended there [12:33] mborzecki: I added some sanity detectors to sourcing pkgdb [12:33] hi! can anybody please verify https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1778936 ? [12:33] Bug #1778936: please re-add Support-system-image-read-only-etc.patch [12:33] otherwise, i guess i will need to drop the writable-etc patch to get systemd to be released =/ [12:34] xnox: what's the timeline? [12:34] zyga, well, mvo has been chasing me about this for a long time, and as far as i understand it affects/blocks usablity of core18 by default..... [12:35] yes, I mean, how soon do you need this verified [12:35] mvo is away this week [12:35] zyga, i have a new SRU ready to upload with severe regression fixes. [12:35] zyga, thus i do need to release this update soon. [12:35] sergiusens, hi, do you have any insight on #1802345 [12:35] Bug #1802345: can't build python part with C modules [12:35] zyga, i'm away from thursday.... today/tomorrow would be really nice [12:36] xnox: I see, I will get back to you shortly [12:36] zyga, mostly i have no idea how to build and boot core18 with systemd from bionic-proposed. [12:36] otherwise i can validate this issue myself.... [12:36] zyga, are there core18 images built with bionic-propsoed somewhere? [12:36] sadly, I don't know this [12:36] I'm trying to find out [12:37] the pipeline around this is a bit of a black magic [12:37] ackk: might just be a python plugin bug; the xenial issue feels tricky as snapcraft wouldn't be able to build as we use yaml, macaroons and other things so I will have to look into it [12:39] sergiusens, wdym "snapcraft wouldn't be able to build" ? you mean build itself? [12:39] yes, build itself [12:40] zyga, also the systemd update, triggered snapd autopkgtests with regressions on all arches. [12:40] sergiusens, ackk, try adding libc6-dev if you want the headers for libc [12:40] sergiusens, why is that relevant? the bug is about building a python snap [12:41] diddledan, the problem is not that headers are not there, it's that they are not found without that export [12:41] zyga, is 2018-11-20 16:29:44 Failed tasks: 1 [12:41] - autopkgtest:ubuntu-18.04-i386:tests/main/dirs-not-shared-with-host:alternatives [12:41] known? [12:41] ackk: snapcraft is a python snap, what do you mean it is not relevant? [12:41] in that case i can ask release team to add a hint to ignore that. [12:41] sergiusens, does it specify that env var in override-build? [12:42] oh, ackk, you're building with python2 but supplying the python3-dev package [12:42] add `python-version: python3` to your part yaml [12:43] diddledan, oh lemme try that [12:43] xnox: hmmm, probably tests out of sync with code [12:43] xnox: we no longer have /etc/alternatives management in snapd [12:44] and core was changed to not use such symlinks [12:44] xnox: it's a bit unfortunate because I don't participate in that part of snapd (release) and this is really something mvo should answer [12:44] but he's unlikely to this week [12:44] ackk: envvars do not make it to plugin logic, if you are doing something complex, script the whole thing or write a custom plugin [12:46] diddledan, same error [12:46] sergiusens: we should make available a bare-bones template/example for people to base their custom plugins on [12:46] i.e. a template that does nothing without adding code but shows all the standard functions [12:46] stubs all the standard functions** [12:46] diddledan: we had that in the old docs, I cannot for the life in me find anything in the new ones... [12:49] diddledan, fwiw it'd be nice if you could specify the actual python version, like python3.7 as both 3.6 and 3.7 are available on bionic [12:51] off to pick up the kids [12:52] sergiusens, why am I doing something complex? it's just building a pyhton app, with C modules [12:52] sergiusens, seems something that the python plugin should support, which it could by just exporting C_INCLUDE_PATH to /usr/include/ [12:53] well, I said I would look at the bug later, all you will get from me now are shallow replies [12:55] https://media2.giphy.com/media/JMoyE48gJqivS/giphy.gif?cid=3640f6095c06792f566577304103b645 [13:00] mborzecki, hey, could you please take a look to #6217 [13:00] it is close [13:00] thanks [13:00] PR #6217: tests: reset snapd state on tests restore [13:01] * pstolowski lunch [13:09] * zyga -> cofee [13:10] *coffee [13:12] ackk: move `python3-dev` from `build-packages` to `stage-packages`. [13:29] diddledan, ah, that worked! out of curiosity, why is that needed? [13:42] cachio: hey, can you please work with xnox to run regression tests on https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1778936 [13:42] Bug #1778936: please re-add Support-system-image-read-only-etc.patch [13:42] cachio: it needs a core18 snap with systemd from proposed [13:44] (back from coffee and extra dog walk that he demanded) [13:44] mborzecki: I added set -o traces in various points [13:44] running a test now, let's see [13:45] zyga, sure [13:45] thank you [13:45] xnox: ^ cachio will help with testing that [13:45] diddledan, thanks for the tip btw [13:52] mborzecki: set -e is set when the test executes, as expected [13:53] mborzecki: comparing that to what happens in master [13:53] this is really puzzling [13:56] 2018-12-04 14:54:52 Failed tasks: 3 <-- that's with fedora 29 [13:56] mborzecki: which tasks failed? [13:57] zyga: tests/main/document-portal-activation tests/main/prepare-image-uboot and tests/main/snap-run [13:58] interesting [13:58] what did you change? [13:58] added a workaround for snap-device-helper [13:59] mborzecki: heh [13:59] man [13:59] it passes on master [13:59] so yes [13:59] something is broken in our spread test suite [13:59] running with --shell-after now [13:59] hahah [13:59] I will shortly know what's the cause [14:00] we should write go scripts [14:00] with the tracing added, all I need is to diff the right spot [14:00] :/ [14:00] #!/usr/bin/go run [14:00] we should not source stuff [14:00] write tools [14:00] not shell functions [14:00] and migrate away from that [14:00] then tools can become non-shell [14:01] damn, standup [14:01] joining [14:02] pstolowski: standup [14:04] mborzecki: can you ack/merge https://github.com/snapcore/snapd/pull/6257 [14:04] PR #6257: testutils: split checkers, tweak tests [14:05] PR snapd#6257 closed: testutils: split checkers, tweak tests [14:20] zyga: mborzecki: hi, I don't the situation of the arch user that lost the model assertion, afaict the code sets the state only after having added the assertion [14:20] * I don'tunderstand [14:20] ogra: where can i find the ubuntu core arm64 image for the raspberrypi3? [14:20] mborzecki: if you can spend some thinking what could delete it, I don't have brain cycle for it atm [14:20] pedronis: note that i have a core device in similar state, I have the state of that if someone wants to attack that at some point [14:21] ppisati, somewhere in mvo's people account [14:21] IIRC [14:22] ppisati, https://forum.snapcraft.io/t/pi3-64-unofficial-image/8317 [14:22] zyga: that sound bad, I don't know any code that would remove a model assertion that was set [14:22] sounds a serious bug [14:22] ogra: ack, thank you [14:22] ppisati, did you get anywhere with A+ and vc4 ? [14:23] ogra: oh yeah, it already spawned 2/3 lp bugs [14:23] ogra: cma=256M is the problem [14:23] well, thats builtin [14:23] ogra: this is the oops - https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1805117 [14:23] Bug #1805117: Xenial/raspi2 and RaspberryPi3a+: OOPS during boot [14:24] the driver has it hardcoded somewhere ... IIRC [14:24] ogra: and this is the cma / deadlock: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1805174 [14:24] Bug #1805174: Xenial/raspi2 and RaspberryPi3A+: video not working [14:24] ogra: yeah. but you can specify cma=a on the dtparam [14:24] ogra: wait, let me find [14:24] btw, the closed driver works really fine [14:24] yeah, i know [14:28] ogra: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/tree/arch/arm/boot/dts/overlays/README?h=raspi2 [14:28] ogra: search for "vc4-kms-v3d" [14:28] ogra: Params: cma-256 CMA is 256MB, 256MB-aligned (needs 1GB) [14:28] ogra: while the rpi3a+ has 512MB [14:28] ogra: pass cma-128 and it will work [14:29] yeah, i got that ... my prob is that i have no conditional i could use in config.txt to actually set it dynamically [14:29] and i dont want an A+ sepcific image [14:31] mborzecki: zyga: notice that those systems could be in that state since long, is just that 2.36 is surfacing now the bad state [14:31] pedronis: in my case I updated a device after fresh install [14:31] the alternative would be to simply default to cma-128, but that degrades the actual pi3 i guess [14:31] this was circa ~ 3 weeks ago [14:31] the device is off since then [14:32] pedronis: right, that's my hunch as well, added a note in the forum [14:32] zyga: as I said the update would be make the bad state appear [14:32] I don't think it removed the model [14:32] ogra: the problem is that the fw set cma=256 even on that board, and the drivers locks up with that much memory [14:32] zyga: I might be missing something [14:32] tough [14:32] ogra: i can fix the oops, but then ther's another problem after that [14:32] yeah [14:32] right [14:32] pedronis: I will look at that device tomorrow [14:33] working with maciek on mount error now [14:33] i understand ... that why i would like to simply do it via a conditional in config.txt and have it pick the right value ... https://www.raspberrypi.org/documentation/configuration/config-txt/conditional.md [14:33] *that's [14:33] but i can neither filter on pi3 vs A+ nor on available RAM [14:34] (would be all easier if we could apply the overlays from u-boot ... then i could just check what board i'm on easily) [15:08] PR snapd#6262 opened: apparmor: allow snap-update-ns access to common devices [15:14] having a .snapcraft as an alternate to snap is going to be a bigger code change than I wanted [15:15] sudo umount /tmp/dupa-* && sh -c 'for i in $(seq 1000); do mkdir -p /tmp/dupa-$i && umount /tmp/dupa-$i 2>/dev/null; cp foo.snap foo-$i.snap && LIBMOUNT_DEBUG=all mount foo-$i.snap /tmp/dupa-$i >mount.$i.log 2>&1 & done' [15:16] er [15:16] sudo sh -c 'umount /tmp/dupa-* 2>/dev/null; for i in $(seq 1000); do mkdir -p /tmp/dupa-$i && cp foo.snap foo-$i.snap && LIBMOUNT_DEBUG=all mount foo-$i.snap /tmp/dupa-$i >mount.$i.log 2>&1 & done' [15:17] with waiting: sudo sh -c 'umount /tmp/dupa-* 2>/dev/null; for i in $(seq 1000); do mkdir -p /tmp/dupa-$i && cp foo.snap foo-$i.snap && LIBMOUNT_DEBUG=all mount foo-$i.snap /tmp/dupa-$i >mount.$i.log 2>&1 & done; wait' [15:18] mborzecki: ^ those take a lot of CPU from systemd and various mount-observing tools [15:23] ha [15:23] when running many mount in parallel [15:23] I got: [15:23] systemd[PID]: Failed to get udev event [15:25] sudo sh -c 'umount /tmp/dupa-* 2>/dev/null; for i in $(seq 1000); do mkdir -p /tmp/dupa-$i && cp foo.snap foo-$i.snap; done; for i in $(seq 1000); do LIBMOUNT_DEBUG=all mount foo-$i.snap /tmp/dupa-$i >mount.$i.log 2>&1 & done; wait'; grep 'final srcpath' mount.*.log | cut -f 6 -d ':' | sort | uniq -c | cut -c 7- | cut -d ' ' -f 1 | sort | uniq -c [15:27] mborzecki: can you copy any snap you have on hand to foo.snap [15:27] mborzecki: and run that [15:27] * zyga -> dinner [15:38] sergiusens: please tell me this isn't another place where snapcraft.yaml can live? we already have three :( [15:47] * cachio lunch [16:03] re [16:03] jdstrand_: thank you for the review [16:03] I will jump to that shortly [16:08] cjwatson: the snapd team does not want to rename their in code "snap" directory [16:09] so it came as a requirement to change it [16:09] Well, just remember that adding more locations for snapcraft.yaml requires changing at least LP and BSI too [16:10] BSI? [16:10] And may cause operational problems since it will increase the number of queries BSI is making to GitHub (although hopefully not by very much) [16:10] build.snapcraft.io [16:10] ah [16:10] cjwatson: yeah, it sucks [16:10] just to avoid a big sed in the code, we get to add logic all over the place [16:10] sergiusens: can you file bugs on LP and BSI once the snapcraft change lands, if it's unavoidable? [16:19] sure thing [16:19] thanks [16:28] PR snapcraft#2399 closed: cmake plugin: use build snaps to search paths [16:36] mborzecki: still there? [16:36] no luck on 16.04 either [16:39] mborzecki: but interestingly I can easily trigger this: [16:39] https://www.irccloud.com/pastebin/UFMlCwov/ [16:39] so that part is working as intended [16:47] gru 04 08:44:10 ubuntu snapd[897]: 2018/12/04 08:44:10 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n

408 Request Time-out

\nYour browser didn't send a complete request in time.\n\n"; err= [16:48] hmm? [16:55] aloha, I need to change ownership of a snap, specifically https://snapcraft.io/scout-plane/listing, from myself to a colleague. Can I get help with that here? [16:55] zyga: nice [16:55] katamo: yep, but probably requires a forum topic too, [16:55] katamo: best to ask in the store category on the forum [16:55] mborzecki: yep [16:55] mborzecki: so, ... [16:55] popey ack, ty [16:55] mborzecki: I'm running this in parallel with systemd restarts now [16:55] no luck [16:55] restarts? [16:56] daemon-reload [16:56] ah ok [16:56] zyga: have you tried the reproducer script yet? [16:56] zyga: https://gist.github.com/bboozzoo/d4b142229b1915ef7cc0cf8593599ad9 [16:56] yeah but I want to find a deeper angle on this :), the erorr was, clearly, from libmount, not from systemd itself [16:56] system caught and reported it [16:56] that's fine [16:59] zyga: maybe a problem is with loading mountinfo in systemd? [16:59] hmmm [16:59] interesting idea [16:59] zyga: iirc that's done separately in io loop [16:59] worth looking [16:59] zyga: and there's that super ineresting comment: [16:59] /* Note that due to the io event priority logic, we can be sure the new mountinfo is loaded [16:59] * before we process the SIGCHLD for the mount command. */ [16:59] yeah, I read that [16:59] it looks correct though [16:59] one more thing to do [17:00] is to breakpoint there [17:00] when it cannot find it [17:00] and rewind enough state to look around === pstolowski is now known as pstolowski|afk === jdstrand_ is now known as jdstrand [19:11] PR snapcraft#2419 opened: project: state file path change [19:17] PR snapd#6263 opened: Retrieving all pages for 'find' command, not only one [20:46] jdstrand: hey, updated https://github.com/snapcore/snapd/pull/6244 [20:46] PR #6244: release: detect too old apparmor_parser [20:46] I'd like to land it as-is, I'm still doing one aspect [20:47] when we have a object (kernel or parser) and some features [20:47] there are features we want to have to be fully operational [20:47] features we want to have to be partially operational [20:47] and features we must have to be in any way operational [20:47] you are right this is not captured [20:47] I'd like to do that properly with a follow-up [20:47] I added a few missing tests, got coverage to 100% [21:16] * jdstrand responded in the pr [21:32] jdstrand: thanks, lookin g [21:35] jdstrand: replied there [21:35] jdstrand: totally offtopic [21:35] do you know what's the status of the upstreaming effort [21:36] I'm running tumblweed instead of ubuntu recently and I keep taking notice that we're not fully supported yet === gurmble is now known as grumble [21:51] zyga: af_unix is still missing. it is the last thing. I know jjohansen was working to get things into the latest merge window and af_unix was on the table, but not sure if he got it in [21:51] cool, fingers crossed :) [21:51] using TW is like being on debian sid [21:51] but without any debian specific things I may assume are true universally [21:51] zyga: note that once af_unix is in, will need apparmor 3 userspace for (at least) the networking bits [21:51] it's been a fun journey [21:52] is apparmor userspace 3 released? [21:52] not yet [21:53] it is also planned, but behind the af_unix mediation and a few other things. it, those things and af_unix are all for this cycle [21:53] and by cycle, I mean our dev cycle, not the upstream kernel or anything else [21:53] do you know if the eventual release will affect ubuntu that has carried the earlier patches in any way? [21:53] aha [21:54] zyga: I don't hve the details on that, but we'll definitely make sure we have things working everywhere in Ubuntu [22:21] jdstrand: I'm about to EOD but I'd love a one last look at https://github.com/snapcore/snapd/pull/6244 [22:21] PR #6244: release: detect too old apparmor_parser [22:21] ah, I see you replied [22:22] again, stale page :/ [22:22] jdstrand: I'll tweak this tomorrow, thank you :) [22:23] zyga: np. happy to review again tomorrow. take it easy :) [23:16] jdstrand: the tools r1167 are now merged in the store's trunk and should be deployed some time this week, ideally tomorrowish [23:21] roadmr: thanks! :)