[02:14] <Son_Goku> zyga, I need karma for these updates:
[02:14] <Son_Goku> F28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2d75eaae81
[02:15] <Son_Goku> EL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-be55f1a139
[02:15] <Son_Goku> F29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a93e7637d2
[02:30] <mup> PR snapd#6697 opened: interfaces/daemon_notify: add {net,sys}_admin capabilities, update spread test <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/6697>
[05:16] <mborzecki> morning
[05:19] <mborzecki> google:ubuntu-18.10-64:tests/main/refresh-app-awareness failing randomly?
[05:28] <zyga> Good morning
[05:28] <zyga> Oh
[05:28] <zyga> How?
[05:28] <zyga> I am afk with the dog
[05:36] <mborzecki> zyga: hey
[05:44] <zyga> Back
[05:45] <zyga> mborzecki: how are you doing
[05:46] <zyga> mborzecki: I'll test neal's package update
[05:50] <mborzecki> zyga: found some weird things about snap[d] binaries with mvo on friday
[05:50] <zyga> what did you find?
[05:51] <mborzecki> zyga: we don't know why that's happening yet, but armhf binaries coming from core snap would attempt to perform mmap2() with ridiculously large sizes
[05:51] <mborzecki> zyga: afaik that's the initial allocations that go runtime attempts
[05:51] <zyga> anonymous mapping?
[05:51] <zyga> go's new GC
[05:52] <mborzecki> yes, MAP_ANONYMOUS, and some were done with | MAP_FIXED
[05:52] <zyga> oh
[05:52] <zyga> that is interesting
[05:52] <mborzecki> for instance, the `snap` comand would try to alocate 150MB with MAP_ANONYMOUS | MAP_FIXED, and then another 800M with MAP_ANONYMOUS
[05:52] <zyga> fixed implies they use address-based zones
[05:52] <mborzecki> i was like wtf?
[05:52] <zyga> funny
[05:53] <zyga> I was reading a book about this over weekend
[05:53] <zyga> https://www.amazon.de/Garbage-Collection-Handbook-Automatic-Management/dp/1420082795/ref=sr_1_1?__mk_pl_PL=ÅMÅŽÕÑ&keywords=garbage+collection&qid=1554702790&s=gateway&sr=8-1
[05:53] <zyga> I recommend it
[05:53] <zyga> that's the new garbage collector in place
[05:53] <zyga> since it's just address space it's okay
[05:53] <zyga> but pehraps it has some side effects
[05:53] <mborzecki> fwiw, i think it's exhausting system wide overcommit limits
[05:54] <mborzecki> zyga: and we found some bug reports about that for 32bit arches
[05:54] <zyga> mmm
[05:54] <zyga> so
[05:55] <zyga> google doesn't really care about 32bit
[05:55] <mborzecki> zyga: oh, and another thing, i dropped a snap binary built by me (although cross compiled), but it had none of the crazy mmap() calls
[05:55] <zyga> mmmm
[05:55] <zyga> which go versions were involved?
[05:55] <mborzecki> zyga: theoretically same go version as in xenial (1.10.4), though i got mine from golang.org
[05:56] <zyga> what kind of patches do we see in ubuntu?
[05:56] <mborzecki> zyga: haven't looked at that yet, we stopped at ~11pm on friday :P
[05:56] <zyga> mborzecki: good results!
[05:56] <zyga> mborzecki: I'm eager to learn more as you discover what's changed
[05:56] <zyga> mborzecki: one hint:
[05:57] <zyga> look at golang's source code to see if the new garbage collector is uncoditional and see if there are any knobs (environment?) to control it
[05:57] <zyga> perhaps we cannot reverse course on how go behaves but we can tweak it enough to be sufficient for our needs
[05:57] <zyga> though I strongly worry about the long term viability of 32bit snapd
[05:57] <zyga> unless we change our model
[05:57] <zyga> (e.g. snapd doesn't run unless API calls are in place)
[05:58] <zyga> (and snapd has more smaller binaries for particular tasks)
[05:58] <zyga> (e.g. only daemon runs and dispatches to new binaries for tasks)
[05:59] <mborzecki> zyga: some pages i have open since friday: https://github.com/golang/go/issues/21044 https://github.com/golang/go/issues/28114 https://github.com/golang/go/commit/2b415549b813ba36caafa34fc34d72e47ee8335c
[06:00] <mborzecki> zyga: idk, but armhf is 32bit, rpi3 is still running 32 userspace, unless you forfeit the gpu and run 64bit instead
[06:00] <mborzecki> zyga: oh, and if anyone uses ubuntu core on atom, those are 32bit too
[06:00] <zyga> mborzecki: sure, I know we will be stuck with 32bit systems for al ong time
[06:01] <mborzecki> zyga: btw. on friday we got to a state, where it wasn't possible to run any snap command :P, you'd type snap list and got runtime: out of memory :P
[06:02] <zyga> rust baby ;-)
[06:02] <zyga> I'd love a rust dispatcher for tasks with go backends to run stuff we have already
[06:02] <zyga> snapd taking 3MB of ram while idle
[06:03] <zyga> we could call it "project schizo"
[06:04] <mborzecki> heh
[06:04] <mborzecki> zyga: fwiw, if any, i think it's a bug somewhere in the compiler or runtime
[06:04] <zyga> bug as in undesired behavior?
[06:04] <zyga> perhaps
[06:04] <mborzecki> zyga: the binaries that i built would ask for 2M not 150M
[06:05] <zyga> I doubt it is an actual algorithm bug
[06:05] <zyga> I wonder if cross compiling is a factor
[06:05] <zyga> perhaps that is the bug
[06:05] <zyga> compile natively on arm perhaps?
[06:05] <mborzecki> yeah, idk, that's why i'm setting up my rpi
[06:05] <zyga> like our builders do
[06:05] <zyga> ok :)
[06:06] <mborzecki> btw. mvo mentioned when he passed mem=256 on kernel command rpi classic wouldn't boot, wonder why
[06:08] <zyga> memory split for gpu?
[06:08] <zyga> I think though that modern systems struggle at that amount
[06:08] <zyga> (128MB)
[06:17] <mborzecki> i should buy a new case for my rpi, one that exposes the gpio header
[06:18] <zyga> mborzecki: just find one you like and I'll print you a copy
[06:18] <zyga> I have some spare official cases
[06:18] <zyga> those expose the gpio header but not in a useful way for serial
[06:38] <zyga> hmm, I have issues with my fedora system :/
[06:40] <mvo> zyga: hey, good morning
[06:40] <zyga> good morning :)
[06:49] <mborzecki> mvo:  hey
[06:50] <mvo> hey mborzecki
[06:50] <zyga> "a start job is running for /dev/mapper/fedora"
[06:50] <zyga> eh
[06:50] <mvo> mborzecki: anything new in armhf land :) ?
[06:50] <zyga> I think my disk got corrupt somehow
[06:51] <mborzecki> mvo: not yet, pulled down rpi image, flashed, and nothing on the serial port :/ maybe my uart adapter dones't work anymore
[06:51] <zyga> mborzecki: screen?
[06:51] <mvo> mborzecki: meh, ok
[06:51] <zyga> mborzecki: how did you wire the serial?
[06:52] <mborzecki> zyga: same as usual, idk i dragged it some meetups recently, maybe something got loose
[06:56] <zyga> I gave up on my fedora system
[06:56] <zyga> it's odd because it can boot to recovery
[06:56] <zyga> but not to normal mode
[06:56] <zyga> I will try some more later
[06:56]  * zyga breaks for quick breakfast and then back to RPs
[06:56] <zyga> PRs
[07:10] <pstolowski> morning
[07:12] <zyga> back from breakfast
[07:12] <zyga> hey pawel
[07:49] <mvo> ppisati: can I run a 64bit pi3 kernel on a 32bit userland, what do I need to do to get this working? tracking down a nasty go bug and I want to replicate our buildd setup as closely as possible
[07:50] <mvo> sil2100: do you know if we have a machine to native build on arm that is close to a buildd? i.e. 64bit kernel, 32bit userland ? I tried the armhf porter box but for some reason I don't get access there
[07:50] <pedronis> mvo: mborzecki: hi, is this that you are observing: https://github.com/golang/go/issues/28081 ?
[07:51] <mborzecki> pedronis: maybe, we looked at that on friday too
[07:51] <mborzecki> pedronis: if you scroll back, i liked some other issues too
[07:53] <mvo> pedronis: we are still a bit in the dark, the worst part it is not reproducible on a pi2 (neither core nor classic)
[07:53] <mvo> pedronis: and it goes away when cross building on amd64 and scp to arm even with the same toolchain (or we think its the same toolchain, we need to 100% replicate I think)
[07:54] <mvo> pedronis: some (very hard to read notes) https://gist.github.com/mvo5/f3a4e330f8bf32bd514d9a038665f591 - mborzecki has a similar stakc of notes
[07:55] <mvo> pedronis: my theory right now is that the allocator tries to map a way to large chunk of "mheap_arena_used" but its totally unclear why - it may be a red herring but its definitely wrong and we don't see it anywhere else
[07:55] <pedronis> mvo: well they tell they are allocating much larger chunks
[07:55] <pedronis> 64mb vs 2mb
[07:55] <mborzecki> pedronis: fwiw, there's a request for 800M at some point
[07:56] <pedronis> ah
[07:56] <mborzecki> in the notes, ther's a strace log, first mmap requests 115MB (well, large but ok), and then the next requests 800M+
[07:56] <mborzecki> and that's when even snap commands would keep failing
[07:57] <pedronis> mborzecki: that's on the device ?
[07:57] <mborzecki> pedronis: yes
[07:58] <pedronis> but we don't see it on rpi2
[07:58] <pedronis> (so far)
[07:58] <mvo> mborzecki, pedronis AIUI the big problem is MAP_FIXED here
[08:04] <sil2100> mvo: hm hm, did you have access to the porter boxes previously?
[08:08] <pedronis> mborzecki: are we seeing the same allocation (800M+) on rpi2 and they succeed ? or we not seeing them instead?
[08:16] <mvo> sil2100: do we have a arm porter box? maybe I was doing it wrong
[08:30] <mborzecki> wow, my rpi3 does not even boot anymore
[08:49] <mvo> mborzecki: did you add mem=256M the the kernel commandline ? I tried that and nothing worked anymore
[08:50] <mvo> mborzecki: as in - I would not even get kernel messages that it initializes correctly
[08:50] <mborzecki> mvo: i can't boot any of the rpi images on my rpi3, only raspbian works
[08:50] <mvo> mborzecki: woah, thats strange
[08:50] <zyga> mborzecki: do you have the 3B+ model?
[08:50] <zyga> that is not supported yet officially AFAIR
[08:51] <zyga> for that reason my 3B+ is also running raspbian
[08:51] <zyga> mborzecki: note, if you need a 3B I can arrange on in ~1 hour
[08:51] <zyga> exposed over the network for you
[08:51] <mborzecki> zyga: i used a core image before on this device
[08:51] <ppisati> mvo: if you don't use uboot, copy the arm64 kernel in the vfat partition and call it kernel8.img, remove kernel7.img, and have a fairly recent firmware, it should work
[08:51] <zyga> mborzecki: oh
[08:51] <zyga> that's new
[08:52] <mborzecki> i remembed i did some tweaking, but don't recall what exactly that was
[08:55] <mvo> ppisati: thank you!
[09:14] <zyga> mvo, sil2100: https://github.com/snapcore/core18/pull/126
[09:14] <mup> PR core18#126: hooks: reduce snapd skeleton directories <Created by zyga> <https://github.com/snapcore/core18/pull/126>
[09:15] <mup> PR core18#126 opened: hooks: reduce snapd skeleton directories <Created by zyga> <https://github.com/snapcore/core18/pull/126>
[10:22] <zyga> mborzecki: question
[10:22] <zyga> mborzecki: when I mkdir a thing from snap-confine
[10:22] <zyga> mborzecki: should I worry about selinux label?
[10:24] <zyga> mborzecki: specifically, can you review this patch https://github.com/snapcore/snapd/pull/6642/commits/44a0f5283bbf753151e45c6d3fdf7482bf6c2430
[10:24] <mup> PR #6642: cmd/snap-confine: prevent cwd restore permission bypass <Created by zyga> <https://github.com/snapcore/snapd/pull/6642>
[10:24] <zyga> er
[10:24] <zyga> https://github.com/snapcore/snapd/pull/6642/commits/47018f084e66965ef51a67bc80c8820be8b75893
[10:29] <mborzecki> afaict it should be fine, /var/lib/snapd is snappy_var_t, everything created under that dir will get the same label
[10:37] <mup> PR snapd#6698 opened: overlord/ifacestate: introduce HotplugKey type use short key in change summaries <Created by stolowski> <https://github.com/snapcore/snapd/pull/6698>
[11:29] <zyga> pstolowski, mborzecki: thank you for the reviews for https://github.com/snapcore/snapd/pull/6643
[11:29] <mup> PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643>
[11:30] <zyga> I updated the branch in response to the review
[11:30] <zyga> I would appreciate a 2nd look
[11:31]  * zyga switched to investigate google:ubuntu-16.04-64:tests/main/refresh-app-awareness
[11:32] <mborzecki> looks like we may indeed have some memory problems with go 1.7 - 1.10, https://paste.ubuntu.com/p/FqczCC22WN/
[11:33] <mborzecki> that's a request that actually counts towards overcommit, go.16 requested 1M, 1.7-1.10 requests 17M, 1.11 requests 4M
[11:33] <pstolowski> np, zyga: will do later today!
[11:34] <mborzecki> oh, and that's for the minimal program which does virtually nothing
[11:35] <pstolowski> mborzecki: ouch
[11:41] <zyga> in case anyone notices regressions in refresh-app-awareness, please let me know
[11:43] <zyga> Chipaca: hello sir, would you mind attempting to review https://github.com/snapcore/snapd/pull/6690 today?
[11:43] <mup> PR #6690: overlord/snapstate: inhibit refresh for up to a week <Created by zyga> <https://github.com/snapcore/snapd/pull/6690>
[11:43] <Chipaca> zyga: I would not
[11:44] <Chipaca> zyga: but, note today I am not doing too good
[11:44] <zyga> I wish you to get better soon, I understand
[11:50] <pedronis> pstolowski: I did pass a on #6662
[11:50] <mup> PR #6662: overlord/snapstate,snapshotstate: create snapshot on snap removal (3/4) <Created by stolowski> <https://github.com/snapcore/snapd/pull/6662>
[11:50] <pedronis> zyga: hi, have you seen the incoming comments on the new tmp logic ?
[11:51] <pstolowski> pedronis: ty, also for the remarks on 4/4!
[11:53] <pedronis> pstolowski: np
[11:54] <zyga> pedronis: hey, perhaps not, where are they?
[11:55] <pedronis> zyga: https://github.com/snapcore/snapd/pull/6614#discussion_r272974008
[11:55] <mup> PR #6614: cmd/snap-confine: use fixed private tmp directory <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6614>
[11:56] <zyga> ah, interesting, I didn't notice that
[11:56] <zyga> thank you
[11:56] <pedronis> zyga: I will look at 6690 today
[11:57] <zyga> thank you!
[11:57] <zyga> I will check if there are any other comments on closed branches
[11:57] <zyga> my github.com/notifications is rather long
[12:05] <mup> PR snapd#6699 opened: daemon: add RootOnly flag to commands <Simple 😃> <Created by chipaca> <https://github.com/snapcore/snapd/pull/6699>
[12:05] <zyga> pstolowski: reviewed https://github.com/snapcore/snapd/pull/6698#pullrequestreview-223799068
[12:05] <mup> PR #6698: overlord/ifacestate: introduce HotplugKey type use short key in change summaries <Created by stolowski> <https://github.com/snapcore/snapd/pull/6698>
[12:07] <zyga> Chipaca: is https://github.com/snapcore/snapd/pull/6699/files driven by the suse audit?
[12:07] <mup> PR #6699: daemon: add RootOnly flag to commands <Simple 😃> <Created by chipaca> <https://github.com/snapcore/snapd/pull/6699>
[12:07] <Chipaca> zyga: no, it's one of the changes to users raised last week
[12:07] <Chipaca> there's another two coming
[12:07] <Chipaca> (one of which needs agreement before i do it)
[12:07] <zyga> so it's unrelated to the security review we received?
[12:07] <Chipaca> maybe even three
[12:08] <Chipaca> zyga: it changes nothing security-wise beyond perhaps making the logic easier to follow
[12:08] <zyga> understood, thank you
[12:08] <Chipaca> (which is a win)
[12:08] <Chipaca> zyga: one of the things coming is to not have the user endpoints when not needed/supported, which might impact the sec review
[12:09] <zyga> thank you
[12:09] <Chipaca> zyga: another change is moving the user stuff to api_user
[12:09] <Chipaca> zyga: a further change that i'd like to do but it's arguably v3 material is moving all the user actions to /v2/users
[12:12] <zyga> Chipaca: I reviewed the auth code
[12:28]  * zyga -> small break
[12:33] <pedronis> Chipaca: hi, what's the status of #6653 ?
[12:33] <mup> PR #6653: tests: try to be a little bit invariant <Created by chipaca> <https://github.com/snapcore/snapd/pull/6653>
[12:34] <Chipaca> pedronis: need to fix it
[12:43]  * zyga reads tests/main/security-private-tmp and is puzzled
[12:43] <zyga> looking at it now
[13:07] <pedronis> Chipaca: standup?
[13:07] <Chipaca> pedronis: ye
[13:19] <zyga> ah, I think I understand what the problem is
[13:19] <zyga> with refresh awareness thing
[14:24] <zyga> re, partially
[15:18] <zyga> mborzecki: I'm fixing that bug you ran into in the morning
[15:18] <zyga> just still in partial capacity away from the office
[15:19] <Chipaca> what is /usr/lib/sysimage/rpm/Basenames and why do i hate it
[15:19] <zyga> whaat?
[15:19] <zyga> no idea, which system is that on?
[15:44] <Chipaca> zyga: opensuse-15.0-64
[16:14] <pstolowski> zyga: have you seen my general meta-question re #6643?
[16:14] <mup> PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643>
[17:02] <Chipaca> hrmph, it looks like core18 spread tests are rather messy wrt leaving core behind
[17:22] <Chipaca> EOD for me
[17:22] <Chipaca> o/
[17:22] <zyga> pstolowski|afk: not yet, looking now
[17:24] <zyga> pstolowski|afk: responded
[17:24] <pstolowski|afk> zyga: ty
[17:25]  * zyga resumes work
[17:25] <zyga> sorry, took a nap after dinner
[17:43] <zyga> hmm
[17:43] <zyga> core18 prepare hanging ... ? https://www.irccloud.com/pastebin/MyVsgbcc/
[17:47] <zyga> hmm
[17:47] <zyga> but the system has seeded
[17:47] <zyga> changes from the system https://www.irccloud.com/pastebin/loXhyaB3/
[17:58] <zyga> I think I fixed that  now
[17:58] <zyga> another round of testing...
[18:59] <zyga> almost there
[19:12] <om26er> is there a way to tell multipass through snapcraft to use more cores when packing a snap ? currently it default to 2, I would like it to use all available threads
[19:23] <om26er> zyga Hey! Were you able to find a fix for https://bugs.launchpad.net/snapd/+bug/1819318 ?
[19:23] <mup> Bug #1819318: no interfaces when installing only core18 <snapd:In Progress by zyga> <https://launchpad.net/bugs/1819318>
[19:24] <om26er> So we have a server software as a snap and it won't build on 16.04 because a package it depends on, requires a version of python that is not in 16.04
[19:25] <om26er> If we go with base: core18 then the above issue is problematic as Ubuntu Server on many VPSs doesn't come with core preinstalled
[19:28] <zyga> and ... got it
[19:28] <zyga> om26er: yes, but it's not near yet
[19:28] <zyga> om26er: we need to install core in such cases
[19:28] <zyga> I spent some time researching this but it's the closes solution for now
[19:29]  * zyga just understood a pair of bugs
[19:30] <om26er> that's kind of the workaround I have already been using, even before discovering ogra's bug ;-)
[19:30] <om26er> will see if we can update our docs...
[19:30] <zyga> it will get properly fixed by snapd.snap
[19:30] <zyga> but for now...
[19:30] <zyga> nothing else
[19:56] <om26er> re: the python plugin in snapcraft, is there a way to set the `source` to a specific branch ?
[20:19] <zyga> GAAH
[20:22] <roadmr>  /o\
[20:35] <mup> PR snapcraft#2524 closed: Snapcraft try <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2524>
[20:38] <mup> PR snapcraft#2519 closed: packaging: use local patchelf <Created by cmatsuoka> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2519>
[20:38] <mup> PR snapcraft#2525 opened: catkin plugin: check stage-snaps for ROS dependencies <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2525>
[20:39] <mup> PR # closed: core-build#11, core-build#22, core-build#26, core-build#37
[20:40] <mup> PR # opened: core-build#11, core-build#22, core-build#26, core-build#37
[20:44] <mup> PR snapcraft#2526 opened: Release changelog for 3.4 <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2526>