[02:14] zyga, I need karma for these updates: [02:14] F28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2d75eaae81 [02:15] EL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-be55f1a139 [02:15] F29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a93e7637d2 [02:30] PR snapd#6697 opened: interfaces/daemon_notify: add {net,sys}_admin capabilities, update spread test === chihchun_afk is now known as chihchun [05:16] morning [05:19] google:ubuntu-18.10-64:tests/main/refresh-app-awareness failing randomly? [05:28] Good morning [05:28] Oh [05:28] How? [05:28] I am afk with the dog [05:36] zyga: hey [05:44] Back [05:45] mborzecki: how are you doing [05:46] mborzecki: I'll test neal's package update [05:50] zyga: found some weird things about snap[d] binaries with mvo on friday [05:50] what did you find? [05:51] zyga: we don't know why that's happening yet, but armhf binaries coming from core snap would attempt to perform mmap2() with ridiculously large sizes [05:51] zyga: afaik that's the initial allocations that go runtime attempts [05:51] anonymous mapping? [05:51] go's new GC [05:52] yes, MAP_ANONYMOUS, and some were done with | MAP_FIXED [05:52] oh [05:52] that is interesting [05:52] for instance, the `snap` comand would try to alocate 150MB with MAP_ANONYMOUS | MAP_FIXED, and then another 800M with MAP_ANONYMOUS [05:52] fixed implies they use address-based zones [05:52] i was like wtf? [05:52] funny [05:53] I was reading a book about this over weekend [05:53] https://www.amazon.de/Garbage-Collection-Handbook-Automatic-Management/dp/1420082795/ref=sr_1_1?__mk_pl_PL=ÅMÅŽÕÑ&keywords=garbage+collection&qid=1554702790&s=gateway&sr=8-1 [05:53] I recommend it [05:53] that's the new garbage collector in place [05:53] since it's just address space it's okay [05:53] but pehraps it has some side effects [05:53] fwiw, i think it's exhausting system wide overcommit limits [05:54] zyga: and we found some bug reports about that for 32bit arches [05:54] mmm [05:54] so [05:55] google doesn't really care about 32bit [05:55] zyga: oh, and another thing, i dropped a snap binary built by me (although cross compiled), but it had none of the crazy mmap() calls [05:55] mmmm [05:55] which go versions were involved? [05:55] zyga: theoretically same go version as in xenial (1.10.4), though i got mine from golang.org [05:56] what kind of patches do we see in ubuntu? [05:56] zyga: haven't looked at that yet, we stopped at ~11pm on friday :P [05:56] mborzecki: good results! [05:56] mborzecki: I'm eager to learn more as you discover what's changed [05:56] mborzecki: one hint: [05:57] look at golang's source code to see if the new garbage collector is uncoditional and see if there are any knobs (environment?) to control it [05:57] perhaps we cannot reverse course on how go behaves but we can tweak it enough to be sufficient for our needs [05:57] though I strongly worry about the long term viability of 32bit snapd [05:57] unless we change our model [05:57] (e.g. snapd doesn't run unless API calls are in place) [05:58] (and snapd has more smaller binaries for particular tasks) [05:58] (e.g. only daemon runs and dispatches to new binaries for tasks) [05:59] zyga: some pages i have open since friday: https://github.com/golang/go/issues/21044 https://github.com/golang/go/issues/28114 https://github.com/golang/go/commit/2b415549b813ba36caafa34fc34d72e47ee8335c [06:00] zyga: idk, but armhf is 32bit, rpi3 is still running 32 userspace, unless you forfeit the gpu and run 64bit instead [06:00] zyga: oh, and if anyone uses ubuntu core on atom, those are 32bit too [06:00] mborzecki: sure, I know we will be stuck with 32bit systems for al ong time [06:01] zyga: btw. on friday we got to a state, where it wasn't possible to run any snap command :P, you'd type snap list and got runtime: out of memory :P [06:02] rust baby ;-) [06:02] I'd love a rust dispatcher for tasks with go backends to run stuff we have already [06:02] snapd taking 3MB of ram while idle [06:03] we could call it "project schizo" [06:04] heh [06:04] zyga: fwiw, if any, i think it's a bug somewhere in the compiler or runtime [06:04] bug as in undesired behavior? [06:04] perhaps [06:04] zyga: the binaries that i built would ask for 2M not 150M [06:05] I doubt it is an actual algorithm bug [06:05] I wonder if cross compiling is a factor [06:05] perhaps that is the bug [06:05] compile natively on arm perhaps? [06:05] yeah, idk, that's why i'm setting up my rpi [06:05] like our builders do [06:05] ok :) [06:06] btw. mvo mentioned when he passed mem=256 on kernel command rpi classic wouldn't boot, wonder why [06:08] memory split for gpu? [06:08] I think though that modern systems struggle at that amount [06:08] (128MB) [06:17] i should buy a new case for my rpi, one that exposes the gpio header [06:18] mborzecki: just find one you like and I'll print you a copy [06:18] I have some spare official cases [06:18] those expose the gpio header but not in a useful way for serial [06:38] hmm, I have issues with my fedora system :/ [06:40] zyga: hey, good morning [06:40] good morning :) [06:49] mvo: hey [06:50] hey mborzecki [06:50] "a start job is running for /dev/mapper/fedora" [06:50] eh [06:50] mborzecki: anything new in armhf land :) ? [06:50] I think my disk got corrupt somehow [06:51] mvo: not yet, pulled down rpi image, flashed, and nothing on the serial port :/ maybe my uart adapter dones't work anymore [06:51] mborzecki: screen? [06:51] mborzecki: meh, ok [06:51] mborzecki: how did you wire the serial? [06:52] zyga: same as usual, idk i dragged it some meetups recently, maybe something got loose [06:56] I gave up on my fedora system [06:56] it's odd because it can boot to recovery [06:56] but not to normal mode [06:56] I will try some more later [06:56] * zyga breaks for quick breakfast and then back to RPs [06:56] PRs [07:10] morning [07:12] back from breakfast [07:12] hey pawel [07:49] ppisati: can I run a 64bit pi3 kernel on a 32bit userland, what do I need to do to get this working? tracking down a nasty go bug and I want to replicate our buildd setup as closely as possible [07:50] sil2100: do you know if we have a machine to native build on arm that is close to a buildd? i.e. 64bit kernel, 32bit userland ? I tried the armhf porter box but for some reason I don't get access there [07:50] mvo: mborzecki: hi, is this that you are observing: https://github.com/golang/go/issues/28081 ? [07:51] pedronis: maybe, we looked at that on friday too [07:51] pedronis: if you scroll back, i liked some other issues too [07:53] pedronis: we are still a bit in the dark, the worst part it is not reproducible on a pi2 (neither core nor classic) [07:53] pedronis: and it goes away when cross building on amd64 and scp to arm even with the same toolchain (or we think its the same toolchain, we need to 100% replicate I think) [07:54] pedronis: some (very hard to read notes) https://gist.github.com/mvo5/f3a4e330f8bf32bd514d9a038665f591 - mborzecki has a similar stakc of notes [07:55] pedronis: my theory right now is that the allocator tries to map a way to large chunk of "mheap_arena_used" but its totally unclear why - it may be a red herring but its definitely wrong and we don't see it anywhere else [07:55] mvo: well they tell they are allocating much larger chunks [07:55] 64mb vs 2mb [07:55] pedronis: fwiw, there's a request for 800M at some point [07:56] ah [07:56] in the notes, ther's a strace log, first mmap requests 115MB (well, large but ok), and then the next requests 800M+ [07:56] and that's when even snap commands would keep failing [07:57] mborzecki: that's on the device ? [07:57] pedronis: yes [07:58] but we don't see it on rpi2 [07:58] (so far) [07:58] mborzecki, pedronis AIUI the big problem is MAP_FIXED here [08:04] mvo: hm hm, did you have access to the porter boxes previously? [08:08] mborzecki: are we seeing the same allocation (800M+) on rpi2 and they succeed ? or we not seeing them instead? [08:16] sil2100: do we have a arm porter box? maybe I was doing it wrong [08:30] wow, my rpi3 does not even boot anymore === chihchun is now known as chihchun_afk [08:49] mborzecki: did you add mem=256M the the kernel commandline ? I tried that and nothing worked anymore [08:50] mborzecki: as in - I would not even get kernel messages that it initializes correctly [08:50] mvo: i can't boot any of the rpi images on my rpi3, only raspbian works [08:50] mborzecki: woah, thats strange [08:50] mborzecki: do you have the 3B+ model? [08:50] that is not supported yet officially AFAIR [08:51] for that reason my 3B+ is also running raspbian [08:51] mborzecki: note, if you need a 3B I can arrange on in ~1 hour [08:51] exposed over the network for you [08:51] zyga: i used a core image before on this device [08:51] mvo: if you don't use uboot, copy the arm64 kernel in the vfat partition and call it kernel8.img, remove kernel7.img, and have a fairly recent firmware, it should work [08:51] mborzecki: oh [08:51] that's new [08:52] i remembed i did some tweaking, but don't recall what exactly that was [08:55] ppisati: thank you! === chihchun_afk is now known as chihchun [09:14] mvo, sil2100: https://github.com/snapcore/core18/pull/126 [09:14] PR core18#126: hooks: reduce snapd skeleton directories [09:15] PR core18#126 opened: hooks: reduce snapd skeleton directories === cpaelzer__ is now known as cpaelzer === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun [10:22] mborzecki: question [10:22] mborzecki: when I mkdir a thing from snap-confine [10:22] mborzecki: should I worry about selinux label? [10:24] mborzecki: specifically, can you review this patch https://github.com/snapcore/snapd/pull/6642/commits/44a0f5283bbf753151e45c6d3fdf7482bf6c2430 [10:24] PR #6642: cmd/snap-confine: prevent cwd restore permission bypass [10:24] er [10:24] https://github.com/snapcore/snapd/pull/6642/commits/47018f084e66965ef51a67bc80c8820be8b75893 [10:29] afaict it should be fine, /var/lib/snapd is snappy_var_t, everything created under that dir will get the same label [10:37] PR snapd#6698 opened: overlord/ifacestate: introduce HotplugKey type use short key in change summaries [11:29] pstolowski, mborzecki: thank you for the reviews for https://github.com/snapcore/snapd/pull/6643 [11:29] PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits [11:30] I updated the branch in response to the review [11:30] I would appreciate a 2nd look [11:31] * zyga switched to investigate google:ubuntu-16.04-64:tests/main/refresh-app-awareness [11:32] looks like we may indeed have some memory problems with go 1.7 - 1.10, https://paste.ubuntu.com/p/FqczCC22WN/ [11:33] that's a request that actually counts towards overcommit, go.16 requested 1M, 1.7-1.10 requests 17M, 1.11 requests 4M [11:33] np, zyga: will do later today! [11:34] oh, and that's for the minimal program which does virtually nothing [11:35] mborzecki: ouch [11:41] in case anyone notices regressions in refresh-app-awareness, please let me know [11:43] Chipaca: hello sir, would you mind attempting to review https://github.com/snapcore/snapd/pull/6690 today? [11:43] PR #6690: overlord/snapstate: inhibit refresh for up to a week [11:43] zyga: I would not [11:44] zyga: but, note today I am not doing too good [11:44] I wish you to get better soon, I understand [11:50] pstolowski: I did pass a on #6662 [11:50] PR #6662: overlord/snapstate,snapshotstate: create snapshot on snap removal (3/4) [11:50] zyga: hi, have you seen the incoming comments on the new tmp logic ? [11:51] pedronis: ty, also for the remarks on 4/4! [11:53] pstolowski: np [11:54] pedronis: hey, perhaps not, where are they? [11:55] zyga: https://github.com/snapcore/snapd/pull/6614#discussion_r272974008 [11:55] PR #6614: cmd/snap-confine: use fixed private tmp directory [11:56] ah, interesting, I didn't notice that [11:56] thank you [11:56] zyga: I will look at 6690 today [11:57] thank you! [11:57] I will check if there are any other comments on closed branches [11:57] my github.com/notifications is rather long === chihchun is now known as chihchun_afk [12:05] PR snapd#6699 opened: daemon: add RootOnly flag to commands [12:05] pstolowski: reviewed https://github.com/snapcore/snapd/pull/6698#pullrequestreview-223799068 [12:05] PR #6698: overlord/ifacestate: introduce HotplugKey type use short key in change summaries [12:07] Chipaca: is https://github.com/snapcore/snapd/pull/6699/files driven by the suse audit? [12:07] PR #6699: daemon: add RootOnly flag to commands [12:07] zyga: no, it's one of the changes to users raised last week [12:07] there's another two coming [12:07] (one of which needs agreement before i do it) [12:07] so it's unrelated to the security review we received? [12:07] maybe even three [12:08] zyga: it changes nothing security-wise beyond perhaps making the logic easier to follow [12:08] understood, thank you [12:08] (which is a win) [12:08] zyga: one of the things coming is to not have the user endpoints when not needed/supported, which might impact the sec review [12:09] thank you [12:09] zyga: another change is moving the user stuff to api_user [12:09] zyga: a further change that i'd like to do but it's arguably v3 material is moving all the user actions to /v2/users [12:12] Chipaca: I reviewed the auth code === ricab is now known as ricab|lunch [12:28] * zyga -> small break [12:33] Chipaca: hi, what's the status of #6653 ? [12:33] PR #6653: tests: try to be a little bit invariant [12:34] pedronis: need to fix it [12:43] * zyga reads tests/main/security-private-tmp and is puzzled [12:43] looking at it now === chrisccoulson_ is now known as chrisccoulson === chihchun_afk is now known as chihchun [13:07] Chipaca: standup? [13:07] pedronis: ye [13:19] ah, I think I understand what the problem is [13:19] with refresh awareness thing === ddstreet_away is now known as ddstreet [14:24] re, partially [15:18] mborzecki: I'm fixing that bug you ran into in the morning [15:18] just still in partial capacity away from the office [15:19] what is /usr/lib/sysimage/rpm/Basenames and why do i hate it [15:19] whaat? [15:19] no idea, which system is that on? [15:44] zyga: opensuse-15.0-64 [16:14] zyga: have you seen my general meta-question re #6643? [16:14] PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits [17:02] hrmph, it looks like core18 spread tests are rather messy wrt leaving core behind === pstolowski is now known as pstolowski|afk [17:22] EOD for me [17:22] o/ [17:22] pstolowski|afk: not yet, looking now [17:24] pstolowski|afk: responded [17:24] zyga: ty [17:25] * zyga resumes work [17:25] sorry, took a nap after dinner [17:43] hmm [17:43] core18 prepare hanging ... ? https://www.irccloud.com/pastebin/MyVsgbcc/ [17:47] hmm [17:47] but the system has seeded [17:47] changes from the system https://www.irccloud.com/pastebin/loXhyaB3/ [17:58] I think I fixed that now [17:58] another round of testing... [18:59] almost there [19:12] is there a way to tell multipass through snapcraft to use more cores when packing a snap ? currently it default to 2, I would like it to use all available threads [19:23] zyga Hey! Were you able to find a fix for https://bugs.launchpad.net/snapd/+bug/1819318 ? [19:23] Bug #1819318: no interfaces when installing only core18 [19:24] So we have a server software as a snap and it won't build on 16.04 because a package it depends on, requires a version of python that is not in 16.04 [19:25] If we go with base: core18 then the above issue is problematic as Ubuntu Server on many VPSs doesn't come with core preinstalled [19:28] and ... got it [19:28] om26er: yes, but it's not near yet [19:28] om26er: we need to install core in such cases [19:28] I spent some time researching this but it's the closes solution for now [19:29] * zyga just understood a pair of bugs [19:30] that's kind of the workaround I have already been using, even before discovering ogra's bug ;-) [19:30] will see if we can update our docs... [19:30] it will get properly fixed by snapd.snap [19:30] but for now... [19:30] nothing else [19:56] re: the python plugin in snapcraft, is there a way to set the `source` to a specific branch ? [20:19] GAAH [20:22] /o\ [20:35] PR snapcraft#2524 closed: Snapcraft try [20:38] PR snapcraft#2519 closed: packaging: use local patchelf [20:38] PR snapcraft#2525 opened: catkin plugin: check stage-snaps for ROS dependencies [20:39] PR # closed: core-build#11, core-build#22, core-build#26, core-build#37 [20:40] PR # opened: core-build#11, core-build#22, core-build#26, core-build#37 [20:44] PR snapcraft#2526 opened: Release changelog for 3.4