Son_Gokuzyga, I need karma for these updates:02:14
Son_GokuF28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2d75eaae8102:14
Son_GokuEL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-be55f1a13902:15
Son_GokuF29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a93e7637d202:15
mupPR snapd#6697 opened: interfaces/daemon_notify: add {net,sys}_admin capabilities, update spread test <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/6697>02:30
=== chihchun_afk is now known as chihchun
mborzeckigoogle:ubuntu-18.10-64:tests/main/refresh-app-awareness failing randomly?05:19
zygaGood morning05:28
zygaI am afk with the dog05:28
mborzeckizyga: hey05:36
zygamborzecki: how are you doing05:45
zygamborzecki: I'll test neal's package update05:46
mborzeckizyga: found some weird things about snap[d] binaries with mvo on friday05:50
zygawhat did you find?05:50
mborzeckizyga: we don't know why that's happening yet, but armhf binaries coming from core snap would attempt to perform mmap2() with ridiculously large sizes05:51
mborzeckizyga: afaik that's the initial allocations that go runtime attempts05:51
zygaanonymous mapping?05:51
zygago's new GC05:51
mborzeckiyes, MAP_ANONYMOUS, and some were done with | MAP_FIXED05:52
zygathat is interesting05:52
mborzeckifor instance, the `snap` comand would try to alocate 150MB with MAP_ANONYMOUS | MAP_FIXED, and then another 800M with MAP_ANONYMOUS05:52
zygafixed implies they use address-based zones05:52
mborzeckii was like wtf?05:52
zygaI was reading a book about this over weekend05:53
zygaI recommend it05:53
zygathat's the new garbage collector in place05:53
zygasince it's just address space it's okay05:53
zygabut pehraps it has some side effects05:53
mborzeckifwiw, i think it's exhausting system wide overcommit limits05:53
mborzeckizyga: and we found some bug reports about that for 32bit arches05:54
zygagoogle doesn't really care about 32bit05:55
mborzeckizyga: oh, and another thing, i dropped a snap binary built by me (although cross compiled), but it had none of the crazy mmap() calls05:55
zygawhich go versions were involved?05:55
mborzeckizyga: theoretically same go version as in xenial (1.10.4), though i got mine from golang.org05:55
zygawhat kind of patches do we see in ubuntu?05:56
mborzeckizyga: haven't looked at that yet, we stopped at ~11pm on friday :P05:56
zygamborzecki: good results!05:56
zygamborzecki: I'm eager to learn more as you discover what's changed05:56
zygamborzecki: one hint:05:56
zygalook at golang's source code to see if the new garbage collector is uncoditional and see if there are any knobs (environment?) to control it05:57
zygaperhaps we cannot reverse course on how go behaves but we can tweak it enough to be sufficient for our needs05:57
zygathough I strongly worry about the long term viability of 32bit snapd05:57
zygaunless we change our model05:57
zyga(e.g. snapd doesn't run unless API calls are in place)05:57
zyga(and snapd has more smaller binaries for particular tasks)05:58
zyga(e.g. only daemon runs and dispatches to new binaries for tasks)05:58
mborzeckizyga: some pages i have open since friday: https://github.com/golang/go/issues/21044 https://github.com/golang/go/issues/28114 https://github.com/golang/go/commit/2b415549b813ba36caafa34fc34d72e47ee8335c05:59
mborzeckizyga: idk, but armhf is 32bit, rpi3 is still running 32 userspace, unless you forfeit the gpu and run 64bit instead06:00
mborzeckizyga: oh, and if anyone uses ubuntu core on atom, those are 32bit too06:00
zygamborzecki: sure, I know we will be stuck with 32bit systems for al ong time06:00
mborzeckizyga: btw. on friday we got to a state, where it wasn't possible to run any snap command :P, you'd type snap list and got runtime: out of memory :P06:01
zygarust baby ;-)06:02
zygaI'd love a rust dispatcher for tasks with go backends to run stuff we have already06:02
zygasnapd taking 3MB of ram while idle06:02
zygawe could call it "project schizo"06:03
mborzeckizyga: fwiw, if any, i think it's a bug somewhere in the compiler or runtime06:04
zygabug as in undesired behavior?06:04
mborzeckizyga: the binaries that i built would ask for 2M not 150M06:04
zygaI doubt it is an actual algorithm bug06:05
zygaI wonder if cross compiling is a factor06:05
zygaperhaps that is the bug06:05
zygacompile natively on arm perhaps?06:05
mborzeckiyeah, idk, that's why i'm setting up my rpi06:05
zygalike our builders do06:05
zygaok :)06:05
mborzeckibtw. mvo mentioned when he passed mem=256 on kernel command rpi classic wouldn't boot, wonder why06:06
zygamemory split for gpu?06:08
zygaI think though that modern systems struggle at that amount06:08
mborzeckii should buy a new case for my rpi, one that exposes the gpio header06:17
zygamborzecki: just find one you like and I'll print you a copy06:18
zygaI have some spare official cases06:18
zygathose expose the gpio header but not in a useful way for serial06:18
zygahmm, I have issues with my fedora system :/06:38
mvozyga: hey, good morning06:40
zygagood morning :)06:40
mborzeckimvo:  hey06:49
mvohey mborzecki06:50
zyga"a start job is running for /dev/mapper/fedora"06:50
mvomborzecki: anything new in armhf land :) ?06:50
zygaI think my disk got corrupt somehow06:50
mborzeckimvo: not yet, pulled down rpi image, flashed, and nothing on the serial port :/ maybe my uart adapter dones't work anymore06:51
zygamborzecki: screen?06:51
mvomborzecki: meh, ok06:51
zygamborzecki: how did you wire the serial?06:51
mborzeckizyga: same as usual, idk i dragged it some meetups recently, maybe something got loose06:52
zygaI gave up on my fedora system06:56
zygait's odd because it can boot to recovery06:56
zygabut not to normal mode06:56
zygaI will try some more later06:56
* zyga breaks for quick breakfast and then back to RPs06:56
zygaback from breakfast07:12
zygahey pawel07:12
mvoppisati: can I run a 64bit pi3 kernel on a 32bit userland, what do I need to do to get this working? tracking down a nasty go bug and I want to replicate our buildd setup as closely as possible07:49
mvosil2100: do you know if we have a machine to native build on arm that is close to a buildd? i.e. 64bit kernel, 32bit userland ? I tried the armhf porter box but for some reason I don't get access there07:50
pedronismvo: mborzecki: hi, is this that you are observing: https://github.com/golang/go/issues/28081 ?07:50
mborzeckipedronis: maybe, we looked at that on friday too07:51
mborzeckipedronis: if you scroll back, i liked some other issues too07:51
mvopedronis: we are still a bit in the dark, the worst part it is not reproducible on a pi2 (neither core nor classic)07:53
mvopedronis: and it goes away when cross building on amd64 and scp to arm even with the same toolchain (or we think its the same toolchain, we need to 100% replicate I think)07:53
mvopedronis: some (very hard to read notes) https://gist.github.com/mvo5/f3a4e330f8bf32bd514d9a038665f591 - mborzecki has a similar stakc of notes07:54
mvopedronis: my theory right now is that the allocator tries to map a way to large chunk of "mheap_arena_used" but its totally unclear why - it may be a red herring but its definitely wrong and we don't see it anywhere else07:55
pedronismvo: well they tell they are allocating much larger chunks07:55
pedronis64mb vs 2mb07:55
mborzeckipedronis: fwiw, there's a request for 800M at some point07:55
mborzeckiin the notes, ther's a strace log, first mmap requests 115MB (well, large but ok), and then the next requests 800M+07:56
mborzeckiand that's when even snap commands would keep failing07:56
pedronismborzecki: that's on the device ?07:57
mborzeckipedronis: yes07:57
pedronisbut we don't see it on rpi207:58
pedronis(so far)07:58
mvomborzecki, pedronis AIUI the big problem is MAP_FIXED here07:58
sil2100mvo: hm hm, did you have access to the porter boxes previously?08:04
pedronismborzecki: are we seeing the same allocation (800M+) on rpi2 and they succeed ? or we not seeing them instead?08:08
mvosil2100: do we have a arm porter box? maybe I was doing it wrong08:16
mborzeckiwow, my rpi3 does not even boot anymore08:30
=== chihchun is now known as chihchun_afk
mvomborzecki: did you add mem=256M the the kernel commandline ? I tried that and nothing worked anymore08:49
mvomborzecki: as in - I would not even get kernel messages that it initializes correctly08:50
mborzeckimvo: i can't boot any of the rpi images on my rpi3, only raspbian works08:50
mvomborzecki: woah, thats strange08:50
zygamborzecki: do you have the 3B+ model?08:50
zygathat is not supported yet officially AFAIR08:50
zygafor that reason my 3B+ is also running raspbian08:51
zygamborzecki: note, if you need a 3B I can arrange on in ~1 hour08:51
zygaexposed over the network for you08:51
mborzeckizyga: i used a core image before on this device08:51
ppisatimvo: if you don't use uboot, copy the arm64 kernel in the vfat partition and call it kernel8.img, remove kernel7.img, and have a fairly recent firmware, it should work08:51
zygamborzecki: oh08:51
zygathat's new08:51
mborzeckii remembed i did some tweaking, but don't recall what exactly that was08:52
mvoppisati: thank you!08:55
=== chihchun_afk is now known as chihchun
zygamvo, sil2100: https://github.com/snapcore/core18/pull/12609:14
mupPR core18#126: hooks: reduce snapd skeleton directories <Created by zyga> <https://github.com/snapcore/core18/pull/126>09:14
mupPR core18#126 opened: hooks: reduce snapd skeleton directories <Created by zyga> <https://github.com/snapcore/core18/pull/126>09:15
=== cpaelzer__ is now known as cpaelzer
=== chihchun is now known as chihchun_afk
=== chihchun_afk is now known as chihchun
zygamborzecki: question10:22
zygamborzecki: when I mkdir a thing from snap-confine10:22
zygamborzecki: should I worry about selinux label?10:22
zygamborzecki: specifically, can you review this patch https://github.com/snapcore/snapd/pull/6642/commits/44a0f5283bbf753151e45c6d3fdf7482bf6c243010:24
mupPR #6642: cmd/snap-confine: prevent cwd restore permission bypass <Created by zyga> <https://github.com/snapcore/snapd/pull/6642>10:24
mborzeckiafaict it should be fine, /var/lib/snapd is snappy_var_t, everything created under that dir will get the same label10:29
mupPR snapd#6698 opened: overlord/ifacestate: introduce HotplugKey type use short key in change summaries <Created by stolowski> <https://github.com/snapcore/snapd/pull/6698>10:37
zygapstolowski, mborzecki: thank you for the reviews for https://github.com/snapcore/snapd/pull/664311:29
mupPR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643>11:29
zygaI updated the branch in response to the review11:30
zygaI would appreciate a 2nd look11:30
* zyga switched to investigate google:ubuntu-16.04-64:tests/main/refresh-app-awareness11:31
mborzeckilooks like we may indeed have some memory problems with go 1.7 - 1.10, https://paste.ubuntu.com/p/FqczCC22WN/11:32
mborzeckithat's a request that actually counts towards overcommit, go.16 requested 1M, 1.7-1.10 requests 17M, 1.11 requests 4M11:33
pstolowskinp, zyga: will do later today!11:33
mborzeckioh, and that's for the minimal program which does virtually nothing11:34
pstolowskimborzecki: ouch11:35
zygain case anyone notices regressions in refresh-app-awareness, please let me know11:41
zygaChipaca: hello sir, would you mind attempting to review https://github.com/snapcore/snapd/pull/6690 today?11:43
mupPR #6690: overlord/snapstate: inhibit refresh for up to a week <Created by zyga> <https://github.com/snapcore/snapd/pull/6690>11:43
Chipacazyga: I would not11:43
Chipacazyga: but, note today I am not doing too good11:44
zygaI wish you to get better soon, I understand11:44
pedronispstolowski: I did pass a on #666211:50
mupPR #6662: overlord/snapstate,snapshotstate: create snapshot on snap removal (3/4) <Created by stolowski> <https://github.com/snapcore/snapd/pull/6662>11:50
pedroniszyga: hi, have you seen the incoming comments on the new tmp logic ?11:50
pstolowskipedronis: ty, also for the remarks on 4/4!11:51
pedronispstolowski: np11:53
zygapedronis: hey, perhaps not, where are they?11:54
pedroniszyga: https://github.com/snapcore/snapd/pull/6614#discussion_r27297400811:55
mupPR #6614: cmd/snap-confine: use fixed private tmp directory <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6614>11:55
zygaah, interesting, I didn't notice that11:56
zygathank you11:56
pedroniszyga: I will look at 6690 today11:56
zygathank you!11:57
zygaI will check if there are any other comments on closed branches11:57
zygamy github.com/notifications is rather long11:57
=== chihchun is now known as chihchun_afk
mupPR snapd#6699 opened: daemon: add RootOnly flag to commands <Simple 😃> <Created by chipaca> <https://github.com/snapcore/snapd/pull/6699>12:05
zygapstolowski: reviewed https://github.com/snapcore/snapd/pull/6698#pullrequestreview-22379906812:05
mupPR #6698: overlord/ifacestate: introduce HotplugKey type use short key in change summaries <Created by stolowski> <https://github.com/snapcore/snapd/pull/6698>12:05
zygaChipaca: is https://github.com/snapcore/snapd/pull/6699/files driven by the suse audit?12:07
mupPR #6699: daemon: add RootOnly flag to commands <Simple 😃> <Created by chipaca> <https://github.com/snapcore/snapd/pull/6699>12:07
Chipacazyga: no, it's one of the changes to users raised last week12:07
Chipacathere's another two coming12:07
Chipaca(one of which needs agreement before i do it)12:07
zygaso it's unrelated to the security review we received?12:07
Chipacamaybe even three12:07
Chipacazyga: it changes nothing security-wise beyond perhaps making the logic easier to follow12:08
zygaunderstood, thank you12:08
Chipaca(which is a win)12:08
Chipacazyga: one of the things coming is to not have the user endpoints when not needed/supported, which might impact the sec review12:08
zygathank you12:09
Chipacazyga: another change is moving the user stuff to api_user12:09
Chipacazyga: a further change that i'd like to do but it's arguably v3 material is moving all the user actions to /v2/users12:09
zygaChipaca: I reviewed the auth code12:12
=== ricab is now known as ricab|lunch
* zyga -> small break12:28
pedronisChipaca: hi, what's the status of #6653 ?12:33
mupPR #6653: tests: try to be a little bit invariant <Created by chipaca> <https://github.com/snapcore/snapd/pull/6653>12:33
Chipacapedronis: need to fix it12:34
* zyga reads tests/main/security-private-tmp and is puzzled12:43
zygalooking at it now12:43
=== chrisccoulson_ is now known as chrisccoulson
=== chihchun_afk is now known as chihchun
pedronisChipaca: standup?13:07
Chipacapedronis: ye13:07
zygaah, I think I understand what the problem is13:19
zygawith refresh awareness thing13:19
=== ddstreet_away is now known as ddstreet
zygare, partially14:24
zygamborzecki: I'm fixing that bug you ran into in the morning15:18
zygajust still in partial capacity away from the office15:18
Chipacawhat is /usr/lib/sysimage/rpm/Basenames and why do i hate it15:19
zygano idea, which system is that on?15:19
Chipacazyga: opensuse-15.0-6415:44
pstolowskizyga: have you seen my general meta-question re #6643?16:14
mupPR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643>16:14
Chipacahrmph, it looks like core18 spread tests are rather messy wrt leaving core behind17:02
=== pstolowski is now known as pstolowski|afk
ChipacaEOD for me17:22
zygapstolowski|afk: not yet, looking now17:22
zygapstolowski|afk: responded17:24
pstolowski|afkzyga: ty17:24
* zyga resumes work17:25
zygasorry, took a nap after dinner17:25
zygacore18 prepare hanging ... ? https://www.irccloud.com/pastebin/MyVsgbcc/17:43
zygabut the system has seeded17:47
zygachanges from the system https://www.irccloud.com/pastebin/loXhyaB3/17:47
zygaI think I fixed that  now17:58
zygaanother round of testing...17:58
zygaalmost there18:59
om26eris there a way to tell multipass through snapcraft to use more cores when packing a snap ? currently it default to 2, I would like it to use all available threads19:12
om26erzyga Hey! Were you able to find a fix for https://bugs.launchpad.net/snapd/+bug/1819318 ?19:23
mupBug #1819318: no interfaces when installing only core18 <snapd:In Progress by zyga> <https://launchpad.net/bugs/1819318>19:23
om26erSo we have a server software as a snap and it won't build on 16.04 because a package it depends on, requires a version of python that is not in 16.0419:24
om26erIf we go with base: core18 then the above issue is problematic as Ubuntu Server on many VPSs doesn't come with core preinstalled19:25
zygaand ... got it19:28
zygaom26er: yes, but it's not near yet19:28
zygaom26er: we need to install core in such cases19:28
zygaI spent some time researching this but it's the closes solution for now19:28
* zyga just understood a pair of bugs19:29
om26erthat's kind of the workaround I have already been using, even before discovering ogra's bug ;-)19:30
om26erwill see if we can update our docs...19:30
zygait will get properly fixed by snapd.snap19:30
zygabut for now...19:30
zyganothing else19:30
om26erre: the python plugin in snapcraft, is there a way to set the `source` to a specific branch ?19:56
roadmr /o\20:22
mupPR snapcraft#2524 closed: Snapcraft try <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2524>20:35
mupPR snapcraft#2519 closed: packaging: use local patchelf <Created by cmatsuoka> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2519>20:38
mupPR snapcraft#2525 opened: catkin plugin: check stage-snaps for ROS dependencies <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2525>20:38
mupPR # closed: core-build#11, core-build#22, core-build#26, core-build#3720:39
mupPR # opened: core-build#11, core-build#22, core-build#26, core-build#3720:40
mupPR snapcraft#2526 opened: Release changelog for 3.4 <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2526>20:44

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!