=== chihchun is now known as chihchun_afk
mborzeckibisecting go in the morning05:39
zygamborzecki: hey05:48
zygaI found a bag of bugs around that flaky test05:48
mborzeckizyga: hey05:48
zygaOne core18 seeding bug05:56
zygaOne raciness bug in the test05:57
zygaAnd a bug around my use of MATCH05:57
zygait is a miracle it passed at all05:57
mvomborzecki: anything interessting from the bisect so far?06:12
mvozyga: uh, bugs, bugs, bugs06:12
mborzeckimvo: not yet06:12
mvomborzecki: thank you06:12
zygamvo: hello06:17
zygaJust making coffe06:17
zygaYes, bugs in test code06:17
zygaThe one about core18 is generic06:17
zygaThe rest are just test specific06:18
zygaI will be preparing patches. Just making coffee06:18
mvozyga: thanks!06:18
zygaI spent all evening iterating on those, such a time sink to understand a trivial bug06:18
mvomborzecki: quick question - you mentioned that using qemu -m 256 (or even 128) and buildmode=pie does not trigger the bug? i.e. its only observable on a low-mem arm system?06:31
mupPR snapd#6700 opened: packaging: disable -buildmode=pie to fix memory issue on armhf <⚠ Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/6700>06:32
mborzeckimvo: 256 worked with qemu06:32
mborzeckiand pie ofc06:33
mvomborzecki: and strace showed now abnormal allocations and the auxv no anomal start address?06:33
mborzeckimvo: oh, and couldn't get pi to run with mem=256M, even when tweaking gpu_mem06:33
mvomborzecki: thanks!06:33
mvomborzecki: this is slightly sad as we will not be able to test if we can ever re-enable buildmode=pie :/06:34
mborzeckimvo: didn't look at auxv, but brk was in the lower range06:34
mvomborzecki: yeah, thats fine then06:34
mvomborzecki: brk I think is the key06:34
mvomborzecki: thanks again, I wonder if we should ask the guys if they can loan one of these boards to us or what we can do06:35
mborzeckimvo:  do you know if it's a regular chip board?06:35
mvomborzecki: I don't know much, I think its a cheap/special one and the manufactor is no longer in business06:35
mvomborzecki: I can try to find the datasheet and mail it to you06:36
mborzeckimvo: i have 2 chip boards, though regular ones, not -pro, at least one of those works06:36
mvomborzecki: oh, nice. so you say we could use those for testing?06:37
mborzeckimvo: yes, iff it's the same board and i get the image06:37
mvomborzecki: nice!06:38
zygamvo: I added a comment to the PR06:38
zygaplease check that if you have the setup06:38
zygaI'm trying to wrap up findings from last night06:38
mvomborzecki: there is a datasheet in the "Subject: Re: core 2.38 memory issue06:39
mvo" mail06:39
mvomborzecki: you are on CC :) thats all the data I have06:39
mvozyga: is it? I build snap/snapd without buildmode=pie and "file" showed me a buildid that looks valid06:39
zygawithout buildmode= what is the default mode06:41
zygaI think it differs  across go distributions06:41
zygaI certainly noticed this on suse06:41
mvozyga: what go version is suse using?06:41
zygaI think it's the patches/config more than the version06:41
zygasuse is fully up to date, as usual06:42
zygabut please  just check what you get06:42
zygaand perhaps add a spread test for build-id06:42
mvozyga: yeah, adding a spread test seems to be the right thing to move forward here with confidence06:42
pedronismvo: mborzecki: do we know if the bug exist on 1.11 ? (I was looking at the relevant code and it's quite different in 1.11)06:42
zygamvo: +106:42
mborzeckipedronis: no 1.11 seems ok06:42
mvopedronis: I think mborzecki tested on 1.11 - I let him speak06:42
mborzeckimvo: pedronis: i'm bisecting between 1.10beta2 (bad) and 1.11beta3 (good)06:43
mvomborzecki: nice06:44
pedronisok, as I said, my impression is that they rewrote the relevant code in between06:49
pedronisarena_used doesn't exist at some point in 1.1106:49
mborzeckipedronis: yeah, it might be that they rewrote chunks of code and fixed it (or made it not be triggered) by accident06:54
mborzeckimvo: they use chip PRO, slightly different than my board, idk maybe their image would work with some tweaks to dt06:59
mvomborzecki: nice07:03
mborzeckimvo: heh, this reminds me of the -buildmode=shared that was triggerd by someone with yocto on i38607:04
mupPR snapd#6687 closed: snap-confine: set rootfs_dir in sc_invocation struct <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6687>07:06
mupPR snapcraft#2527 opened: Add -h short option for help <Created by techtonik> <https://github.com/snapcore/snapcraft/pull/2527>07:12
mborzeckimvo: first commit that fixes our issue07:17
mborzeckimvo: https://github.com/golang/go/commit/c0392d2e7fbdcd38aafb959e94daf6bbafe2e4e907:17
mvomborzecki: aha! thank you07:19
zygare after breakfast07:19
mvomborzecki: that makes sense07:19
mborzeckimvo: we've looked at a commit just 3-4 revisions later07:20
zygamvo: could we build with go 1.11?07:20
mvozyga: we don't have that unfortunately07:20
mborzeckimvo: we've looked at this one https://github.com/golang/go/commit/2b415549b813ba36caafa34fc34d72e47ee8335c and it was mentioned in the issues that seemed related07:21
zygamvo: don't have it where? in the repo?07:21
mvozyga: correct, rmadison golang tells me we are on 1.1007:21
mvozyga: even in disco07:21
mvozyga: aha, sorry - we have 1.11 in disco07:22
mvozyga: as non-default but only disco, I doubt we will get a lot of support when we try to get this backported into xenial,bionic and trusty :/07:22
mborzeckimvo: iirc the official policy is use the latest and n-1 releases, otherwise you're on your own07:24
pedronisso we need to discuss07:24
* mvo nods07:25
mborzeckiwe'd probably need to start with having 1.11/12 packaged07:25
* pstolowski test 12307:30
=== pstolowski is now known as pstolowski_
=== pstolowski_ is now known as pstolowski
pstolowskiguess it works now.. morning everyone07:31
mvopstolowski: welcome back!07:35
pstolowskizyga: the failure on #6643 is unrelated to the PR right`?08:01
mupPR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643>08:01
zygapstolowski: yes08:02
zygaI debugged it and the fixes are coming up soon08:02
zygait passed :)08:02
* zyga runs it a few more times08:02
mupPR snapd#6701 opened: [ONLY FOR TESTS] Refresh awareness test debug <⛔ Blocked> <Created by pedronis> <https://github.com/snapcore/snapd/pull/6701>08:06
mborzeckizyga: do you have beaglebone black by any chance?08:07
zygatwo in  fact08:07
mborzeckizyga: any chance you could load 16.04 armhf image on it and hook it up to the network?08:08
zygasure, give me the URL and 15 minutes please08:08
mborzeckiso now we just need to find a suitable image08:09
zygado you want core  or classic?08:11
zygaclassic is more convenient for this test, I suppose08:11
mborzeckizyga: classic08:13
mborzeckihm elinux used to have the images, but i see none now08:13
zygamborzecki: let me look08:19
zygalet me try one  of those08:20
zygaI'll report back soon08:20
mborzeckizyga: hm can't even locate 16.04 armhf rootfs :/08:23
mborzeckizyga: is there a core 16 image maybe? i don't see one either08:24
zygaback in the office08:25
zygalet me look at what 's on my board now08:25
zygamvo: https://paste.ubuntu.com/p/SsqgjbFqfy/ how does this look like?08:27
zygamborzecki: powering up08:35
zygaand nothing yet, obviously08:35
zygaI will look at flashing an image then08:35
zygamborzecki: wait08:39
zygait booted :D08:39
mborzeckiwell, that's good enough i think08:40
zygalet me try to log into it08:41
pstolowskiChipaca: hey, wdyt about Samuele's remark re warnings in https://github.com/snapcore/snapd/pull/6662 ? sounds reasonable to me08:44
mupPR #6662: overlord/snapstate,snapshotstate: create snapshot on snap removal (3/4) <Created by stolowski> <https://github.com/snapcore/snapd/pull/6662>08:44
Chipacapstolowski: sounds like a separate pr to me08:44
zygamborzecki: I have the IP address but I cannot get access yet,08:44
zygamaybe it refreshed to 2.38 :D08:44
Chipacapstolowski: because it's not the only instance of 'this failed but nothing to do except alert the user' in snapshotstate08:44
pstolowskiChipaca: that's fine; isn't it as simple as one API call?08:44
pstolowskiChipaca: ah i see your point08:45
zygamborzecki: when I ssh in I get logged out instantly08:45
Chipacapstolowski: but yes, it's st.Warnf(), probably08:46
zygaI've attachec a keyboard08:46
zygaand ... no luck?08:46
zygaconnection closed again08:47
zygaI rebooted08:47
mborzeckizyga: maybe it's updating?08:49
mborzeckizyga: do you have console attached?08:49
zygaI will leave it for 15 minutes08:49
zygamborzecki: console?08:49
mborzeckizyga: yeah08:49
zygaI have a display08:49
zygabut keyboard doesn't interact with it08:49
mborzeckizyga: not once have i used display with bbb :D08:50
mborzeckizyga: if it's an old core iamge, it might be getting updated now08:50
mborzeckibtw. figured out why the addresses are so high for the the PIE binaries on that kernel08:51
zygamborzecki: woot08:51
* zyga is really impressed by the debugging on this issue08:51
zygafantastic work guys!08:51
mborzeckiit's actually kernel config, need to check what's used elsewhere08:52
mborzeckioh, and the address changed over kernel versions, 5.x seems to use 0x4000000008:53
mborzeckias in fixed08:53
mvomborzecki: nice job! so if CONFIG_PAGE_OFFSET = 0xc0000000 was lower things must also work?08:57
mborzeckimvo: 'may' :P08:58
mvomborzecki: yeah, not asking for this, just curious08:58
mvomborzecki: is ELF_ET_DYN_BASE = 0x7f555555 derived from this config somehow?08:58
mborzeckimvo: yes, see the paste08:58
mvomborzecki: aha, nice, yeah, now I see it08:59
mvomborzecki: great job08:59
mvomborzecki: that was pretty hard to debug, thanks for all the effort here08:59
Chipacaon core, what's the easiest way of knowing if i'm core18 or core16?09:10
Chipacaalso, on core18, can i remove core?09:10
* Chipaca is about to try it but thought he'd ask first09:10
mvozyga: it looks like build-id is available everywhere (or the system-key spread test is broken)09:11
zygamvo: +109:11
mvozyga: would you mind double checking ? just "go build github.com/snapcore/snapd/cmd/snap -o /tmp/lala" file /tmp/lala09:11
mvozyga: to see if there is a build-id?09:11
mvozyga: on suse09:11
zygachecking now09:11
mvozyga: again, just as an extra pre-caution09:11
mvozyga: thank you!09:12
zygazyga@yantra:/tmp> go build -o lala github.com/snapcore/snapd/cmd/snap; file lala09:12
zygalala: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=43261688d0e2e49bede15de8619def1bbea5e0fa, for GNU/Linux 3.2.0, not stripped09:12
zygalooks ok09:12
zygaperhaps that aspect of go has changed in suse since09:13
zyga+1 from me09:13
mvozyga: \o/09:19
mvozyga: yeah, I think it did09:19
mvozyga: (but a quick google search did not bring up anything interessting :/09:19
zygamborzecki: one more try, if it fails I will reflash09:20
zygamborzecki: fixed auth09:42
zygamborzecki: I can now log in09:42
zygamborzecki: I'm refreshing core to stable09:44
zygait was running a local build09:44
zygamborzecki: I will install some more tools09:45
zygamborzecki: do you still need access?09:46
mborzeckizyga: yes, that'd be great09:47
zygamborzecki: how do we add an user to an existing system?09:47
mborzeckisnapd api?09:48
zygait's still installing a few things09:49
zygamborzecki: what kind of username do you prefer on my gateway?09:49
mborzeckizyga: mborzecki?09:50
zygamborzecki: ping10:03
=== pstolowski is now known as pstolowski|lunch
zygahttps://github.com/snapcore/snapd/pull/6702 <- refresh-app-awareness fixes11:09
mupPR #6702: tests: fixes discovered debugging refresh-app-awareness <Created by zyga> <https://github.com/snapcore/snapd/pull/6702>11:09
mupPR snapd#6702 opened: tests: fixes discovered debugging refresh-app-awareness <Created by zyga> <https://github.com/snapcore/snapd/pull/6702>11:09
zygamborzecki, mvo : ^11:10
zygaplease look, it's affecting master11:10
mvozyga: thanks11:11
zygaif you feel like it I will gladly split this into core vs test fixes11:11
mborzeckiogra: hi, any clue where i can find the source code of linux-generic-bbb 4.4.0-93-1 rev 21 kernel?11:12
ogramborzecki, apt-get source linux (in xenial)11:12
ogramborzecki, it is only re-packing the binary deb11:13
mborzeckiogra: ah, excellent then11:13
ogra(might be a bit behind in versions though)11:13
ogramborzecki, for core18 images, you can use the pc-kernel snap, in core18 we have an officially supported armhf build from the kernel team, so there is no need anymore for my hackish bbb kernel11:14
ograhttp://people.canonical.com/~ogra/snappy/pocketbeagle/ uses it ...11:15
mupPR snapd#6701 closed: [ONLY FOR TESTS] Refresh awareness test debug <⛔ Blocked> <Created by pedronis> <Closed by pedronis> <https://github.com/snapcore/snapd/pull/6701>11:20
zygaChipaca: hey11:24
zygaChipaca: shall I merge https://github.com/snapcore/snapd/pull/6621 now?11:24
mupPR #6621: overlord/snapstate: track time of postponed refreshes <⛔ Blocked> <Created by zyga> <https://github.com/snapcore/snapd/pull/6621>11:24
Chipacazyga: sure11:25
zygacool, thank you11:25
mupPR snapd#6621 closed: overlord/snapstate: track time of postponed refreshes <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6621>11:26
zygapedronis: hey, I noticed you were looking at https://github.com/snapcore/snapd/pull/669011:26
zygado you have any idea about time.Time comparison?11:26
mupPR #6690: overlord/snapstate: inhibit refresh for up to a week <Created by zyga> <https://github.com/snapcore/snapd/pull/6690>11:26
pedronisyes, I'll comment there in a bit11:26
pedronis(not the most important thing overall but I got curious)11:27
zygacool, I'm eager to see what you found out11:27
zygait depends on TZ that is different in tests and on local machines, I didn't dig deeper though11:27
ograjust add a snap that sets the timezone (iz trivial) :)11:37
pedroniszyga: commented there11:44
=== pstolowski|lunch is now known as pstolowski
cmatsuokado we run binaries using an explicit elf interpreter at some point?12:03
cmatsuokaI'm having a problem with binaries with patched interpreters, but only when running with an explicit interpreter, and only on s390x :\12:04
cachiomvo, pedronis zyga pstolowski mborzecki Chipaca cmatsuoka hey, I am having troubles with freenode, please if you need to contact me today please find me in the internal snappy channel12:26
Chipacapedronis: all the tests pass \o/12:32
Chipacapedronis: they really shouldn't /o\12:32
* Chipaca is having fun12:32
dot-tobiasjdstrand / ogra: I'm using the timezone-control interface on Core 16 (Raspi 3). No denials and the timezone shows up in timedatectl briefly, but afterwards timedatectl as well as date revert back to UTC. If I set the TZ with sudo timedatectl via SSH, it sticks. Am I doing something wrong, or is https://bugs.launchpad.net/snappy/+bug/1733881 affecting this?12:48
mupBug #1733881: timezone set to n/a on ubuntu core <Snappy:Confirmed> <https://launchpad.net/bugs/1733881>12:48
dot-tobias^ addendum: My app calls systemd-timezoned via DBus (tried with Ruby gem ruby-dbus as well as a test shell script which invokes dbus-send)12:49
* jdstrand_ defers to ogra (I would expect the dbus interface to work)12:53
=== jdstrand_ is now known as jdstrand
ogradot-tobias, it only gets reset on reboot right ?13:10
dot-tobiasogra: Just rebooted to confirm, I think I'm hitting two issues here: 1) timedatectl does not retain timezone information across reboots, but date +"%Z %z" still has proper output 2) when I set the timezone via DBus from a snap (as opposed to sudo timedatectl via SSH), /etc/writable/localtime is not written and not symlinked to /etc/localtime, thus both date and timedatectl report UTC13:19
ogradot-tobias, i have a hack to work around this ... gimme a bit13:20
dot-tobiasogra: Great, thanks!13:21
=== sparkieg` is now known as sparkiegeek
ogradot-tobias, this should work (untested) https://github.com/ogra1/timezone-snap13:30
ograjust install it in your image and make sure timezone-control is connected for it13:30
dot-tobiasogra: Thank you! Will try out and let you know. FWIW: Is this also required on Core 18? Otherwise I could migrate my snap to core1813:33
ograi think core18 will have the same issue ... there was another bug in core18 (cant set any of hostname, timezone or locale) that i'm not sure is completely fixed yet13:34
mupPR core#38 closed: Add another pi-config option <Created by sergey-borovkov> <https://github.com/snapcore/core/pull/38>13:44
mupPR core#83 closed: move most of the ubuntu-core config deb into the snap snap build <Created by mvo5> <https://github.com/snapcore/core/pull/83>13:44
mupPR core#104 closed: snapcraft.yaml: use remote fc-cache-builder <Created by mvo5> <https://github.com/snapcore/core/pull/104>13:44
mupPR core#38 opened: Add another pi-config option <Created by sergey-borovkov> <https://github.com/snapcore/core/pull/38>13:45
mupPR core#83 opened: move most of the ubuntu-core config deb into the snap snap build <Created by mvo5> <https://github.com/snapcore/core/pull/83>13:45
mupPR core#104 opened: snapcraft.yaml: use remote fc-cache-builder <Created by mvo5> <https://github.com/snapcore/core/pull/104>13:45
zygahttps://github.com/snapcore/snapd/pull/6702 <= please review to fix master13:52
mupPR #6702: tests: fixes discovered debugging refresh-app-awareness <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/6702>13:52
* zyga breaks for lunch14:13
mupPR snapcraft#2528 opened: packaging: use our patchelf branch <Created by cmatsuoka> <https://github.com/snapcore/snapcraft/pull/2528>15:04
pedronisChipaca: have you seen this:  https://bugs.launchpad.net/snapd/+bug/1823768 ?15:20
mupBug #1823768: snap refresh failure during task: Consider re-refresh of snap: snap has "refresh-snap" change in progress <snapd:New> <https://launchpad.net/bugs/1823768>15:21
Chipacapedronis: i had not15:21
pedronisChipaca: it seems we are detecting a self-conflict with the change, or we are missing something15:22
pedronisChipaca: can I assing to you, to look at it when you have some time?15:23
Chipacapedronis: yep15:27
zygaChipaca: could you do a 2nd review on https://github.com/snapcore/snapd/pull/6702 please15:51
mupPR #6702: tests: fixes discovered debugging refresh-app-awareness <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/6702>15:51
mupBug #1823988 opened: CAP_NET_ADMIN not being provided with the recommended plugs <Snappy:New> <https://launchpad.net/bugs/1823988>15:51
Chipacazyga: +115:52
zygaChipaca: thank you!15:53
mupPR snapd#6702 closed: tests: fixes discovered debugging refresh-app-awareness <⚠ Critical> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6702>15:53
mupPR snapcraft#2529 opened: build providers: idempotent destroy for LXD <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2529>15:56
mupPR snapd#6703 opened: tests: add deferred actions <Created by zyga> <https://github.com/snapcore/snapd/pull/6703>15:57
oSoMoNjdstrand, there's a couple of chromium snap revisions awaiting manual approval, if you don't mind approving those16:06
jdstrandoSoMoN: yep, I saw earlier. I'll get to them in a bit16:06
zygaChipaca: updated https://github.com/snapcore/snapd/pull/6690 based on pedronis review, could you have one last look before merging16:09
mupPR #6690: overlord/snapstate: inhibit refresh for up to a week <Created by zyga> <https://github.com/snapcore/snapd/pull/6690>16:09
zygajdstrand: hey, how are you doing :)16:09
* Chipaca closes his eyes and goes LA LA LA LA16:09
Chipacazyga: it's a-ok16:10
jdstrandzyga: I'm good. you? lots of meetings today. plan on picking up more PR reviews toady16:11
zygajdstrand: good, feel happy about making progress lately :)16:11
zygajdstrand: also endless debugging of shell :)16:11
zygajdstrand: also some improvement towards better shell tests16:11
zygajdstrand: good luck with the meetings, don't get lost in them16:11
jdstrandnice :)16:11
jdstrandthey're done now. got some takeaways to get to right away, but hopefully can finish the PR reviews then pickup daemon user tomorrow16:12
zygajdstrand: question, do you want to review https://github.com/snapcore/snapd/pull/664316:14
mupPR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643>16:14
zygait's the regression test and helper program for the ioctl bug16:14
jdstrandzyga: nah. I looked at it before. I only care about the result cause it is just build code and we have spread tests that will detect if we generate it wrong16:18
zygajdstrand: +116:21
mupPR snapd#6704 opened: overlord/devicestate,snapstate: measurements around ensure and related tasks <Created by stolowski> <https://github.com/snapcore/snapd/pull/6704>16:22
=== pstolowski is now known as pstolowski|afk
zygaI think I should break now16:25
zygajdstrand: one last word today: I noticed your branch and I will get to it soon, some of the changes there collide with upcoming changes to group handling but I will deconflict those as conflicts arise in, either branch16:25
jdstrandzyga: ack16:26
jdstrandI can deconflict too, but if you're keen to do it, that works16:26
mupPR snapcraft#2528 closed: packaging: use our patchelf branch <Created by cmatsuoka> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2528>17:05
Chipacazyga: getopt, not getopts17:05
Chipacazyga: util-linux, not bash17:05
Chipacaoh wait your comment was about local17:06
zygaChipaca: thank you for the review!17:06
Chipacazyga: /bin/sh also has 'local'17:07
zygaChipaca: I replied, happy to iterate on the points raised17:07
zygaChipaca: /bin/sh may have it but it is not POSIX and shellcheck complains17:07
zygaChipaca: but I will adjust the shebang and use local happily17:07
Chipacaah ok17:07
zygaChipaca: I'm very curious about the RETURN idea, can you tell me more please17:08
zygaChipaca: ideally scope would be gone17:08
zygaChipaca: and defer would do the right thing by sourcing the file17:08
zygaChipaca: if it wasn't for prepare/restore it would be totally useless17:08
Chipacazyga: well, the execute thing is run inside a function, right?17:16
zygaChipaca: perhaps... I don't know17:17
zygaI think they are all concatenated internally17:17
zygawe could look at spread source code or at spread -vv output17:17
Chipacasomething for tomorrow, if it's me doing it17:18
ChipacaI'm gonna EOD17:18
Chipacazyga: you should too :-)17:18
zygaChipaca: gladly17:18
zygaChipaca: thank you for the review!17:18
Chipacanp! looks good17:18
zygaChipaca: defer review :)17:18
mupPR snapcraft#2525 closed: catkin plugin: check stage-snaps for ROS dependencies <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2525>17:26
mupPR snapcraft#2529 closed: build providers: idempotent destroy for LXD <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2529>19:02
mupPR snapd#6705 opened: bootloader: little kernel support <Created by kubiko> <https://github.com/snapcore/snapd/pull/6705>22:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!