=== chihchun is now known as chihchun_afk [05:17] morning [05:39] bisecting go in the morning [05:48] mborzecki: hey [05:48] I found a bag of bugs around that flaky test [05:48] zyga: hey [05:56] One core18 seeding bug [05:57] One raciness bug in the test [05:57] And a bug around my use of MATCH [05:57] it is a miracle it passed at all [06:12] mborzecki: anything interessting from the bisect so far? [06:12] zyga: uh, bugs, bugs, bugs [06:12] mvo: not yet [06:12] mborzecki: thank you [06:17] mvo: hello [06:17] Just making coffe [06:17] Yes, bugs in test code [06:17] The one about core18 is generic [06:18] The rest are just test specific [06:18] I will be preparing patches. Just making coffee [06:18] zyga: thanks! [06:18] I spent all evening iterating on those, such a time sink to understand a trivial bug [06:26] re [06:31] mborzecki: quick question - you mentioned that using qemu -m 256 (or even 128) and buildmode=pie does not trigger the bug? i.e. its only observable on a low-mem arm system? [06:32] PR snapd#6700 opened: packaging: disable -buildmode=pie to fix memory issue on armhf <⚠ Critical> [06:32] mvo: 256 worked with qemu [06:33] and pie ofc [06:33] mborzecki: and strace showed now abnormal allocations and the auxv no anomal start address? [06:33] mvo: oh, and couldn't get pi to run with mem=256M, even when tweaking gpu_mem [06:33] mborzecki: thanks! [06:34] mborzecki: this is slightly sad as we will not be able to test if we can ever re-enable buildmode=pie :/ [06:34] mvo: didn't look at auxv, but brk was in the lower range [06:34] mborzecki: yeah, thats fine then [06:34] mborzecki: brk I think is the key [06:35] mborzecki: thanks again, I wonder if we should ask the guys if they can loan one of these boards to us or what we can do [06:35] mvo: do you know if it's a regular chip board? [06:35] mborzecki: I don't know much, I think its a cheap/special one and the manufactor is no longer in business [06:36] mborzecki: I can try to find the datasheet and mail it to you [06:36] mvo: i have 2 chip boards, though regular ones, not -pro, at least one of those works [06:37] mborzecki: oh, nice. so you say we could use those for testing? [06:37] mvo: yes, iff it's the same board and i get the image [06:38] mborzecki: nice! [06:38] mvo: I added a comment to the PR [06:38] please check that if you have the setup [06:38] I'm trying to wrap up findings from last night [06:39] mborzecki: there is a datasheet in the "Subject: Re: core 2.38 memory issue [06:39] " mail [06:39] mborzecki: you are on CC :) thats all the data I have [06:39] zyga: is it? I build snap/snapd without buildmode=pie and "file" showed me a buildid that looks valid [06:41] without buildmode= what is the default mode [06:41] I think it differs across go distributions [06:41] I certainly noticed this on suse [06:41] zyga: what go version is suse using? [06:41] I think it's the patches/config more than the version [06:42] suse is fully up to date, as usual [06:42] but please just check what you get [06:42] and perhaps add a spread test for build-id [06:42] zyga: yeah, adding a spread test seems to be the right thing to move forward here with confidence [06:42] mvo: mborzecki: do we know if the bug exist on 1.11 ? (I was looking at the relevant code and it's quite different in 1.11) [06:42] mvo: +1 [06:42] pedronis: no 1.11 seems ok [06:42] pedronis: I think mborzecki tested on 1.11 - I let him speak [06:43] mvo: pedronis: i'm bisecting between 1.10beta2 (bad) and 1.11beta3 (good) [06:44] mborzecki: nice [06:49] ok, as I said, my impression is that they rewrote the relevant code in between [06:49] arena_used doesn't exist at some point in 1.11 [06:54] pedronis: yeah, it might be that they rewrote chunks of code and fixed it (or made it not be triggered) by accident [06:59] mvo: they use chip PRO, slightly different than my board, idk maybe their image would work with some tweaks to dt [07:03] mborzecki: nice [07:04] mvo: heh, this reminds me of the -buildmode=shared that was triggerd by someone with yocto on i386 [07:06] PR snapd#6687 closed: snap-confine: set rootfs_dir in sc_invocation struct [07:12] PR snapcraft#2527 opened: Add -h short option for help [07:13] morning [07:17] mvo: first commit that fixes our issue [07:17] mvo: https://github.com/golang/go/commit/c0392d2e7fbdcd38aafb959e94daf6bbafe2e4e9 [07:19] mborzecki: aha! thank you [07:19] re after breakfast [07:19] mborzecki: that makes sense [07:20] mvo: we've looked at a commit just 3-4 revisions later [07:20] mvo: could we build with go 1.11? [07:20] zyga: we don't have that unfortunately [07:21] mvo: we've looked at this one https://github.com/golang/go/commit/2b415549b813ba36caafa34fc34d72e47ee8335c and it was mentioned in the issues that seemed related [07:21] mvo: don't have it where? in the repo? [07:21] zyga: correct, rmadison golang tells me we are on 1.10 [07:21] zyga: even in disco [07:21] understood [07:22] zyga: aha, sorry - we have 1.11 in disco [07:22] zyga: as non-default but only disco, I doubt we will get a lot of support when we try to get this backported into xenial,bionic and trusty :/ [07:24] mvo: iirc the official policy is use the latest and n-1 releases, otherwise you're on your own [07:24] yes [07:24] so we need to discuss [07:25] * mvo nods [07:25] we'd probably need to start with having 1.11/12 packaged [07:30] * pstolowski test 123 === pstolowski is now known as pstolowski_ === pstolowski_ is now known as pstolowski [07:31] guess it works now.. morning everyone [07:35] pstolowski: welcome back! [08:01] zyga: the failure on #6643 is unrelated to the PR right`? [08:01] PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits [08:02] pstolowski: yes [08:02] I debugged it and the fixes are coming up soon [08:02] k [08:02] whhee [08:02] it passed :) [08:02] * zyga runs it a few more times [08:06] PR snapd#6701 opened: [ONLY FOR TESTS] Refresh awareness test debug <⛔ Blocked> [08:07] zyga: do you have beaglebone black by any chance? [08:07] yes [08:07] two in fact [08:08] ha! [08:08] zyga: any chance you could load 16.04 armhf image on it and hook it up to the network? [08:08] sure, give me the URL and 15 minutes please [08:09] so now we just need to find a suitable image [08:11] do you want core or classic? [08:11] classic is more convenient for this test, I suppose [08:13] zyga: classic [08:13] hm elinux used to have the images, but i see none now [08:19] mborzecki: let me look [08:20] https://elinux.org/BeagleBoardUbuntu [08:20] let me try one of those [08:20] I'll report back soon [08:23] zyga: hm can't even locate 16.04 armhf rootfs :/ [08:24] zyga: is there a core 16 image maybe? i don't see one either [08:25] h [08:25] back in the office [08:25] let me look at what 's on my board now [08:27] mvo: https://paste.ubuntu.com/p/SsqgjbFqfy/ how does this look like? [08:35] mborzecki: powering up [08:35] and nothing yet, obviously [08:35] I will look at flashing an image then [08:39] mborzecki: wait [08:39] it booted :D [08:39] woot [08:39] core16 [08:40] well, that's good enough i think [08:41] let me try to log into it [08:44] Chipaca: hey, wdyt about Samuele's remark re warnings in https://github.com/snapcore/snapd/pull/6662 ? sounds reasonable to me [08:44] PR #6662: overlord/snapstate,snapshotstate: create snapshot on snap removal (3/4) [08:44] pstolowski: sounds like a separate pr to me [08:44] mborzecki: I have the IP address but I cannot get access yet, [08:44] maybe it refreshed to 2.38 :D [08:44] pstolowski: because it's not the only instance of 'this failed but nothing to do except alert the user' in snapshotstate [08:44] Chipaca: that's fine; isn't it as simple as one API call? [08:45] Chipaca: ah i see your point [08:45] mborzecki: when I ssh in I get logged out instantly [08:45] hmmm [08:46] pstolowski: but yes, it's st.Warnf(), probably [08:46] I've attachec a keyboard [08:46] and ... no luck? [08:47] connection closed again [08:47] I rebooted [08:47] odd [08:49] zyga: maybe it's updating? [08:49] maybe [08:49] zyga: do you have console attached? [08:49] I will leave it for 15 minutes [08:49] mborzecki: console? [08:49] serial? [08:49] no [08:49] zyga: yeah [08:49] I have a display [08:49] but keyboard doesn't interact with it [08:50] zyga: not once have i used display with bbb :D [08:50] zyga: if it's an old core iamge, it might be getting updated now [08:51] btw. figured out why the addresses are so high for the the PIE binaries on that kernel [08:51] mborzecki: woot [08:51] * zyga is really impressed by the debugging on this issue [08:51] fantastic work guys! [08:52] https://paste.ubuntu.com/p/c8MQWPy7fn/ [08:52] it's actually kernel config, need to check what's used elsewhere [08:53] oh, and the address changed over kernel versions, 5.x seems to use 0x40000000 [08:53] as in fixed [08:57] mborzecki: nice job! so if CONFIG_PAGE_OFFSET = 0xc0000000 was lower things must also work? [08:58] mvo: 'may' :P [08:58] mborzecki: yeah, not asking for this, just curious [08:58] mborzecki: is ELF_ET_DYN_BASE = 0x7f555555 derived from this config somehow? [08:58] mvo: yes, see the paste [08:59] mborzecki: aha, nice, yeah, now I see it [08:59] mborzecki: great job [08:59] mborzecki: that was pretty hard to debug, thanks for all the effort here [09:10] on core, what's the easiest way of knowing if i'm core18 or core16? [09:10] also, on core18, can i remove core? [09:10] * Chipaca is about to try it but thought he'd ask first [09:11] zyga: it looks like build-id is available everywhere (or the system-key spread test is broken) [09:11] mvo: +1 [09:11] zyga: would you mind double checking ? just "go build github.com/snapcore/snapd/cmd/snap -o /tmp/lala" file /tmp/lala [09:11] zyga: to see if there is a build-id? [09:11] zyga: on suse [09:11] checking now [09:11] zyga: again, just as an extra pre-caution [09:12] zyga: thank you! [09:12] zyga@yantra:/tmp> go build -o lala github.com/snapcore/snapd/cmd/snap; file lala [09:12] lala: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=43261688d0e2e49bede15de8619def1bbea5e0fa, for GNU/Linux 3.2.0, not stripped [09:12] looks ok [09:13] perhaps that aspect of go has changed in suse since [09:13] +1 from me [09:19] zyga: \o/ [09:19] zyga: yeah, I think it did [09:19] zyga: (but a quick google search did not bring up anything interessting :/ [09:20] mborzecki: one more try, if it fails I will reflash [09:42] mborzecki: fixed auth [09:42] mborzecki: I can now log in [09:44] mborzecki: I'm refreshing core to stable [09:44] it was running a local build [09:45] mborzecki: I will install some more tools [09:46] mborzecki: do you still need access? [09:47] zyga: yes, that'd be great [09:47] k [09:47] mborzecki: how do we add an user to an existing system? [09:47] oh [09:47] haha [09:48] snapd api? [09:49] perhaps [09:49] it's still installing a few things [09:49] mborzecki: what kind of username do you prefer on my gateway? [09:50] zyga: mborzecki? [09:50] k [10:03] mborzecki: ping === pstolowski is now known as pstolowski|lunch [11:09] https://github.com/snapcore/snapd/pull/6702 <- refresh-app-awareness fixes [11:09] PR #6702: tests: fixes discovered debugging refresh-app-awareness [11:09] PR snapd#6702 opened: tests: fixes discovered debugging refresh-app-awareness [11:10] mborzecki, mvo : ^ [11:10] please look, it's affecting master [11:11] zyga: thanks [11:11] if you feel like it I will gladly split this into core vs test fixes [11:12] ogra: hi, any clue where i can find the source code of linux-generic-bbb 4.4.0-93-1 rev 21 kernel? [11:12] mborzecki, apt-get source linux (in xenial) [11:13] mborzecki, it is only re-packing the binary deb [11:13] ogra: ah, excellent then [11:13] (might be a bit behind in versions though) [11:14] mborzecki, for core18 images, you can use the pc-kernel snap, in core18 we have an officially supported armhf build from the kernel team, so there is no need anymore for my hackish bbb kernel [11:15] http://people.canonical.com/~ogra/snappy/pocketbeagle/ uses it ... [11:20] PR snapd#6701 closed: [ONLY FOR TESTS] Refresh awareness test debug <⛔ Blocked> [11:24] Chipaca: hey [11:24] Chipaca: shall I merge https://github.com/snapcore/snapd/pull/6621 now? [11:24] PR #6621: overlord/snapstate: track time of postponed refreshes <⛔ Blocked> [11:25] zyga: sure [11:25] cool, thank you [11:26] PR snapd#6621 closed: overlord/snapstate: track time of postponed refreshes [11:26] pedronis: hey, I noticed you were looking at https://github.com/snapcore/snapd/pull/6690 [11:26] do you have any idea about time.Time comparison? [11:26] PR #6690: overlord/snapstate: inhibit refresh for up to a week [11:26] yes, I'll comment there in a bit [11:27] (not the most important thing overall but I got curious) [11:27] cool, I'm eager to see what you found out [11:27] it depends on TZ that is different in tests and on local machines, I didn't dig deeper though [11:37] just add a snap that sets the timezone (iz trivial) :) [11:44] zyga: commented there [11:44] checking === pstolowski|lunch is now known as pstolowski [11:50] re [12:03] do we run binaries using an explicit elf interpreter at some point? [12:04] I'm having a problem with binaries with patched interpreters, but only when running with an explicit interpreter, and only on s390x :\ [12:26] mvo, pedronis zyga pstolowski mborzecki Chipaca cmatsuoka hey, I am having troubles with freenode, please if you need to contact me today please find me in the internal snappy channel [12:27] ack [12:32] pedronis: all the tests pass \o/ [12:32] pedronis: they really shouldn't /o\ [12:32] * Chipaca is having fun [12:48] jdstrand / ogra: I'm using the timezone-control interface on Core 16 (Raspi 3). No denials and the timezone shows up in timedatectl briefly, but afterwards timedatectl as well as date revert back to UTC. If I set the TZ with sudo timedatectl via SSH, it sticks. Am I doing something wrong, or is https://bugs.launchpad.net/snappy/+bug/1733881 affecting this? [12:48] Bug #1733881: timezone set to n/a on ubuntu core [12:49] ^ addendum: My app calls systemd-timezoned via DBus (tried with Ruby gem ruby-dbus as well as a test shell script which invokes dbus-send) [12:53] * jdstrand_ defers to ogra (I would expect the dbus interface to work) === jdstrand_ is now known as jdstrand [13:10] dot-tobias, it only gets reset on reboot right ? [13:19] ogra: Just rebooted to confirm, I think I'm hitting two issues here: 1) timedatectl does not retain timezone information across reboots, but date +"%Z %z" still has proper output 2) when I set the timezone via DBus from a snap (as opposed to sudo timedatectl via SSH), /etc/writable/localtime is not written and not symlinked to /etc/localtime, thus both date and timedatectl report UTC [13:20] dot-tobias, i have a hack to work around this ... gimme a bit [13:21] ogra: Great, thanks! === sparkieg` is now known as sparkiegeek [13:30] dot-tobias, this should work (untested) https://github.com/ogra1/timezone-snap [13:30] just install it in your image and make sure timezone-control is connected for it [13:33] ogra: Thank you! Will try out and let you know. FWIW: Is this also required on Core 18? Otherwise I could migrate my snap to core18 [13:34] i think core18 will have the same issue ... there was another bug in core18 (cant set any of hostname, timezone or locale) that i'm not sure is completely fixed yet [13:44] PR core#38 closed: Add another pi-config option [13:44] PR core#83 closed: move most of the ubuntu-core config deb into the snap snap build [13:44] PR core#104 closed: snapcraft.yaml: use remote fc-cache-builder [13:45] PR core#38 opened: Add another pi-config option [13:45] PR core#83 opened: move most of the ubuntu-core config deb into the snap snap build [13:45] PR core#104 opened: snapcraft.yaml: use remote fc-cache-builder [13:52] https://github.com/snapcore/snapd/pull/6702 <= please review to fix master [13:52] PR #6702: tests: fixes discovered debugging refresh-app-awareness <⚠ Critical> [14:13] * zyga breaks for lunch [15:04] PR snapcraft#2528 opened: packaging: use our patchelf branch [15:20] Chipaca: have you seen this: https://bugs.launchpad.net/snapd/+bug/1823768 ? [15:21] Bug #1823768: snap refresh failure during task: Consider re-refresh of snap: snap has "refresh-snap" change in progress [15:21] pedronis: i had not [15:22] Chipaca: it seems we are detecting a self-conflict with the change, or we are missing something [15:23] yep [15:23] Chipaca: can I assing to you, to look at it when you have some time? [15:27] pedronis: yep [15:27] thx [15:51] Chipaca: could you do a 2nd review on https://github.com/snapcore/snapd/pull/6702 please [15:51] PR #6702: tests: fixes discovered debugging refresh-app-awareness <⚠ Critical> [15:51] Bug #1823988 opened: CAP_NET_ADMIN not being provided with the recommended plugs [15:52] zyga: +1 [15:53] Chipaca: thank you! [15:53] PR snapd#6702 closed: tests: fixes discovered debugging refresh-app-awareness <⚠ Critical> [15:56] PR snapcraft#2529 opened: build providers: idempotent destroy for LXD [15:57] PR snapd#6703 opened: tests: add deferred actions [16:06] jdstrand, there's a couple of chromium snap revisions awaiting manual approval, if you don't mind approving those [16:06] oSoMoN: yep, I saw earlier. I'll get to them in a bit [16:07] thanks [16:09] Chipaca: updated https://github.com/snapcore/snapd/pull/6690 based on pedronis review, could you have one last look before merging [16:09] PR #6690: overlord/snapstate: inhibit refresh for up to a week [16:09] jdstrand: hey, how are you doing :) [16:09] no [16:09] * Chipaca closes his eyes and goes LA LA LA LA [16:10] zyga: it's a-ok [16:11] zyga: I'm good. you? lots of meetings today. plan on picking up more PR reviews toady [16:11] jdstrand: good, feel happy about making progress lately :) [16:11] jdstrand: also endless debugging of shell :) [16:11] jdstrand: also some improvement towards better shell tests [16:11] jdstrand: good luck with the meetings, don't get lost in them [16:11] nice :) [16:12] they're done now. got some takeaways to get to right away, but hopefully can finish the PR reviews then pickup daemon user tomorrow [16:14] jdstrand: question, do you want to review https://github.com/snapcore/snapd/pull/6643 [16:14] PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits [16:14] it's the regression test and helper program for the ioctl bug [16:18] zyga: nah. I looked at it before. I only care about the result cause it is just build code and we have spread tests that will detect if we generate it wrong [16:21] jdstrand: +1 [16:22] PR snapd#6704 opened: overlord/devicestate,snapstate: measurements around ensure and related tasks === pstolowski is now known as pstolowski|afk [16:25] I think I should break now [16:25] jdstrand: one last word today: I noticed your branch and I will get to it soon, some of the changes there collide with upcoming changes to group handling but I will deconflict those as conflicts arise in, either branch [16:26] zyga: ack [16:26] I can deconflict too, but if you're keen to do it, that works [17:05] PR snapcraft#2528 closed: packaging: use our patchelf branch [17:05] zyga: getopt, not getopts [17:05] zyga: util-linux, not bash [17:06] oh wait your comment was about local [17:06] hm [17:06] Chipaca: thank you for the review! [17:07] zyga: /bin/sh also has 'local' [17:07] Chipaca: I replied, happy to iterate on the points raised [17:07] Chipaca: /bin/sh may have it but it is not POSIX and shellcheck complains [17:07] Chipaca: but I will adjust the shebang and use local happily [17:07] ah ok [17:08] Chipaca: I'm very curious about the RETURN idea, can you tell me more please [17:08] Chipaca: ideally scope would be gone [17:08] Chipaca: and defer would do the right thing by sourcing the file [17:08] Chipaca: if it wasn't for prepare/restore it would be totally useless [17:16] zyga: well, the execute thing is run inside a function, right? [17:17] Chipaca: perhaps... I don't know [17:17] I think they are all concatenated internally [17:17] we could look at spread source code or at spread -vv output [17:17] hm [17:18] something for tomorrow, if it's me doing it [17:18] I'm gonna EOD [17:18] zyga: you should too :-) [17:18]