=== chihchun_afk is now known as chihchun === JanC is now known as Guest16348 === JanC_ is now known as JanC [06:16] good morning [06:23] tvoss: morning! [06:23] o/ [06:23] @pedronis happy new week :) would be great if you would find some time for https://github.com/snapcore/snapd/pull/3130 [06:23] tvoss: No such command! [06:23] PR snapd#3130: overlord/devicestate: switch to keygen for device key generation [06:51] thresh: ah, good that it's reported. I believe it may require some snapcraft changes to be able to properly bypass the problem, like having conditional packages. [06:51] /usr/lib/snapd/snap-confine: error while loading shared libraries: libudev.so.1: failed to map segment from shared object [06:51] ^^ anyone know what is going on there? [06:52] Inside a xenial lxd container, hosted under xenial [07:02] Its a classic snap, so it should certainly be able to access stuff. And the same snap was working two weeks ago. [07:10] Bug #1681328 opened: classic snap now failing with udev errors [07:11] morning distro [07:16] good morning seb128 [07:16] hey zyga ;-) [07:17] distro->snappy (in fact I was typing on another channel, don't do that when xchat-gnome loads, it moves focus to newly joined channel as it does) [07:23] hey ara [07:23] hey zyga [07:51] zyga: good morning. I'm seeing a test failure for main/static, with the test complaining about CGO being unexpected for overlord/devicestate. where would I need to adjust the build/test setup to change that expectation? [07:55] tvoss: good morning [07:56] tvoss: not sure I understand, can you please post a log? [07:57] mvo: good morning [07:58] zyga: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-image/xenial/amd64/s/snapd/20170410_072000_9284e@/log.gz [07:59] hey zyga [08:00] tvoss: hmmm [08:01] tvoss: I honestly don't know [08:03] zyga: okay, the test checks that snapd can be built without cgo. that's certainly not true anymore :) [08:03] tvoss: ah, I just had a look at the test [08:04] tvoss: yes, feels like something that should be removed now [08:04] zyga: okay, let me remove it and see what the infrastructure thinks [08:05] tvoss: I will look [08:07] pedronis: thx [08:19] Son_Goku, zyga: looks like all packages are published now [08:36] morphis: nice, I had issues with f26 installation on Friday but I will give it a test as well and commit the changes to ship tab completion [08:36] zyga: what kind of issues? [08:36] morphis: with f26 installer, not with snaps [08:36] ah [08:36] morphis: not sure, it would freeze; I rebooted a few times before it finished [08:36] Son_Goku: you want to blog about this today? [08:36] Chipaca: hey, good morning! I had a look at the hashsum issues this morning because quite a few of the current branches fail with it - I suspect we just don't handle an EOF during download not correctly, i.e. we do not set the right resume point. I think somehting like http://paste.ubuntu.com/24353432/ help - what do you think? is that something you invesitgated yet? [08:37] zyga: if I remember well I had problems last week too with F26 [08:37] Chipaca: i.e. I hope I did not duplicate some reaseach :/ [08:37] mvo— heh. I was just starting with this. [08:37] mvo: nice! [08:38] mvo— that does seem correct to me :-) [08:38] Chipaca: it needs a test to be certain but I don't see that we handle the seeking but maybe I'm wrong [08:38] mvo— we do a SEEK_END i think [08:39] Chipaca: yeah, I think the problem is that we don't do the hashsum update, its some lines above. without a test just a theory for now [08:40] Chipaca: whats funny is that all the hash errors are the same, wonder if there is a too-short file on some of the cdn server somewhere [08:40] mvo— oh? [08:42] * zyga eats breakfast, sorry for not coding yet, I was checking email all morning [08:43] Chipaca: e.g. https://github.com/snapcore/snapd/pull/2833#issuecomment-292874676 matches the one in https://forum.snapcraft.io/t/hashsum-failures-during-tests/198 [08:43] PR snapd#2833: many: allow core refresh.schedule setting [08:47] mvo— maybe it's the hash of "503 Go away I'm Comfy" [08:48] Chipaca: haha, possible [08:49] Chipaca: I'm writing a test, lets see how that goes [08:50] Chipaca: no more like Oh not you again, what do you want now? [08:54] mvo, zyga: if I define in my spread.yaml a new system debian-9-64 is spread/linode able to spawn up a machine with the right image just with this automatically or do we need someone adding an image on the Linode setup itself? [08:55] morphis: you will need niemeyer to create this machine for you AFAIK [08:55] mvo: ok [08:58] mvo: I'm not so sure, with our credentials we can spawn any machine [08:58] morphis: just try it [08:59] zyga: nice! I was not aware of that [09:00] zyga: lets see .. [09:01] zyga, mvo: https://github.com/snapcore/snapd/pull/3156 [09:01] PR snapd#3156: WIP: debian support for spread testing [09:02] PR snapd#3156 opened: WIP: debian support for spread testing [09:03] morphis: do you know you can try it locally? [09:03] morphis: with spread linode:... [09:03] zyga: if I would have an API token, yes [09:03] zyga: need to ask niemeyer for one [09:03] morphis: ah, definitely [09:03] qemu sucks for these things [09:04] morphis: s/qemu/network at home/ [09:05] yeah both [09:05] mvo, zyga: 2017-04-10 09:04:12 Allocating linode:debian-9-64... [09:05] looks like its working [09:05] morphis: I don't think qemu is any worse than linode [09:06] morphis: nice [09:06] ah no [09:06] 2017-04-10 09:04:24 Cannot allocate linode:debian-9-64: no Linode image or distribution for "debian-9-64" [09:10] morphis: quick idea: add internal snap command like `is-confined` or `is-forced-devmode` or something like that [09:10] morphis: and use that in spread tests to skip confinement parts [09:10] zyga: yeah I was thinking about the same [09:10] morphis: this way we don't have to maintain the exclusion lists [09:11] morphis: and we can really test a lot of useful functionality at the same time (only parts of each test would be skipped) [09:11] PR snapd#3152 closed: store: make hash error message more accurate [09:11] morphis: I think you got the ID of the system wrong, drop the kernel line and give me a sec, I'll give you the right ID [09:12] zyga: followed what is described on https://github.com/snapcore/spread [09:13] zyga: would you mind restarting the travis run for https://github.com/snapcore/snapd/pull/3130 [09:13] PR snapd#3130: overlord/devicestate: switch to lib(open)ssl for device key generation [09:13] zyga: timedout [09:15] morphis: there's no `debian-9-64`, you can do `debian-8` (sans -64) [09:16] zyga: were we do not support debian-8 [09:16] however worth a try [09:17] morphis: yeah, I'm just telling you that's what is supported on linode now :/ [09:18] yeah, we may have to do our own debian-9 image [09:18] same for fedora 24 and 26 [09:18] morphis: fedora 24 is available AFAIR [09:18] morphis: but yeah, [09:19] https://www.linode.com/distributions doesn't have f24 [09:19] maybe there is an outdated image still and not listed there [09:20] morphis: https://blog.linode.com/2016/06/22/introducing-fedora-24/ [09:20] I think the distributions page is outdated [09:20] morphis: maybe we could reach out to linode and ask [09:20] morphis: e.g. add a debian-9 image, fedora-25 image [09:20] morphis: maybe they will refuse [09:21] morphis: but maybe it's cheap for them and they will just do it [09:21] zyga: just checked, they have on for f24 but listed under "old" [09:22] zyga: you can easily do your own images on linode [09:22] so linode doesn't have to care at all [09:23] morphis: it's easier if someone else cares :) [09:23] morphis: but I think we will need our images anyway [09:23] zyga: it is :-) [09:32] mvo— you wouldn't happen to have the whole journalctl output of one of those hashing failures would you? [09:33] Chipaca: of course [09:34] Chipaca: https://travis-ci.org/snapcore/snapd/builds/219559725#L1320 [09:34] Chipaca: and https://travis-ci.org/snapcore/snapd/builds/219416996#L994 [09:34] cheers [09:35] Chipaca: I struggle a bit with the test, I can get a test working but I need to send enough data that the first batch of data is commited to disk so that we trigger the rety. when I do that I get a strange errTrailerEOF error when I cut the connection in the middle (might be a new go 1.7 thing, I'm on zesty :/) [09:36] mvo— how're you injecting the failure? [09:38] Chipaca: http://paste.ubuntu.com/24353606/ is the test with a small amount of data, the amount is too small to get flushed to the local tempfile so when I check what position we are in our temp file I still get "zero" [09:38] Chipaca: i..e I use a full mock server and close the client connections (trying to be close to the real problem) [09:44] mvo— what's missing there is that you're not setting the ContentLength, you're letting go do that [09:44] Chipaca: I tried that too, but maybe not hard enough - let me do that again [09:52] Chipaca: hm, making progress with the test now (yay) but look like the resume case is actually handled down below, so its not that [09:54] mvo— i think it properly is the cdn returning some error page instead of the content [09:54] but I'm not sure [09:54] mvo— do we check the response status before hashing? [09:54] Chipaca: yeah, actually I think you are right, its probably something like "range-requests are not supported" [09:56] that should be some 4xx though? [09:56] are we really not checking error codes? [09:56] hence my question about whether we check the status before just hashing [09:56] :-) [09:57] tvoss: done [09:57] i'm afraid we've made the download code too smart [09:57] heh [09:58] re [09:58] * zyga is grumpy today, woke up at 5:00 AM [09:59] why I know? [09:59] because modem company resets modems at 5AM [09:59] PR snapd#3157 opened: store: add more logs around retry in download [09:59] and the relay clicking woke me up for some reason [10:11] PR snapd#3158 opened: store: add download test with EOF in the middle [10:11] Chipaca: my theory is wrong, I think our code is fine. unless I miss someting in -^ [10:15] pedronis: thx [10:31] mvo, thank you very much for this test! btw, just in the meantime there was a comment on this bug: https://bugs.launchpad.net/snapd/+bug/1677557 [10:31] Bug #1677557: EOF not properly retried under some circumstances [10:31] mvo, I'm going to look at provoking 'unexpected eof' in the test and retrying it [10:32] mvo: can you have a look at https://github.com/snapcore/snapd/pull/3154 [10:32] PR snapd#3154: many: rename two core plugs that clash with slot names [10:32] pstolowski: wait a sec [10:32] pstolowski: there is a new ranch comming [10:32] branch [10:32] mvo: I'm trying to come up with a fix for the clashing plugs, as you know [10:32] mvo: I was thinking about spitting this branch so that connections are in separate PR [10:32] mvo, sure... not going to look at this today, maybe tommorrow-ish [10:32] mvo: as it may not land in the end [10:33] (specifically this one https://github.com/snapcore/snapd/pull/3154/commits/a69290f0c7836738c41da6f87f859e2971576630 ) [10:33] PR snapd#3154: many: rename two core plugs that clash with slot names [10:39] PR snapd#3159 opened: store: retry once on hashsum mismatches in a Download() [10:40] pstolowski: -^ [10:40] pstolowski: but I think the EOF case in download() is hanlded, no? the test at least indicates it is [10:40] pstolowski: or is EOF and unprovoced eof different? [10:41] zyga: sure, I have a look after lunch, I was under the impression you are discussing this with gustavo still, what the best fix is. or is there agreement? [10:42] mvo: gustavo discussed if the fixup function should be public or private so I made it private at his request [10:42] mvo: as for my feeling on how this should be fixed, we can do it without touching state at all if we pull out the patch I referenced above [10:57] PR snapd#3160 opened: overlord/ifacestate: automatically rename connections on core snap [11:00] Chipaca: hey, I'm seeing this timeout in completiontest [11:00] Chipaca: is this something expected of all expect-based tests? [11:00] Chipaca: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-image/xenial/amd64/s/snapd/20170406_130441_94744@/log.gz [11:01] zyga— [11:01] no [11:01] wtf [11:02] zyga— how does a telephone get in there [11:02] ???? [11:02] wat? [11:02] snap buy [11:02] hah [11:02] maybe it's my browser being silly [11:03] browsers are like humans [11:03] zyga— there's a real bug somewhere [11:03] we see faces in ascii, browsers see calendar events and phone numbers [11:03] zyga— expect times out when it fails to match something [11:03] so all epect errors are timeouts [11:03] you need to look at the output to determine what didn't match [11:04] zyga— that's the same error [11:04] zyga— that test is checking that "snap buy " tab-completes with purchasable snaps [11:05] zyga— and it's not [11:05] * zyga reads the output carefuly [11:05] aha [11:05] zyga— so somehting's broken (probably something changed in the store?) [11:05] so is that a store bug, snapd bog? [11:05] Chipaca: can we log from tab completion handlers [11:05] Chipaca: adding a debug section that collects such logs could be useful [11:06] zyga— the snapd-side of completion looks just like a regular search [11:06] zyga— and that will be in the journalctl output [11:06] Chipaca: aha, perfect, let me read the debug section then [11:07] : DEBUG: Retrying https://search.apps.ubuntu.com/api/v1/snaps/details ... [11:07] maybe it is just another EOF [11:07] finished after 1 retries, elapsed time=136.412205ms, status: 200 [11:07] zyga— what store is it looking at? and does that store have a purchasable test-assumes and test-snapd-tools? [11:07] pstolowski: ^ does that mean we got what we asked for or that we didn't and we gave up? [11:08] Chipaca: I don't know how to check [11:08] right now me neither [11:08] Chipaca: this is just tests/main/completion [11:08] (meaning it requires some more context that i haven't loaded) [11:08] the logging would be easier to grok if it said "after 1 try and 0 retries" for instance [11:09] Apr 06 12:44:13 autopkgtest snapd[5022]: 2017/04/06 12:44:13.129725 daemon.go:176: DEBUG: uid=0;@ GET /v2/find?name=test-%2A 5.359029622s 200 [11:09] *five seconds* [11:10] a query to the store took *five* *seconds* [11:10] Chipaca: aha, I see it [11:10] but the reply is 200 [11:10] stop using GSM [11:11] ogra_: but I like my nokia :D [11:11] :D [11:11] Chipaca: so do you think this is it? [11:11] Chipaca: store having a unbelievably slow reply? [11:12] if it comes to core you can drop "unbelivable" from that sentence [11:12] zyga— that test's expect timeouts at exactly 5 seconds [11:12] anything over 5 seconds will fail [11:13] Chipaca: interesting, thanks [11:13] realistically, any response from the store over ~200ms is too slow [11:13] Chipaca: I'll open a store thread about this [11:13] rendering the myapps.u.c page for core can easily take a min, there are simply many revisions ... processing them takes a while [11:13] so i wouldnt be surprised if the download request takes longer too [11:13] ogra_— myapps.u.c is a different service though [11:14] same backend, no [11:14] nope [11:14] ? [11:14] ah, k [11:14] the thing we hit for searches is a wrapped elasticsearch [11:14] probably to elastic then :P [11:15] (so elastic, it streches time) [11:15] ogra_— I think I need to have lunch to appreciate your sense of humour :-) [11:15] https://forum.snapcraft.io/t/chasing-unreliable-tests/158/8 [11:15] heh [11:16] Chipaca: thanks for your assistance! [11:18] fg [11:26] zyga, yes, 1 retry and 200 means we got the data [11:27] zyga: 3160 is also needed for 2.24 ? are all of your PRs that deal with the connection thing marked? [11:28] morphis, did you request whatever golang library does the progress thing to get updated in Fedora? [11:28] I recall zyga mentioned something about an issue with that in the bodhi updates [11:29] zyga: are all your PRs needed? also having them split out, doesn't that mean the testing is only partial? [11:31] zyga: so what needs to land? the core PR that renames the plugs on the core side and your three PRs? [11:31] mvo: I think it may not be needed in the end [11:32] mvo: the question is this, when we revert, do we auto-connect the reverted snap? [11:32] mvo: is it just like an install? [11:32] zyga: I think we do but we need to double check [11:32] mvo: if it is just like an install then we can land this branch, it matters little in practice because other branches will automatically connect core plugs anyway [11:33] mvo: and on refresh new connections will be made [11:33] mvo: I left it out specifically because it is a gray area and just a cleanup on top of the rest [11:33] zyga: it sounds like something we can test relatively easily? even a manual test would help to give me confidence in this [11:34] zyga: so its 3145,3154 and 3153 that we need to land for 2.24? [11:34] mvo: checking [11:34] zyga: could you make the forum entry for this a wiki ? then I can add a checklist [11:35] mvo: 3145 yes, for sure, 3154 (gee that was super confusing for my mind that swaps last two things around) yes; 3153 yes but I think it can only land after the others have helped [11:36] mvo: done, https://forum.snapcraft.io/t/duplicate-plug-slot-names-inside-the-core-snap/184 is now a wiki [11:36] a snap I've been working on in bits for a while is now stuck with: https://pastebin.ubuntu.com/24354062/ [11:37] mvo: are you going to edit it or should I? [11:37] zyga: please go ahead [11:37] the very first line of my main() function should also print out: "Main!" to stdout [11:37] icey: this is a bug in golang [11:37] zyga: a todo with the minimal fix we absolutely need for 2.24 would be great [11:37] such fun, it's running a compiled Rust binary :-/ [11:37] zyga: any ETA on a fix for that? [11:38] icey: it's fixed in ubuntu now I think but it needs to be fixed in other places that use go [11:38] icey: I think Chipaca is the person to ask [11:38] mvo: doing now [11:38] zyga: people ask about 2.24 and its not fully clear to me what we need to do now for the fix and what we can do later [11:38] zyga: in ubuntu now == 17.10 on Thursday? [11:38] zyga: thank you [11:38] icey: not sure really, sorry [11:38] 17.04 rather [11:38] zyga: talking about releases too much today [11:39] zyga do you know what version of snap it's fixed in? https://pastebin.ubuntu.com/24354069/ is what I'm on now [11:44] mmh, getting this kind of errors in autokpkgtest atm: error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. [11:45] mvo: I'm seeing a lot of tihs [11:45] Cloning into '/tmp/go/.cache/govendor/gopkg.in/mgo.v2'... [11:45] zyga: also is https://github.com/snapcore/snapd/pull/3145#discussion_r110329005 still relevant in 3145? i.e. do we need those plugs on the core snap afterall? or will the followup branches DTRT? sorry that I have so many questions [11:45] error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. [11:45] PR snapd#3145: overlord/ifacestate: fix auto-connect during core snap transition [11:45] context: # cd .; git clone https://gopkg.in/mgo.v2 /tmp/go/.cache/govendor/gopkg.in/mgo.v2 [11:45] Cloning into '/tmp/go/.cache/govendor/gopkg.in/mgo.v2'... [11:45] something wrong with gopkg.in? [11:45] mvo: yes, I think we still want to rename them, the internal rename in snapd is just a fixup but we should correct the actual yaml on disk later [11:45] pedronis: ha [11:45] pedronis: funny that we both reported it :) [11:46] yes, looks like something wrong, what is mgo.v2 [11:46] zyga: so the internal rename does not have to be in sync with the core snap? so 3145 is fine as is? [11:47] mvo: yes [11:48] mvo: collecting PRs [11:48] mvo: only one left [11:48] zyga: mgo.v2, mongo driver but actually where we get bson encoding support from , I think was added recently [11:48] mvo: please check https://forum.snapcraft.io/t/duplicate-plug-slot-names-inside-the-core-snap/184 again [11:49] see errtracker stuff [11:49] pedronis: aha, makes sense [11:49] right, I remember that [11:49] mvo: have a look at that and tell me if it makes sense [11:53] mvo: trivial fix on https://github.com/snapcore/core/pull/32#pullrequestreview-31819934 [11:53] PR core#32: Rename core-support,network-bind plugs to make them unique [11:58] * Chipaca back from lunch [11:58] icey, zyga, what's the question again? [11:58] Chipaca: golang bug with clone+exec [11:59] Chipaca: https://pastebin.ubuntu.com/24354062/ [11:59] Chipaca: pthread_create doesn't work while execing [11:59] icey— what're you running there? [11:59] Chipaca: alacritty :) [11:59] a snap I've been working on in bits for a while now [11:59] alright [11:59] icey— so, *most* of that family of errors are just warnings [11:59] it builds fine, but hangs when running; if I leave it long enough, I get that output [11:59] icey— but not all of them :-( [12:00] if I run the binary itself without snapd, it runs fine [12:00] the confined version never runs the first line of the main() function [12:00] icey— but none of it involves hanging [12:00] this particular binary also does exciting things with opengl and such [12:00] and is a classic snap [12:00] icey— that is: either it'll work (sometimes printing a spuriouos warning) or it won't (with something silly about permissions) [12:00] yeah, this one will literally hang forever (as far as I can tell) and periodically output that line [12:01] icey— so, the pthread warnings are *probably* ignorable; you're probably getting denies in dmesg [12:01] Chipaca: with a classic sna? :-/ [12:01] icey: with any kind IMO [12:01] icey— any snap can get denies [12:01] classic just means the jail is built differently (wider, roomier, but still a jail) [12:02] even decmode, hah (but harder to do) [12:02] zyga: Chipaca I'm getting https://pastebin.ubuntu.com/24354169/ in dmesg every time I start it [12:03] icey: are you connected over ssh? [12:03] zyga: I'm running locally [12:03] hmmm [12:03] odd [12:03] bash on laptop [12:03] I don't think this would work well over ssh, it [12:03] * zyga looks at the leet PID [12:03] it is a terminal using GPU acceletarion :) [12:03] acceleration* [12:04] icey: can you please report that bug [12:04] icey: it feels like something else now [12:04] icey: I recall seeing this somewhere, I'll talk to the security team about it [12:04] zyga: what data would you like on the bug? this dmesg output? [12:04] icey: please open a bug on launchpad.net/snap-confine [12:04] icey: that denial is key, please describe the context, maybe how to reproduce it [12:04] icey: does it happen with other snaps? [12:04] icey: or just this? [12:05] zyga: not that I have experienced. This snap has hit show-stopping bugs every time I've tried to move forward with it :) [12:05] icey— have you looked at snappy-debug.security scanlog? [12:05] Chipaca: ... no ... ? [12:05] Chipaca: how can I do that :) [12:06] icey— snap install snappy-debug [12:06] icey— snappy-debug.security scanlog your.service [12:06] icey— then start the service [12:06] icey— see if it says anything [12:07] jdstrand— ping [12:08] icey— or, snappy-debug.security --help [12:08] icey— for more details [12:08] * zyga AFK for a moment [12:09] although, the aduit log being on snap-confine might mean that it's bug#1672819 [12:09] Chipaca: running "snappy-debug.security scanlog alacritty", followed by starting `alacritty` in a new tab shows no output [12:09] zyga: 3154 has test failures that look real [12:09] ahem [12:09] * Chipaca pokes hal [12:09] bug 1672819 [12:09] Bug #1672819: exec'ing a setuid binary from a threaded program sometimes fails to setuid

[12:09] mup— thank you [12:10] icey— and you get that audit log every time? [12:10] as requested zyga: bug 1681421 [12:10] Bug #1681421: snap-confine dmesg error and snap hangs