[00:18] davecheney: so did I do something wrong to cause that panic on power? I've gone back and looked at that code and nothing's jumping out at me. [00:20] menn0: no idea [00:20] haven't looked [00:20] yet [00:20] fighting with the bot for my previous commit [00:20] menn0: if three was a mistake, it would relate to tools selection [00:20] and an unchecked error path [00:20] this is just suposition and does not indicate assignment of blame [00:27] davecheney: hmmm, I'll wait to see what you figure out. Let me know if I can help. [00:28] this bug is all fucked up [00:28] https://bugs.launchpad.net/bugs/1331744/+attachment/4134402/+files/run-unit-tests-ppc64el.log [00:28] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests

[00:28] is just one long line [00:28] no \r's [00:36] davecheney: view source [00:36] davecheney: it is being rendered as html [00:36] davecheney: it should have the content type of "text/plain" [00:36] but it doesn't [00:38] :sadtrombone: [00:52] grr, mongo keeps shitting itself in the tests [00:53] panic: Session already closed [00:53] goroutine 3633 [running]: [00:53] runtime.panic(0xa66ca0, 0xc2103c9720) /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6 [00:53] labix.org/v2/mgo.(*Session).cluster(0xc210a96c60, 0xc21035a820) /home/ubuntu/juju-core_1.19.4/src/labix.org/v2/mgo/session.go:1153 +0x66 [01:03] thumper: reschedule 1:1 after core meeting? [01:24] wallyworld: axw is there an issue for these mgo panics ? [01:24] https://github.com/juju/juju/pull/120 [01:24] 3 in a row trying to land this PR [01:25] yes [01:25] there's a couple [01:25] apiserver seems to fail more than others [01:27] wallyworld: yeah... [01:29] is it possible to get a build plan that runs with -race ? [01:29] it'd be a very small amount of SABDFL's massivel pile of SSL money that there is a race in there [01:30] s/be/bet [01:31] I've run the race detector and already fixed the races - sadly they were not related to the failures [01:32] :( [01:32] do I get SSL money now? [01:33] axw: sure, it's in the Isle of Man, i'll wait [01:46] sinzui: it is very naive for me to expect for you to be still here right? [01:47] perrito666, i am here. I have no social life [01:48] sinzui: neither do I, that is why I am also here [01:48] sinzui: how flexible is CI (can you replay old rev tests?) [01:50] I can play old revs, though only a few tests can play a specific rev. I often need to setup ci to retest a revision, but that is all the tests [01:50] perrito666, which rev do you care about? [01:51] I assume you want the backup-restore tests run again...they use the published streams so the early part of the test suite needs to be run to put those streams in place [01:51] sinzui: r0cbd5b87 its the last to have succesfully run [01:51] sinzui: I am trying to reduce the possible culprits of this odd breakage [01:51] I see it [01:53] sinzui: if r0cbd5b87 doesnt run I think I will say its something outside juju that changed if not I can look inside juju, in any case I can produce a fix somewhere tomorrow but I would really like to go less blind about it [01:53] perrito666, I ran the tests on Hp and aws [01:53] and switched between trusty and precise [01:57] perrito666, We can wait for CI to run that revision, I can could publish an alternate stream from these debs: http://juju-ci.vapour.ws:8080/job/publish-revision/496/ [01:58] sinzui: well in the beginning apt-get upgrade is run, so that could produce a version change although it should not [02:00] perrito666, the tools/streams we tested where made from those debs. The function tests use streams, so we need to republish to test again [02:00] sinzui: as you wish, let me know if you can replay r0cbd5b87, if you cant I think I can do a fairly decent work reproducing the test conditions but will be tomorrow [02:00] ok I think its better that I do it then [02:03] its a good thing I neved deleted that script [02:05] * perrito666 creates cheap stream [02:08] sinzui: running testsuite with the old revision, will see the result tomorrow, I am going to bed now [02:35] thumper: want to get it over and done with now? [02:35] wallyworld: yeah [02:35] wallyworld: just use the hangout on our normal meeting [02:35] already there === thumper is now known as thumper-cooking [04:26] * thumper-cooking back in an hour or so after kids have eaten [04:34] menn0: the bug is herer [04:34] status, err := apiclient.Status(c.patterns) [04:34] // Display any error, but continue to print status if some was returned [04:34] if err != nil { [04:34] fmt.Fprintf(ctx.Stderr, "%v\n", err) [04:34] } [04:34] result := newStatusFormatter(status).format() [04:35] aha [04:35] i'll fix [04:35] funny because that's mostly pre-existing code [04:35] I guess the way that status is used has changed [04:36] there may be several errors [04:37] i don't understand why comment 2 is unrelated to the original error [04:37] https://bugs.launchpad.net/juju-core/+bug/1331744 [04:37] menn0, thumper-cooking, davecheney: user info cmd PR ready for review: https://github.com/juju/juju/pull/126 [04:37] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests

[04:37] s/error/bug [04:37] I had to rewrite the branch because rebasing made a mess of it [04:40] run_test.go:293: c.Check(testing.Stdout(context), gc.Equals, string(jsonFormatted)+"\n") [04:40] ... obtained string = "[{\"Error\":\"command timed out\",\"MachineId\":\"1\",\"Stdout\":\"\"},{\"MachineId\":\"0\",\"Stdout\":\"megatron\\n\"}]\n" [04:40] ... expected string = "[{\"MachineId\":\"0\",\"Stdout\":\"megatron\\n\"},{\"Error\":\"command timed out\",\"MachineId\":\"1\",\"Stdout\":\"\"}]\n" [04:40] yay, map ordering errors [04:41] hmm, that is odd, these should always be sorted [04:45] davecheney: jc.SameContents ? [04:46] i think both are strings [04:46] not slices [04:46] oh right [04:58] % juju destroy-environment -y $(juju switch) [05:04] uh oh [05:04] lucky(~/src/github.com/juju/juju) % juju bootstrap --upload-tools [05:04] uploading tools for series [precise trusty] [05:04] Launching instance - i-4376dc11 [05:04] Waiting for address [05:04] Attempting to connect to ec2-54-91-67-54.compute-1.amazonaws.com:22 [05:04] Attempting to connect to ip-10-183-142-21.ec2.internal:22 [05:04] Attempting to connect to 54.91.67.54:22 [05:04] Attempting to connect to 10.183.142.21:22 [05:04] Logging to /var/log/cloud-init-output.log on remote host [05:04] Installing add-apt-repository [05:04] Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main [05:07] oh phew [05:07] it bootstrapped a precise machine [05:17] menn0: https://github.com/juju/juju/pull/127 [05:19] davecheney: are you serios? that printf caused the bug? [05:19] wallyworld_: nope [05:20] wallyworld_: the printf didn't cause the bug [05:20] continuing to use the value of status even though it's value could be nil caused the bug [05:21] ok, ta. i just skimmed the pr [05:21] but yes, i see now [05:22] i'm looking at the original report [05:22] i think the first report was unrelated to the bug in #2 [05:22] when i looked at it briefly, it seemed like another of those call stack differences [05:23] not sure, i'm trying on ppc now [05:23] if i can reproduce the issue i'll have to propose a branch with some extra debugging information [05:24] davecheney: just responded. I'm wondering if we should solve this in a different way (described on the PR). [05:24] crap [05:24] wallyworld_: ^^ [05:24] i already $$Merge$$ [05:25] i'll propose another branch [05:25] i can kill the jib [05:25] ok [05:25] wallyworld_: do that [05:25] davecheney, wallyworld_: cheers [05:25] then i'll push another change [05:25] sorry menn0 [05:27] menn0: wallyworld_ PTAL [05:29] davecheney: can't we stop the double print [05:30] s/can't/shouldn't ? [05:30] we can return nil [05:30] when the command will exit with 0 [05:30] that's what I just suggested [05:30] or not print it in the command? [05:30] and let the top level command print it? [05:30] wallyworld_: then we have two errors to return [05:30] the one from the call to get the status [05:30] and one from possibly printing the status [05:31] solution: move the if status == nil above the if err != nil [05:31] ok [05:31] (and keep returning err if status == nil) [05:31] fwereade, cmars happy Thursday: https://github.com/juju/juju/pull/128 [05:32] easy reading: diffstat +92,-0 [05:32] davecheney: menn0 : oh, can status be not nil even with an err? [05:33] wallyworld_: traditionally you can make no assertion about the contents of status if err != nil [05:33] wallyworld_: that's what the comment appears to suggest [05:33] well that sucks [05:34] i assumed if err != nil status would be unusable as you just said [05:34] wallyworld_: i agree [05:34] I guess the scenario will be that the status output was partially generated when an error was encountered [05:34] i don't know who put in the original logic [05:34] that was my original fix [05:34] it's still better to show something [05:34] menn0: you could argue that the contents of status aren't incomplete [05:34] they are unknown [05:34] original logic has been there for a while I suspect (before my time) [05:34] yeah [05:35] davecheney: you could argue that but the original author seemed to think it was better to show partial results than nothing at all [05:35] sure confusing if you don't look in detail [05:35] which I can see [05:35] menn0: so, i will be away at the sprint when you are in brisbane :-( [05:36] wallyworld_: I gathered that from the dates I heard during today's meeting :( [05:37] wallyworld_: no matter, I'll work from my parent's place. [05:37] you only here a week right? [05:37] A little over a week - 5th to the 15th [05:37] * menn0 will be in davecheney's timezone [05:37] menn0: you can stay at jools' place, right ? [05:37] davecheney: my parents live in Brisneyland ... that's the main reason I'll be there [05:38] not to visit jools? [05:38] he'll be hurt [05:38] working with Ian was going to be a side benefit [05:38] that's an odd turn of phrase [05:38] menn0: i'll be back morning of the 14th, i won't work that day but maybe we can catch up [05:39] I don't think I've met jools in person before although we are indirectly connected outside of canonical [05:39] wallyworld_: that'd be good if we can make it work [05:39] I'm flying on the 15th so it'd have to be the 14th [05:39] yup [05:39] leave it with me [05:42] davecheney: responded again to the PR [05:42] wallyworld_: which version of gccgo is installed on the build machine ? [05:42] actaully, maybe I answer that [05:42] davecheney: sec, otp [05:42] nope i can't [05:43] need help [05:45] davecheney: it's an ephemeral image so it pulls down whatever is in the archive [05:45] the console logs should show it [05:45] http://juju-ci.vapour.ws:8080/job/local-deploy-trusty-ppc64/99/console [05:46] doesn't show me [05:47] http://juju-ci.vapour.ws:8080/job/github-merge-juju/203/consoleFull [05:47] ah [05:47] you mean the ppc machine [05:47] yup [05:47] sorry [05:47] gotta check if it isn't using the broken compiler we shipped with trusty [05:51] still looking, but i have to duck out to get my son, i'll be back soon [05:54] morning [06:41] wallyworld_: ping [06:41] davecheney: hi, just got back [06:41] excellent [06:41] still figuring out how to tell what version [06:41] gccgo -v [06:41] will do [06:43] davecheney: gotta figure out the machine ip address, it's not obvious from the job that i can see [06:46] Morning [06:47] Will start a bit later today, I'm in the garage with my car. [06:47] At least some net access via phone. === vladk|offline is now known as vladk [06:51] wallyworld_: i've reproduce the panic on ppc [06:51] still need to know the compiler version your running [06:51] althought I suspect it is the unfixed versino [06:52] because the fixed version has not landed in trusty-updates [06:52] davecheney: yeah, i still can't find the ip address of the machine running the tests [06:54] fuk it [06:54] i'm 90% sure that the fix hasn't made it to trusty yet [06:54] so we're probably boned [06:58] davecheney: [06:58] ubuntu@stilson-07:~$ gccgo -v [06:58] Using built-in specs. [06:58] COLLECT_GCC=gccgo [06:58] COLLECT_LTO_WRAPPER=/usr/lib/gcc/powerpc64le-linux-gnu/4.9/lto-wrapper [06:58] Target: powerpc64le-linux-gnu [06:58] Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9-20140406-0ubuntu1' --with-bugurl=file:///usr/share/doc/gccgo-4.9/README.Bugs --enable-languages=c,c++,go --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu -- [06:58] enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --disable-libsanitizer --disable-libquadmath --enable-plugin --with-system-zlib --enable-secureplt --with-cpu=power7 --with-tune=power8 --disable-multilib --enable-multiarch --disable-werror --with-long-double-128 --enable-checking=release --build=powerpc64le-linux-gnu --host=powerpc64le-linux-gnu --target=powerpc64le-linux-gnu [06:58] Thread model: posix [06:58] gcc version 4.9.0 20140405 (experimental) [trunk revision 209157] (Ubuntu 4.9-20140406-0ubuntu1) [06:58] yup [06:58] hmm [06:58] finally found the ip address [06:59] i've definitely got the dud version on winton-09 [07:00] https://bugs.launchpad.net/ubuntu/+source/gccgo-4.9/+bug/1304754 [07:00] <_mup_> Bug #1304754: gccgo has issues when page size is not 4kB

[07:00] not fix released [07:00] :-( [07:00] * davecheney tries throwing ppa's at the problem [07:18] wallyworld_: good news and bad news [07:18] 1. bad news, compiler bug is not fixed [07:18] good news, this isn't the compiler bug [07:20] wallyworld_: [LOG] 0:00.458 DEBUG juju.state.apiserver -> [95] machine-0 171us {"RequestId":2,"Error":"objId is empty, dipshit","Response":{}} Machiner[""].Life [07:20] server_test.go:77: c.Assert(err, gc.IsNil) [07:20] ... value *params.Error = ¶ms.Error{"", "objId is empty, dipshit"} ("objId is empty, dipshit" [07:21] so, there is the cause [07:34] if I have two units of the same subordinate service "hulk-smashed" onto a single machine, will I get hooks being run concurrently? (e.g. unit/0 hookX and unit/1 hookX being run at the same time)? [07:35] I tried asking last night but didn't get a definitive answer :) [07:45] sparkiegeek: no [07:45] fwereade: rogpeppe1: ping about interfaces and reflect, I'm trying to fix bug https://bugs.launchpad.net/juju-core/+bug/1331744 I'm pretty sure I understand *why* it is failing, what I can't understand is why it ever worked. [07:45] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests

[07:45] wallyworld_: [LOG] 0:00.376 DEBUG juju.state.apiserver <- [95] machine-0 {"RequestId":2,"Type":"Machiner","Request":"Life","Params":{"Entities":[{"Tag":"machine-0"}]}} [07:45] [LOG] 0:00.377 DEBUG juju.state.apiserver -> [95] machine-0 184.387us {"RequestId":2,"Error":"objId is empty, dipshit","Response":{}} Machiner[""].Life [07:45] davecheney: great, thank you [07:45] jam: looking [07:45] fwereade: rogpeppe1: I'm in https://plus.google.com/hangouts/_/canonical.com/juju-sapphire [07:45] rogpeppe1: I can give the context on the hangout [07:46] the logs there like to be formatted as "html" which makes them wrap in horrible horrible ways :) [07:47] jam: look at the debug I sent wallyworld_ [07:47] objId is empty [07:48] davecheney: doesn't matter, I know what the bug is, I'm trying to figure out how to fix it [08:02] morning all [08:22] rogpeppe1: a thought about why it might have worked in golang go [08:22] if it puts its methods in sorted order [08:22] and all of our types that were being exposed via interfaces [08:22] jam: yeah, if gccgo doesn't, then that's the reason [08:22] didn't actually implement more public methods than just the interface methods [08:22] rogpeppe1: well gccgo could, but it looks like it puts private methods first [08:22] jam: right [08:22] and Ping comes before ping depending on how you sort :) [08:22] jam: that makes sense [08:23] jam: i suspected something like that [08:23] so I might have a good test case for golang go then [08:23] sounds like it's not a bug [08:23] rogpeppe1: well its a bug in our code, because of assumptions that only happened to match most of the time [08:23] (well, in all our current use cases :) [08:24] jam: yeah, worth adding a test case, sure [08:24] jam: if you have a test case [08:24] i can submit it to iant [08:24] he's pretty receptive to stuff like this [08:24] even if it's 'in spec' [08:24] davecheney: so again, it doesn't appear to be a gccgo bug [08:24] he knows weird arse shit that makes gccgo act differently drives adoption away [08:24] davecheney: it is a case where we were relying on accidental behavior which isn't even always true in golanggo but happened to be for how we were structuring our code. [08:40] rogpeppe1: yeah, I have a test which shows that it fails if your concrete type has extra methods. [08:40] jam: cool [08:48] bugfix for bug #1331744 https://github.com/juju/juju/pull/129 [08:48] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests

[08:50] rogpeppe1: ^^ if you want to review it. [08:50] Without the code fix, the test returned "A" instead of "second" [08:53] bbiab === Ursinha is now known as Ursinha-afk === vds` is now known as vds [10:28] mgz: i just got back and need to eat, can i ping you after the team lead meeting? [10:29] sure thing [10:34] fwereade, thanks for reviewing the networking model doc [10:35] jam: by the way, rfc 3230 digest headers don't support sha256 :-/ (only sha1 and md5) [10:35] fwereade, can you join us on our standup to discuss and agree upon what's not clear? [10:36] jam: we could use the digest header and an "officially non-standard" algorithm [10:37] jam: Digest: sha256= [10:37] dimitern, yeah, I'm not 100% sure I've finished orthat I'm very coherent today either [10:37] jam: that would be slightly more standard than our current X-Content header [10:37] jam: which I forgot to fix yesterday [10:38] dimitern, ...and hmm we have the leads meeting after the standup, and I have wallyworld_ after that [10:38] dimitern, after that maybe? [10:39] fwereade, whenever works for you [10:40] voidspace: are we using sha256? we could still stick with sha1 [10:41] mgz: i finished eating, let's do it quickly now if you're free [10:45] dimitern: TheMue: standup? [10:45] jam: yep === thumper-cooking is now known as thumper [11:05] vladk: I like that it shows you are thinking about it, and we can discuss how we've solved some of the problems you're bringing up, I just ran out of time. [11:06] * perrito666 is seconds away to having to use git bissect [11:06] perrito666: I've changed your backup functions to return a base64 encoded sha1 hash instead of a hex-encoded sha 256 hash [11:07] perrito666: I hope you have no objections to that [11:07] voidspace: nope, some reason in particular? [11:08] perrito666: the only standard http response header (defined by rfc) for sending a file hash [11:08] perrito666: uses base64 encoded sha1 hash [11:08] perrito666: and doesn't support sha 256 [11:09] mfoord: I see, I really hope you added a comment there :p [11:09] perrito666: git biset is really useful .. most people use git in a way that makes it less useful, but I've used it plenty of times to locate bugs. [11:09] perrito666: http://www.ietf.org/rfc/rfc3230.txt [11:09] s/biset/bisect [11:09] perrito666: I added as many comments as there were already... [11:09] perrito666: j/k - I'll add one [11:10] mfoord: "the code was self explanatory"® [11:10] perrito666: hehe [11:10] perrito666: the code is just as easy to read [11:10] perrito666: I've added a comment anyway [11:13] wwitzel3: yes, git bissect rocks, sadly I have no time to build a run command so it does all the test automatically [11:20] * perrito666 notices gmail header when the email has a flight booking ... creepy [11:27] perrito666: have we just cranked up the timeouts and see if that fixes things? Like making the timeouts 60 minutes or something? [11:28] natefinch: I have identified the what is not working, not yet the why [11:29] I have narrowed to a breaking commit between 240db2a2 0cbd5b87 [11:29] so now I am bisecting with hope to obtain the culprit and be able to work a solution from there [11:31] luckily I have a setup in place that allows me to re-create what jenkins does, stream included [11:31] perrito666: awesome [11:33] a lovely simple PR for someone to review [11:33] https://github.com/juju/juju/pull/130/files [11:33] natefinch: have you decided to stop sleeping? you where here when I left lastnight and are still here when I come back [11:33] perrito666: he has kids ;) [11:35] heh [11:35] perrito666: I would say the same to you [11:36] natefinch: I gave up sleeping [11:37] I stayed late debugging this and woke up to have breakfast with my wife [11:38] wwitzel3: thanks [11:38] that reminds me, I am really hungry .. thanks perrito666 [11:38] wwitzel3: np [11:38] mfoord: np [11:39] I like the way github diffs highlight the changed *part* of a line that was only modified [11:42] mfoord: you can get that behavior on the command line too using the diff-highlight script [11:42] cool [11:42] mfoord: you just need to enable it, it should be in your git/contrib [11:42] will look, thanks [11:55] natefinch: should we do our meeting befores standup day since we missed it yesterday? [11:58] wwitzel3: sounds good [12:09] cmars: I'd like to send this email, shall we meet up in 5-10 ? [12:10] jam, sounds good [12:15] morning all [12:15] jcw4, saw your Actions PR, woot! [13:12] this might be useful to others (I know at least jcw4 was asking about it) [13:12] https://gist.github.com/piscisaureus/3342247 === vladk is now known as vladk|offline [13:16] ok, says git bisect that 4a2f56528641c12f2e80cb17cc84e58d425d73bd is the culprit [13:30] natefinch: I'm in moonstone [13:30] wwitzel3: coming [13:36] fwereade, cmars: a PR for review: https://github.com/juju/juju/pull/131 thanks [13:37] wwitzel3: you see frozen? [13:54] Hi juju devs. Joyent had a storage outage 10 hours ago, that was the cause of the joyent deploy failure. I am stopping CI to reenable publication to joyent. I will restart the current revision. [13:56] sinzui: I think I found the offending commit, sadly its a very long one :s [13:58] perrito666: link me, maybe I can help [13:58] perrito666, I am SO SO sorry for not getting your rev tested. I had a power outage and joyent failed all about the same time. === Ursinha-afk is now known as Ursinha [13:59] sinzui: no hurries, I have a pretty good setup for that already [13:59] sinzui: I wonder if we could set up some kind of provider baseline test that proves the provider is currently working in an acceptable fashion before running our tests... that way we'd better understand what's failing, and our CI tests would look less spotty. [14:00] natefinch, we set that up last november [14:00] natefinch, http://juju-ci.vapour.ws:8080/view/Cloud%20Health/ [14:00] sinzui: oh awesome, I didn't realize [14:00] ^ the joyent job has a sad recent history [14:02] ericsnow: standup? [14:28] blocking on https://github.com/juju/juju/pull/128 if anyone can spare a sec to lgtm [14:40] Hey guys do we have Juju API docs on the web anywhere? [14:41] jcastro: I was looking yesterday and couldn't find any [14:42] jcastro: the best thing I could find was bindings for Python: https://launchpad.net/python-jujuclient [14:48]