[00:18] <menn0> davecheney: so did I do something wrong to cause that panic on power? I've gone back and looked at that code and nothing's jumping out at me.
[00:20] <davecheney> menn0: no idea
[00:20] <davecheney> haven't looked
[00:20] <davecheney> yet
[00:20] <davecheney> fighting with the bot for my previous commit
[00:20] <davecheney> menn0: if three was a mistake, it would relate to tools selection
[00:20] <davecheney> and an unchecked error path
[00:20] <davecheney> this is just suposition and does not indicate assignment of blame
[00:27] <menn0> davecheney: hmmm, I'll wait to see what you figure out. Let me know if I can help.
[00:28] <davecheney> this bug is all fucked up
[00:28] <davecheney> https://bugs.launchpad.net/bugs/1331744/+attachment/4134402/+files/run-unit-tests-ppc64el.log
[00:28] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests <ci> <gccgo> <ppc64el> <regression> <juju-core:Triaged by jameinel> <https://launchpad.net/bugs/1331744>
[00:28] <davecheney> is just one long line
[00:28] <davecheney> no \r's
[00:36] <thumper> davecheney: view source
[00:36] <thumper> davecheney: it is being rendered as html
[00:36] <thumper> davecheney: it should have the content type of "text/plain"
[00:36] <thumper> but it doesn't
[00:38] <davecheney> :sadtrombone:
[00:52] <davecheney> grr, mongo keeps shitting itself in the tests
[00:53] <davecheney> panic: Session already closed
[00:53] <davecheney> goroutine 3633 [running]:
[00:53] <davecheney> runtime.panic(0xa66ca0, 0xc2103c9720) /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
[00:53] <davecheney> labix.org/v2/mgo.(*Session).cluster(0xc210a96c60, 0xc21035a820) /home/ubuntu/juju-core_1.19.4/src/labix.org/v2/mgo/session.go:1153 +0x66
[01:03] <wallyworld> thumper: reschedule 1:1 after core meeting?
[01:24] <davecheney> wallyworld: axw is there an issue for these mgo panics ?
[01:24] <davecheney> https://github.com/juju/juju/pull/120
[01:24] <davecheney> 3 in a row trying to land this PR
[01:25] <axw> yes
[01:25] <axw> there's a couple
[01:25] <axw> apiserver seems to fail more than others
[01:27] <thumper> wallyworld: yeah...
[01:29] <davecheney> is it possible to get a build plan that runs with -race ?
[01:29] <davecheney> it'd be a very small amount of SABDFL's massivel pile of SSL money that there is a race in there
[01:30] <davecheney> s/be/bet
[01:31] <axw> I've run the race detector and already fixed the races - sadly they were not related to the failures
[01:32] <davecheney> :(
[01:32] <axw> do I get SSL money now?
[01:33] <davecheney> axw: sure, it's in the Isle of Man, i'll wait
[01:46] <perrito666> sinzui: it is very naive for me to expect for you to be still here right?
[01:47] <sinzui> perrito666, i am here. I have no social life
[01:48] <perrito666> sinzui: neither do I, that is why I am also here
[01:48] <perrito666> sinzui: how flexible is CI (can you replay old rev tests?)
[01:50] <sinzui> I can play old revs, though only a few tests can play a specific rev. I often need to setup ci to retest a revision, but that is all the tests
[01:50] <sinzui> perrito666, which rev do you care about?
[01:51] <sinzui> I assume you want the backup-restore tests run again...they use the published streams so the early part of the test suite needs to be run to put those streams in place
[01:51] <perrito666> sinzui: r0cbd5b87 its the last to have succesfully run
[01:51] <perrito666> sinzui: I am trying to reduce the possible culprits of this odd breakage
[01:51] <sinzui> I see it
[01:53] <perrito666> sinzui: if r0cbd5b87 doesnt run I think I will say its something outside juju that changed if not I can look inside juju, in any case I can produce a fix somewhere tomorrow but I would really like to go less blind about it
[01:53] <sinzui> perrito666, I ran the tests on Hp and aws
[01:53] <sinzui> and switched between trusty and precise
[01:57] <sinzui> perrito666, We can wait for CI to run that revision, I can could publish an alternate stream from these debs: http://juju-ci.vapour.ws:8080/job/publish-revision/496/
[01:58] <perrito666> sinzui: well in the beginning apt-get upgrade is run, so that could produce a version change although it should not
[02:00] <sinzui> perrito666, the tools/streams we tested where made from those debs. The function tests use streams, so we need to republish to test again
[02:00] <perrito666> sinzui: as you wish, let me know if you can replay r0cbd5b87, if you cant I think I can do a fairly decent work reproducing the test conditions but will be tomorrow
[02:00] <perrito666> ok I think its better that I do it then
[02:03] <perrito666> its a good thing I neved deleted that script
[02:05]  * perrito666 creates cheap stream
[02:08] <perrito666> sinzui: running testsuite with the old revision, will see the result tomorrow, I am going to bed now
[02:35] <wallyworld> thumper: want to get it over and done with now?
[02:35] <thumper> wallyworld: yeah
[02:35] <thumper> wallyworld: just use the hangout on our normal meeting
[02:35] <wallyworld> already there
[04:26]  * thumper-cooking back in an hour or so after kids have eaten
[04:34] <davecheney> menn0: the bug is herer
[04:34] <davecheney>         status, err := apiclient.Status(c.patterns)
[04:34] <davecheney>         // Display any error, but continue to print status if some was returned
[04:34] <davecheney>         if err != nil {
[04:34] <davecheney>                 fmt.Fprintf(ctx.Stderr, "%v\n", err)
[04:34] <davecheney>         }
[04:34] <davecheney>         result := newStatusFormatter(status).format()
[04:35] <menn0> aha
[04:35] <davecheney> i'll fix
[04:35] <menn0> funny because that's mostly pre-existing code
[04:35] <menn0> I guess the way that status is used has changed
[04:36] <davecheney> there may be several errors
[04:37] <davecheney> i don't understand why comment 2 is unrelated to the original error
[04:37] <davecheney> https://bugs.launchpad.net/juju-core/+bug/1331744
[04:37] <waigani> menn0, thumper-cooking, davecheney: user info cmd PR ready for review: https://github.com/juju/juju/pull/126
[04:37] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests <ci> <gccgo> <ppc64el> <regression> <juju-core:Triaged by jameinel> <https://launchpad.net/bugs/1331744>
[04:37] <davecheney> s/error/bug
[04:37] <waigani> I had to rewrite the branch because rebasing made a mess of it
[04:40] <davecheney> run_test.go:293: c.Check(testing.Stdout(context), gc.Equals, string(jsonFormatted)+"\n")
[04:40] <davecheney> ... obtained string = "[{\"Error\":\"command timed out\",\"MachineId\":\"1\",\"Stdout\":\"\"},{\"MachineId\":\"0\",\"Stdout\":\"megatron\\n\"}]\n"
[04:40] <davecheney> ... expected string = "[{\"MachineId\":\"0\",\"Stdout\":\"megatron\\n\"},{\"Error\":\"command timed out\",\"MachineId\":\"1\",\"Stdout\":\"\"}]\n"
[04:40] <davecheney> yay, map ordering errors
[04:41] <davecheney> hmm, that is odd, these should always be sorted
[04:45] <waigani> davecheney: jc.SameContents ?
[04:46] <davecheney> i think both are strings
[04:46] <davecheney> not slices
[04:46] <waigani> oh right
[04:58] <davecheney>  % juju destroy-environment -y $(juju switch)
[05:04] <davecheney> uh oh
[05:04] <davecheney> lucky(~/src/github.com/juju/juju) % juju bootstrap --upload-tools
[05:04] <davecheney> uploading tools for series [precise trusty]
[05:04] <davecheney> Launching instance - i-4376dc11
[05:04] <davecheney> Waiting for address
[05:04] <davecheney> Attempting to connect to ec2-54-91-67-54.compute-1.amazonaws.com:22
[05:04] <davecheney> Attempting to connect to ip-10-183-142-21.ec2.internal:22
[05:04] <davecheney> Attempting to connect to 54.91.67.54:22
[05:04] <davecheney> Attempting to connect to 10.183.142.21:22
[05:04] <davecheney> Logging to /var/log/cloud-init-output.log on remote host
[05:04] <davecheney> Installing add-apt-repository
[05:04] <davecheney> Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main
[05:07] <davecheney> oh phew
[05:07] <davecheney> it bootstrapped a precise machine
[05:17] <davecheney> menn0: https://github.com/juju/juju/pull/127
[05:19] <wallyworld_> davecheney: are you serios? that printf caused the bug?
[05:19] <davecheney> wallyworld_: nope
[05:20] <davecheney> wallyworld_: the printf didn't cause the bug
[05:20] <davecheney> continuing to use the value of status even though it's value could be nil caused the bug
[05:21] <wallyworld_> ok, ta. i just skimmed the pr
[05:21] <wallyworld_> but yes, i see now
[05:22] <davecheney> i'm looking at the original report
[05:22] <davecheney> i think the first report was unrelated to the bug in #2
[05:22] <wallyworld_> when i looked at it briefly, it seemed like another of those call stack differences
[05:23] <davecheney> not sure, i'm trying on ppc now
[05:23] <davecheney> if i can reproduce the issue i'll have to propose a branch with some extra debugging information
[05:24] <menn0> davecheney: just responded. I'm wondering if we should solve this in a different way (described on the PR).
[05:24] <davecheney> crap
[05:24] <menn0> wallyworld_: ^^
[05:24] <davecheney> i already $$Merge$$
[05:25] <davecheney> i'll propose another branch
[05:25] <wallyworld_> i can kill the jib
[05:25] <davecheney> ok
[05:25] <davecheney> wallyworld_: do that
[05:25] <menn0> davecheney, wallyworld_: cheers
[05:25] <davecheney> then i'll push another change
[05:25] <wallyworld_> sorry menn0
[05:27] <davecheney> menn0: wallyworld_ PTAL
[05:29] <wallyworld_> davecheney: can't we stop the double print
[05:30] <wallyworld_> s/can't/shouldn't ?
[05:30] <davecheney> we can return nil
[05:30] <davecheney> when the command will exit with 0
[05:30] <menn0> that's what I just suggested
[05:30] <wallyworld_> or not print it in the command?
[05:30] <wallyworld_> and let the top level command print it?
[05:30] <davecheney> wallyworld_: then we have two errors to return
[05:30] <davecheney> the one from the call to get the status
[05:30] <davecheney> and one from possibly printing the status
[05:31] <menn0> solution: move the if status == nil above the if err != nil
[05:31] <davecheney> ok
[05:31] <menn0> (and keep returning err if status == nil)
[05:31] <jcw4> fwereade, cmars happy Thursday: https://github.com/juju/juju/pull/128
[05:32] <jcw4> easy reading: diffstat +92,-0
[05:32] <wallyworld_> davecheney: menn0 : oh, can status be not nil even with an err?
[05:33] <davecheney> wallyworld_: traditionally you can make no assertion about the contents of status if err != nil
[05:33] <menn0> wallyworld_: that's what the comment appears to suggest
[05:33] <wallyworld_> well that sucks
[05:34] <wallyworld_> i assumed if err != nil status would be unusable as you just said
[05:34] <davecheney> wallyworld_: i agree
[05:34] <menn0> I guess the scenario will be that the status output was partially generated when an error was encountered
[05:34] <davecheney> i don't know who put in the original logic
[05:34] <davecheney> that was my original fix
[05:34] <menn0> it's still better to show something
[05:34] <davecheney> menn0: you could argue that the contents of status aren't incomplete
[05:34] <davecheney> they are unknown
[05:34] <menn0> original logic has been there for a while I suspect (before my time)
[05:34] <wallyworld_> yeah
[05:35] <menn0> davecheney: you could argue that but the original author seemed to think it was better to show partial results than nothing at all
[05:35] <wallyworld_> sure confusing if you don't look in detail
[05:35] <menn0> which I can see
[05:35] <wallyworld_> menn0: so, i will be away at the sprint when you are in brisbane :-(
[05:36] <menn0> wallyworld_: I gathered that from the dates I heard during today's meeting :(
[05:37] <menn0> wallyworld_: no matter, I'll work from my parent's place.
[05:37] <wallyworld_> you only here a week right?
[05:37] <menn0> A little over a week - 5th to the 15th
[05:37]  * menn0 will be in davecheney's timezone
[05:37] <davecheney> menn0: you can stay at jools' place, right ?
[05:37] <menn0> davecheney: my parents live in Brisneyland ... that's the main reason I'll be there
[05:38] <davecheney> not to visit jools?
[05:38] <davecheney> he'll be hurt
[05:38] <menn0> working with Ian was going to be a side benefit
[05:38] <davecheney> that's an odd turn of phrase
[05:38] <wallyworld_> menn0: i'll be back morning of the 14th, i won't work that day but maybe we can catch up
[05:39] <menn0> I don't think I've met jools in person before although we are indirectly connected outside of canonical
[05:39] <menn0> wallyworld_: that'd be good if we can make it work
[05:39] <menn0> I'm flying on the 15th so it'd have to be the 14th
[05:39] <wallyworld_> yup
[05:39] <wallyworld_> leave it with me
[05:42] <menn0> davecheney: responded again to the PR
[05:42] <davecheney> wallyworld_: which version of gccgo is installed on the build machine ?
[05:42] <davecheney> actaully, maybe I answer that
[05:42] <wallyworld_> davecheney: sec, otp
[05:42] <davecheney> nope i can't
[05:43] <davecheney> need help
[05:45] <wallyworld_> davecheney: it's an ephemeral image so it pulls down whatever is in the archive
[05:45] <wallyworld_> the console logs should show it
[05:45] <davecheney> http://juju-ci.vapour.ws:8080/job/local-deploy-trusty-ppc64/99/console
[05:46] <davecheney> doesn't show me
[05:47] <wallyworld_> http://juju-ci.vapour.ws:8080/job/github-merge-juju/203/consoleFull
[05:47] <wallyworld_> ah
[05:47] <wallyworld_> you mean the ppc machine
[05:47] <davecheney> yup
[05:47] <wallyworld_> sorry
[05:47] <davecheney> gotta check if it isn't using the broken compiler we shipped with trusty
[05:51] <wallyworld_> still looking, but i have to duck out to get my son, i'll be back soon
[05:54] <dimitern> morning
[06:41] <davecheney> wallyworld_: ping
[06:41] <wallyworld_> davecheney: hi, just got back
[06:41] <davecheney> excellent
[06:41] <wallyworld_> still figuring out how to tell what version
[06:41] <davecheney> gccgo -v
[06:41] <davecheney> will do
[06:43] <wallyworld_> davecheney: gotta figure out the machine ip address, it's not obvious from the job that i can see
[06:46] <TheMue> Morning
[06:47] <TheMue> Will start a bit later today, I'm in the garage with my car.
[06:47] <TheMue> At least some net access via phone.
[06:51] <davecheney> wallyworld_: i've reproduce the panic on ppc
[06:51] <davecheney> still need to know the compiler version your running
[06:51] <davecheney> althought I suspect it is the unfixed versino
[06:52] <davecheney> because the fixed version has not landed in trusty-updates
[06:52] <wallyworld_> davecheney: yeah, i still can't find the ip address of the machine running the tests
[06:54] <davecheney> fuk it
[06:54] <davecheney> i'm 90% sure that the fix hasn't made it to trusty yet
[06:54] <davecheney> so we're probably boned
[06:58] <wallyworld_> davecheney:
[06:58] <wallyworld_> ubuntu@stilson-07:~$ gccgo -v
[06:58] <wallyworld_> Using built-in specs.
[06:58] <wallyworld_> COLLECT_GCC=gccgo
[06:58] <wallyworld_> COLLECT_LTO_WRAPPER=/usr/lib/gcc/powerpc64le-linux-gnu/4.9/lto-wrapper
[06:58] <wallyworld_> Target: powerpc64le-linux-gnu
[06:58] <wallyworld_> Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9-20140406-0ubuntu1' --with-bugurl=file:///usr/share/doc/gccgo-4.9/README.Bugs --enable-languages=c,c++,go --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --
[06:58] <wallyworld_> enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --disable-libsanitizer --disable-libquadmath --enable-plugin --with-system-zlib --enable-secureplt --with-cpu=power7 --with-tune=power8 --disable-multilib --enable-multiarch --disable-werror --with-long-double-128 --enable-checking=release --build=powerpc64le-linux-gnu --host=powerpc64le-linux-gnu --target=powerpc64le-linux-gnu
[06:58] <wallyworld_> Thread model: posix
[06:58] <wallyworld_> gcc version 4.9.0 20140405 (experimental) [trunk revision 209157] (Ubuntu 4.9-20140406-0ubuntu1)
[06:58] <davecheney> yup
[06:58] <davecheney> hmm
[06:58] <wallyworld_> finally found the ip address
[06:59] <davecheney> i've definitely got the dud version on winton-09
[07:00] <davecheney> https://bugs.launchpad.net/ubuntu/+source/gccgo-4.9/+bug/1304754
[07:00] <_mup_> Bug #1304754: gccgo has issues when page size is not 4kB <ppc64el> <trusty> <gcc:Fix Released> <gcc-4.9 (Ubuntu):Fix Released> <gccgo-4.9 (Ubuntu):Invalid> <gcc-4.9 (Ubuntu Trusty):Invalid> <gccgo-4.9 (Ubuntu Trusty):Fix Committed by doko> <gcc-4.9 (Ubuntu Utopic):Fix Released> <gccgo-4.9 (Ubuntu Utopic):Invalid> <https://launchpad.net/bugs/1304754>
[07:00] <davecheney> not fix released
[07:00] <wallyworld_> :-(
[07:00]  * davecheney tries throwing ppa's at the problem
[07:18] <davecheney> wallyworld_: good news and bad news
[07:18] <davecheney> 1. bad news, compiler bug is not fixed
[07:18] <davecheney> good news, this isn't the compiler bug
[07:20] <davecheney> wallyworld_: [LOG] 0:00.458 DEBUG juju.state.apiserver -> [95] machine-0 171us {"RequestId":2,"Error":"objId is empty, dipshit","Response":{}} Machiner[""].Life
[07:20] <davecheney> server_test.go:77: c.Assert(err, gc.IsNil)
[07:20] <davecheney> ... value *params.Error = &params.Error{"", "objId is empty, dipshit"} ("objId is empty, dipshit"
[07:21] <davecheney> so, there is the cause
[07:34] <sparkiegeek> if I have two units of the same subordinate service "hulk-smashed" onto a single machine, will I get hooks being run concurrently? (e.g. unit/0 hookX and unit/1 hookX being run at the same time)?
[07:35] <sparkiegeek> I tried asking last night but didn't get a definitive answer :)
[07:45] <davecheney> sparkiegeek: no
[07:45] <jam> fwereade: rogpeppe1: ping about interfaces and reflect, I'm trying to fix bug https://bugs.launchpad.net/juju-core/+bug/1331744  I'm pretty sure I understand *why* it is failing, what I can't understand is why it ever worked.
[07:45] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests <ci> <gccgo> <ppc64el> <regression> <juju-core:Triaged by jameinel> <https://launchpad.net/bugs/1331744>
[07:45] <davecheney> wallyworld_: [LOG] 0:00.376 DEBUG juju.state.apiserver <- [95] machine-0 {"RequestId":2,"Type":"Machiner","Request":"Life","Params":{"Entities":[{"Tag":"machine-0"}]}}
[07:45] <davecheney> [LOG] 0:00.377 DEBUG juju.state.apiserver -> [95] machine-0 184.387us {"RequestId":2,"Error":"objId is empty, dipshit","Response":{}} Machiner[""].Life
[07:45] <sparkiegeek> davecheney: great, thank you
[07:45] <rogpeppe1> jam: looking
[07:45] <jam> fwereade: rogpeppe1: I'm in https://plus.google.com/hangouts/_/canonical.com/juju-sapphire
[07:45] <jam> rogpeppe1: I can give the context on the hangout
[07:46] <jam> the logs there like to be formatted as "html" which makes them wrap in horrible horrible ways :)
[07:47] <davecheney> jam: look at the debug I sent wallyworld_
[07:47] <davecheney> objId is empty
[07:48] <jam> davecheney: doesn't matter, I know what the bug is, I'm trying to figure out how to fix it
[08:02] <voidspace> morning all
[08:22] <jam> rogpeppe1: a thought about why it might have worked in golang go
[08:22] <jam> if it puts its methods in sorted order
[08:22] <jam> and all of our types that were being exposed via interfaces
[08:22] <rogpeppe1> jam: yeah, if gccgo doesn't, then that's the reason
[08:22] <jam> didn't actually implement more public methods than just the interface methods
[08:22] <jam> rogpeppe1: well gccgo could, but it looks like it puts private methods first
[08:22] <rogpeppe1> jam: right
[08:22] <jam> and Ping comes before ping depending on how you sort :)
[08:22] <rogpeppe1> jam: that makes sense
[08:23] <rogpeppe1> jam: i suspected something like that
[08:23] <jam> so I might have a good test case for golang go then
[08:23] <rogpeppe1> sounds like it's not a bug
[08:23] <jam> rogpeppe1: well its a bug in our code, because of assumptions that only happened to match most of the time
[08:23] <jam> (well, in all our current use cases :)
[08:24] <rogpeppe1> jam: yeah, worth adding a test case, sure
[08:24] <davecheney> jam: if you have a test case
[08:24] <davecheney> i can submit it to iant
[08:24] <davecheney> he's pretty receptive to stuff like this
[08:24] <davecheney> even if it's 'in spec'
[08:24] <jam> davecheney: so again, it doesn't appear to be a gccgo bug
[08:24] <davecheney> he knows weird arse shit that makes gccgo act differently drives adoption away
[08:24] <jam> davecheney: it is a case where we were relying on accidental behavior which isn't even always true in golanggo but happened to be for how we were structuring our code.
[08:40] <jam> rogpeppe1: yeah, I have a test which shows that it fails if your concrete type has extra methods.
[08:40] <rogpeppe1> jam: cool
[08:48] <jam> bugfix for bug #1331744 https://github.com/juju/juju/pull/129
[08:48] <_mup_> Bug #1331744: panic: reflect: Call of unexported method ppc64el/gccgo unit tests <ci> <gccgo> <ppc64el> <regression> <juju-core:Triaged by jameinel> <https://launchpad.net/bugs/1331744>
[08:50] <jam> rogpeppe1: ^^ if you want to review it.
[08:50] <jam> Without the code fix, the test returned "A" instead of "second"
[08:53] <jam> bbiab
[10:28] <wallyworld_> mgz: i just got back and need to eat, can i ping you after the team lead meeting?
[10:29] <mgz> sure thing
[10:34] <dimitern> fwereade, thanks for reviewing the networking model doc
[10:35] <voidspace> jam: by the way, rfc 3230 digest headers don't support sha256 :-/ (only sha1 and md5)
[10:35] <dimitern> fwereade, can you join us on our standup to discuss and agree upon what's not clear?
[10:36] <voidspace> jam: we could use the digest header and an "officially non-standard" algorithm
[10:37] <voidspace> jam: Digest: sha256=<base64 encoded hash>
[10:37] <fwereade> dimitern, yeah, I'm not 100% sure I've finished orthat I'm very coherent today either
[10:37] <voidspace> jam: that would be slightly more standard than our current X-Content header
[10:37] <voidspace> jam: which I forgot to fix yesterday
[10:38] <fwereade> dimitern, ...and hmm we have the leads meeting after the standup, and I have wallyworld_ after that
[10:38] <fwereade> dimitern, after that maybe?
[10:39] <dimitern> fwereade, whenever works for you
[10:40] <jam> voidspace: are we using sha256? we could still stick with sha1
[10:41] <wallyworld_> mgz: i finished eating, let's do it quickly now if you're free
[10:45] <jam> dimitern: TheMue: standup?
[10:45] <TheMue> jam: yep
[11:05] <jam> vladk: I like that it shows you are thinking about it, and we can discuss how we've solved some of the problems you're bringing up, I just ran out of time.
[11:06]  * perrito666 is seconds away to having to use git bissect
[11:06] <voidspace> perrito666: I've changed your backup functions to return a base64 encoded sha1 hash instead of a hex-encoded sha 256 hash
[11:07] <voidspace> perrito666: I hope you have no objections to that
[11:07] <perrito666> voidspace: nope, some reason in particular?
[11:08] <mfoord> perrito666: the only standard http response header (defined by rfc) for sending a file hash
[11:08] <mfoord> perrito666: uses base64 encoded sha1 hash
[11:08] <mfoord> perrito666: and doesn't support sha 256
[11:09] <perrito666> mfoord: I see, I really hope you added a comment there :p
[11:09] <wwitzel3> perrito666: git biset is really useful .. most people use git in a way that makes it less useful, but I've used it plenty of times to locate bugs.
[11:09] <mfoord> perrito666: http://www.ietf.org/rfc/rfc3230.txt
[11:09] <wwitzel3> s/biset/bisect
[11:09] <mfoord> perrito666: I added as many comments as there were already...
[11:09] <mfoord> perrito666: j/k - I'll add one
[11:10] <perrito666> mfoord: "the code was self explanatory"®
[11:10] <mfoord> perrito666: hehe
[11:10] <mfoord> perrito666: the code is just as easy to read
[11:10] <mfoord> perrito666: I've added a comment anyway
[11:13] <perrito666> wwitzel3: yes, git bissect rocks, sadly I have no time to build a run command so it does all the test automatically
[11:20]  * perrito666 notices gmail header when the email has a flight booking ... creepy
[11:27] <natefinch> perrito666: have we just cranked up the timeouts and see if that fixes things?  Like making the timeouts 60 minutes or something?
[11:28] <perrito666> natefinch: I have identified the what is not working, not yet the why
[11:29] <perrito666> I have narrowed to a breaking commit between 240db2a2 0cbd5b87
[11:29] <perrito666> so now I am bisecting with hope to obtain the culprit and be able to work a solution from there
[11:31] <perrito666> luckily I have a setup in place that allows me to re-create what jenkins does, stream included
[11:31] <natefinch> perrito666: awesome
[11:33] <mfoord> a lovely simple PR for someone to review
[11:33] <mfoord> https://github.com/juju/juju/pull/130/files
[11:33] <perrito666> natefinch: have you decided to stop sleeping? you where here when I left lastnight and are still here when I come back
[11:33] <wwitzel3> perrito666: he has kids ;)
[11:35] <natefinch> heh
[11:35] <natefinch> perrito666: I would say the same to you
[11:36] <perrito666> natefinch: I gave up sleeping
[11:37] <perrito666> I stayed late debugging this and woke up to have breakfast with my wife
[11:38] <mfoord> wwitzel3: thanks
[11:38] <wwitzel3> that reminds me, I am really hungry .. thanks perrito666
[11:38] <perrito666> wwitzel3: np
[11:38] <wwitzel3> mfoord: np
[11:39] <mfoord> I like the way github diffs highlight the changed *part* of a line that was only modified
[11:42] <wwitzel3> mfoord: you can get that behavior on the command line too using the diff-highlight script
[11:42] <mfoord> cool
[11:42] <wwitzel3> mfoord: you just need to enable it, it should be in your git/contrib
[11:42] <mfoord> will look, thanks
[11:55] <wwitzel3> natefinch: should we do our meeting befores standup day since we missed it yesterday?
[11:58] <natefinch> wwitzel3: sounds good
[12:09] <jam> cmars: I'd like to send this email, shall we meet up in 5-10 ?
[12:10] <cmars> jam, sounds good
[12:15] <bodie_> morning all
[12:15] <bodie_> jcw4, saw your Actions PR, woot!
[13:12] <bodie_> this might be useful to others (I know at least jcw4 was asking about it)
[13:12] <bodie_> https://gist.github.com/piscisaureus/3342247
[13:16] <perrito666> ok, says git bisect that 4a2f56528641c12f2e80cb17cc84e58d425d73bd is the culprit
[13:30] <wwitzel3> natefinch: I'm in moonstone
[13:30] <natefinch> wwitzel3: coming
[13:36] <TheMue> fwereade, cmars: a PR for review: https://github.com/juju/juju/pull/131 thanks
[13:37] <natefinch> wwitzel3: you see frozen?
[13:54] <sinzui> Hi juju devs. Joyent had a storage outage 10 hours ago, that was the cause of the joyent deploy failure. I am stopping CI to reenable publication to joyent. I will restart the current revision.
[13:56] <perrito666> sinzui: I think I found the offending commit, sadly its a very long one :s
[13:58] <natefinch> perrito666: link me, maybe I can help
[13:58] <sinzui> perrito666, I am SO SO sorry for not getting your rev tested. I had a power outage and joyent failed all about the same time.
[13:59] <perrito666> sinzui: no hurries, I have a pretty good setup for that already
[13:59] <natefinch> sinzui: I wonder if we could set up some kind of provider baseline test that proves the provider is currently working in an acceptable fashion before running our tests... that way we'd better understand what's failing, and our CI tests would look less spotty.
[14:00] <sinzui> natefinch, we set that up last november
[14:00] <sinzui> natefinch, http://juju-ci.vapour.ws:8080/view/Cloud%20Health/
[14:00] <natefinch> sinzui: oh awesome, I didn't realize
[14:00] <sinzui> ^ the joyent job has a sad recent history
[14:02] <natefinch> ericsnow: standup?
[14:28] <bodie_> blocking on https://github.com/juju/juju/pull/128 if anyone can spare a sec to lgtm
[14:40] <jcastro> Hey guys do we have Juju API docs on the web anywhere?
[14:41] <ericsnow> jcastro: I was looking yesterday and couldn't find any
[14:42] <ericsnow> jcastro: the best thing I could find was bindings for Python: https://launchpad.net/python-jujuclient
[14:48] <TheMue> jcastro: we’ve so far only got a rough text about the general design
[14:48] <TheMue> jcastro: a developer documentation is currently wip
[14:50] <natefinch> TheMue: it would make me really happy if this could be a reasonable place to look for docs: http://godoc.org/github.com/juju/juju/state/api
[14:50] <ericsnow> TheMue: that would be great
[14:51] <natefinch> TheMue: obviously having something on juju.ubuntu.com would also be good... but the closer you are to the code, the more likely the docs will stay up to date
[14:51] <TheMue> natefinch: they will be written in markdown, so that they can be rendered directly in GH. additionally they will be rendered for juju.ubuntu.com
[14:52] <TheMue> natefinch: the docs are no documentation of the individual funcs, that may be indeed better there
[14:52] <TheMue> natefinch: they are a design documentation about RPC and versioning for maintenance and extension
[14:52] <TheMue> natefinch: so docs != docs
[14:53] <natefinch> TheMue: I think when people ask for docs about the API, they mean "what methods does it contain and what do they do?"
[14:53] <ericsnow> natefinch, TheMue: right
[14:54] <TheMue> natefinch: that’s one perspective, yes. another one is design, to get new team members and contributers faster into our system
[14:54] <TheMue> natefinch: that’s where I’m working on
[14:54] <natefinch> TheMue: cool
[14:55] <TheMue> natefinch: finding a way to generate also a good overview of the API functions automatically too would be nice, yes. will think about it
[14:55] <jcastro> TheMue, what's the timeline for API docs? someone is making a production decision based on when we have that
[14:55] <ericsnow> TheMue: another perspective is people writing bindings--they need both
[14:55] <natefinch> jcastro: are they looking for docs about what endpoints are exposed and what they do, etc?
[14:55] <jcastro> yeah
[14:55] <natefinch> jcastro: this is a good start: http://godoc.org/github.com/juju/juju/state/api
[14:55] <TheMue> jcastro: which perspective is the interesting one here? I think nates view of the available functions?
[14:55] <jcastro> they want to integrate with jenkins etc.
[14:56] <jcastro> he's the guy talking in #juju
[14:57] <jcastro> I have to run the cloud cross team call, if one of you guys can take over in #juju TheMue or natefinch
[14:58] <natefinch> jcastro: I'm on it
[14:59] <natefinch> wwitzel3: can you try helping perrito666 figure out his mongo issue?  It's not fun, but it's quite important
[15:00] <perrito666> natefinch: well I know I am no fun, but its not like you have to say it on public :p
[15:00] <natefinch> haha
[15:00] <natefinch> it's mongo that's no fun
[15:00]  * fwereade goes to lie down a bit more, back later
[15:19] <wwitzel3> natefinch: yep
[15:19] <wwitzel3> perrito666: I'm in moonstone
[15:20] <perrito666> wwitzel3: hold a sec
[15:21] <perrito666> wwitzel3: your room is on moonstone
[15:41]  * perrito666 plays silent cinema with wwitzel3
[15:41] <wwitzel3> perrito666: lol
[15:41] <wwitzel3> perrito666: that CI link you sent me 404, can you link it here
[15:42] <perrito666> http://4.bp.blogspot.com/-5jfw91ZJefw/UvxN4qJNfoI/AAAAAAAATNc/9Jh7whIzWmE/s1600/Cinelists+-+Silent+Movie-+Mel+Brooks+(20).jpg
[15:42] <perrito666> wwitzel3: http://juju-ci.vapour.ws:8080/job/functional-ha-backup-restore/
[15:54] <perrito666> sinzui:
[15:54] <perrito666> this happens only with ha right?
[15:55] <sinzui> perrito666, I have seen just backup-restore fail, but that passed yesterday
[16:03] <ericsnow> voidspace:  with the new backup client I see that you are following the lead of AddLocalCharm and UploadTools
[16:03] <voidspace> ericsnow: in terms of?
[16:03] <voidspace> ericsnow: what's the link to your PR by the way?
[16:03] <voidspace> ericsnow: I'm about ready for it
[16:04] <ericsnow> voidspace: the direct HTTP request stuff rather than using the RPC call machinery
[16:04] <voidspace> ericsnow: right, indeed
[16:04] <rogpeppe1> dimitern, natefinch, voidspace, mgz, perrito666, fwereade, jam: mechanical changes here, preparing for adding bundle support to the charm package, review appreciated: https://github.com/juju/charm/pull/7
[16:05] <ericsnow> voidspace: https://github.com/ericsnowcurrently/juju/pull/1/files
[16:05] <voidspace> ericsnow: it needs to be, I don't want to read the whole (potentially gigabyte sized) file into memory before writing it to disk
[16:06] <ericsnow> voidspace: can't you use a different codec with the RPC/websocket code that does it while still using the existing RPC machinery?
[16:06] <ericsnow> voidspace: WatchDebugLog already does something along these lines
[16:06] <voidspace> ericsnow: I don't know :-)
[16:06] <voidspace> ericsnow: ah right, I looked for examples and the first I found was the charm stuff
[16:06] <voidspace> ericsnow: it isn't much work though
[16:07] <voidspace> ericsnow: five lines of code to prepare and do the call
[16:07] <voidspace> ericsnow: or so
[16:07] <ericsnow> voidspace: I'm guess that those 3 oddball API methods do their own thing because it was the easiest way to get it done :)
[16:07] <voidspace> right
[16:07] <voidspace> I'm not sure how far out of our way we should go to find a harder way... ;-)
[16:08] <ericsnow> voidspace: considering the comments in AddLocalCharm() and UploadTools(), it may be worth factoring at least some of the code into a common binary blob RPC client
[16:09] <dimitern> rogpeppe1, lgtm
[16:09] <voidspace> ericsnow: maybe
[16:09] <ericsnow> voidspace: at which point it may be worth just going the binary-blob-instead-of-JSON codec route
[16:09] <voidspace> ericsnow: grabbing coffee and then will take a proper look at the code
[16:09] <ericsnow> voidspace: if I read all that code right (it gets a little twisty in there)
[16:10] <ericsnow> voidspace: sounds good :)
[16:11] <ericsnow> BTW, I just put up a doc patch for backup/restore (once the CLI changes land): https://github.com/juju/docs/pull/124
[16:20] <voidspace> natefinch: ping
[16:20] <voidspace> actually
[16:20] <voidspace> natefinch: unping
[16:20] <voidspace> alexisb: ping
[16:20] <alexisb> voidspace, pong
[16:41] <voidspace> ericsnow: as far as I know we can't extend the Client type from another file
[16:42] <voidspace> ericsnow: which is why we don't break the client up into multiple files
[16:42] <voidspace> ericsnow: we could move common functionality into a base class and then have several client types
[16:42] <voidspace> ericsnow: but that seems like extra complexity
[16:42] <ericsnow> voidspace: oh, I figured intra-package was okay
[16:43] <voidspace> ericsnow: I *may* be wrong
[16:43] <voidspace> natefinch: ping? ^^^
[16:43] <voidspace> ericsnow: there isn't much common code I can see to factor out between the binary blob methods
[16:44] <ericsnow> voidspace: at least the part around  "resp, err := utils.GetNonValidatingHTTPClient().Do(req)"
[16:45] <voidspace> ericsnow: right - I intend to do that already, for when we switch to a validating client
[16:45] <voidspace> ericsnow: (have getting the client be a single method used by all of them)
[16:45] <ericsnow> voidspace: exactly
[16:46] <voidspace> ericsnow: WatchDebugLog shows how to do that correctly I think, so I'll just copy that code :-)
[16:48] <voidspace> ericsnow: except the error handling looks janky - it reads the first line to see if there's an error
[16:48] <voidspace> ericsnow: I mean, we *could* do that with Backup, assuming a newline appears *somewhere*
[16:48] <voidspace> ericsnow: but it's a bit yucky
[16:49] <ericsnow> voidspace: yeah :)
[16:50] <voidspace> ericsnow: hmmm
[16:50] <voidspace> ericsnow: that looks a little hard to get round
[16:51] <natefinch> voidspace: you can put functions for a type in any file in the same package as that type.  files are just an organizational tool, they don't really interact with the language itself.
[16:51] <voidspace> ericsnow: ah no, I think that API endpoint always sends an error response in the first line
[16:52] <voidspace> ericsnow: and you tell if there was an error by checking the Error of the unmarshalled json of the first line
[16:52] <voidspace> ericsnow: so it's not a problem - it's a janky api endpoint, not a feature of websockets
[16:52] <voidspace> natefinch: ok, cool
[16:52] <ericsnow> voidspace: good
[16:52] <voidspace> natefinch: sooo... for the client Backup method, ericsnow is suggesting we put this in a new file
[16:53] <voidspace> natefinch: especially as Backup will have related methods for restore soon
[16:53] <natefinch> sure
[16:53] <voidspace> okey-dokey
[16:53] <voidspace> ericsnow: I'm going to punt on factoring it out into a different type though
[16:53] <natefinch> files don't really matter.  It can be helpful to break stuff up.  it's your call
[16:53] <voidspace> ericsnow: I want to use some of the client methods for obtaining the websocket
[16:53] <ericsnow> voidspace: no worries (limited resources, etc.)
[16:53] <voidspace> aye
[16:54] <voidspace> ericsnow: as I'm away from Saturday I'll get done what I get done (probably most of it bar complete tests)
[16:54] <voidspace> ericsnow: and hand the rest over if not complete
[16:54] <ericsnow> voidspace: sounds good
[17:20] <jcw4> fwereade, cmars : happy Thursday :) https://github.com/juju/juju/pull/132
[17:20] <jcw4> PTAL
[17:28] <voidspace> ericsnow: hmmm... it seems like the janky api in WatchDebugLog maybe because websockets don't have a statuscode
[17:28] <voidspace> ericsnow: and they may not have headers easily accessible either
[17:28] <voidspace> ericsnow: so this may be a dead end
[17:29] <voidspace> ericsnow: or we duplicate the janky api to send any error response and the sha
[17:29] <voidspace> ericsnow: (i.e. send a json response as the first line and *then* the file)
[17:29] <ericsnow> voidspace: that's what I was thinking :)
[17:29] <voidspace> ericsnow: I'll look tomorrow
[17:29] <voidspace> I'm EOD now
[17:30] <ericsnow> voidspace: alrighty
[17:30] <voidspace> ericsnow: I pushed the changes that split out the Backup method into its own file at least
[17:30] <voidspace> ericsnow: I haven't pushed the websocket changes as I'm not sure whether or not to go down that road
[17:30] <voidspace> see you tomorrow everyone
[17:30] <voidspace> g'night
[18:16] <bodie_> blocking on a LGTM for https://github.com/juju/juju/pull/132
[18:19] <cmars> bodie_, jcw4 not familiar enough with actions or watchers to approve, but I don't see anything wrong with it either
[18:20] <jcw4> cmars: thanks; this part has nothing new that is inherent to actions... just a new filtered watcher on the actions collection
[18:21] <jcw4> cmars appreciate your comments!  I don't mind waiting for someone who knows watchers better to LGTM
[18:22] <jcw4> (isn't that a weird nounification ^^ ?)
[18:22] <jcw4> or wait... verbification?
[18:22] <bodie_> I think it's a verbed noun, yes
[18:23] <bodie_> verbing is the best
[18:23] <jcw4> haha
[18:23] <jcw4> I see what you did there
[18:23] <bodie_> and LGTM itself is a nouned exclamation...
[18:24] <bodie_> english...
[18:24] <bodie_> not a well-typed language
[18:24] <jcw4> ... is why they invented Go
[18:24] <jcw4> har har
[18:24] <bodie_> nyuk nyuk
[18:25] <perrito666> sinzui: will you stop growing that bug?
[18:27] <sinzui> perrito666, sorry, I am just looking for evidence. since branches are linked to bugs any more. I had to read every commit since the last release. I just happened to see things that looked relevant to the tests failing
[18:27] <perrito666> just kidding I identified the commit that breaks at least in aws
[18:28] <sinzui> perrito666, fab, I moved both jobs back to aws this morning
[18:30] <bac> sinzui: are there any known issues with juju and local providers on utopic?
[18:31] <sinzui> bac no
[18:31] <bac> thanks
[18:32] <sinzui> bac: it has a perfect record in fact *after* the template is created. If you abort the first bootstrap, the template is borked
[18:50] <perrito666> mm, I think something is really making our tree ugly
[18:51] <wwitzel3> perrito666: ?
[18:52] <hazmat> rogpeppe1, one item of concern with machines in the bundle spec is the issues of concurrency around usage of those machines
[18:52] <perrito666> wwitzel3: something Was not making sense to me so I went into a graphical git thinguie and our tree looks a bit messy but perhaps its what I am using
[18:52] <hazmat> rogpeppe1, ie. juju will blithely assign a different workload to an unused machine even though it was created for the bundle
[18:57] <bodie_> in case anyone missed the pr, actions work is blocking on https://github.com/juju/juju/pull/132
[18:57] <bodie_> pretty simple one
[18:57] <jcw4> someone with watcher knowledge could review it in just a few minutes ^^
[19:41] <perrito666> natefinch: lol, what a rant
[19:41] <perrito666> natefinch: I think the preferred g-apps way is share this as a gdrive attachment
[19:41] <natefinch> I have a pet peeve about email size limits
[19:42] <natefinch> some people like pretending we're all still on dialup
[19:43] <perrito666> heh, I recall some old dev on my team, some time ago telling me "but what if the user has js dissabled" ... "me: who does that" "him: blind people" .."me: for gods sake, we are making a movies guide site, really? blind people?"
[19:45] <perrito666> and just to make it worse, the discussion was around some js for a button that had the caption "I have seen this"
[19:45] <natefinch> lol
[19:50] <TheMue> natefinch: great pic *smile*
[19:50] <natefinch> :)
[19:50] <natefinch> thrilled to have a boy.  I know a lot of people with three girls.  Of course, now I have to figure out how to raise a boy.
[19:52] <perrito666> natefinch: you should have gone througt that process at least once
[19:52] <natefinch> perrito666: and look how *that* turned out! :)
[19:53] <perrito666> well, you have your kids go to standups, the apple cant fall so far from the tree
[19:58] <hazmat> sinzui, which azure regions are being tested?
[20:10] <sinzui> hazmat, US West...the only one that ever worked
[20:17] <perrito666> wwitzel3: got any luck?
[20:17] <perrito666> wallyworld_: lemme know when you come to life please
[20:19] <wallyworld_> perrito666: hey, having my morning coffee. only 6am here so i have a few things to do to get the kid ready for school so i can't give you my full attention, but i may be able to help
[20:20] <perrito666> well, I dont want to ruin your morning coffee but https://github.com/juju/juju/commit/4a2f56528641c12f2e80cb17cc84e58d425d73bd#diff-dec2c5aafeb31667d3e3abc2232ec107R50 is most likely the commit that broke restore, I could use some help figuring what exactly it is
[20:20] <perrito666> so yea, I can wait until your full attention is available :p
[20:22] <wallyworld_> perrito666: well that sucks :-( do you know what the failure is? a permissions issue?
[20:23] <wallyworld_> perrito666: note that there was a follow up to that branch that did fix a permissions issue
[20:23] <perrito666> wallyworld_: I wish, the failure is, there is no failure, there is a part of the restore process that is undone :| Its a bit of a hassle to explain over chat, we can have a quick call whenever you finish your morning setup
[20:23] <wallyworld_> so if you run with that branch and not the follow up one, it will fail
[20:24] <wallyworld_> the follow up branch adds a ClusterAdmin permission to user creation
[20:24] <perrito666> wallyworld_: even if I am running as admin?
[20:24] <wallyworld_> i think so
[20:24] <perrito666> ok, what is the branch?
[20:25] <perrito666> d1e0262efcf878bad24c4235873b0d740f21f320
[20:25] <sinzui> bodie_, is there anything to announce regarding Actions in 1.19.4?
[20:26] <wallyworld_> perrito666: https://github.com/juju/juju/commit/94e7f692cc2c145d67de9c1f5d3e6cda53097c41
[20:26] <jcw4> sinzui: not for 1.19.4
[20:26] <sinzui> goody
[20:26] <jcw4> :) okay...
[20:27] <wallyworld_> perrito666: i have to be afk for a bit, will chek back with you later
[20:27] <perrito666> tx
[20:53] <hazmat> sinzui, ack, thanks.. apparently there have been some issues in the azure api in eu region.. was talking to cpc folks
[21:03] <wallyworld_> perrito666: raining here so i have to drive the kid to school. so i'll bbiab if you need anything
[21:04] <perrito666> wallyworld_: np, drive safely
[22:48] <thumper> waigani: I have a guy here quoting on windows, may not be done by 11, can you tell dave?
[22:49] <waigani> wow, confusing - I just had the same! yep, shall we go ahead without you?
[22:58] <jcw4> I don't suppose there are any folks down under that want to do a quick review on a new Watcher?
[22:58] <jcw4> The first part already got merged and this is part 2
[22:58] <jcw4> https://github.com/juju/juju/pull/132
[23:06] <thumper> waigani, davecheney: window guy still here, perhaps another 10-15 min
[23:16] <thumper> ok, here now
[23:16] <thumper> waigani, davecheney: go now?
[23:16] <waigani> thumper: we are in hangout now :0
[23:17] <waigani> :)
[23:20] <sinzui> WTF, ec2 trusty cannot bootstrap with a recent rev
[23:20]  * sinzui looks for bad CI
[23:25] <perrito666> wallyworld_: returned?
[23:25] <wallyworld_> perrito666: yup, a while ago, sorry
[23:25] <perrito666> wallyworld_: thats ok, I took a nap, its the night here, wanna do a quick hangout before I go have dinner?
[23:26] <wallyworld_> thumper: has the window guy taken the window out and put in back in again?
[23:26] <wallyworld_> sure
[23:26] <perrito666> wallyworld_: come to https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=3
[23:31] <thumper> wallyworld_: nah, just quoting at this stage
[23:34] <perrito666> thumper: so, how much for the licence :p?
[23:34] <thumper> haha
[23:35]  * sinzui slaps termination protect on the bootstrap node so juju cannot destroy the evidence
[23:36] <sinzui> wallyworld_, thumper , if you have bzr-git and qbzr installed, I have found that qlog makes more sense out of what jujubot merged than git itself
[23:36] <thumper> ha
[23:37] <sinzui> git log  shows me  many possible revs that can break juju, but qlog shows the real merges
[23:39] <sinzui> bugger, I wonder if trusty changed its mongo, or the new trusty images used by aws don't work with juju
[23:49] <sinzui> thumper, wallyworld_ More suck, https://bugs.launchpad.net/juju-core/+bug/1332365
[23:49] <_mup_> Bug #1332365: trusty EC2: cannot initiate replica set <bootstrap> <ec2> <mongodb> <trusty> <juju-core:Triaged> <https://launchpad.net/bugs/1332365>
[23:50] <wallyworld_> :-(