[04:25] <mup> Bug #1576366 changed: juju 2 beta6: show-controller --format=json is broken <landscape> <juju-core:Expired> <https://launchpad.net/bugs/1576366>
[08:00] <perrito666> I believe reviewboard is not having a good day
[08:00] <perrito666> this never got a rb link https://github.com/juju/juju/pull/5747
[08:28] <mup> Bug #1597354 changed: Juju 2.0 Resource Error - cannot add resource failed to write data: read tcp : i/o timeout <juju-core:Incomplete> <https://launchpad.net/bugs/1597354>
[08:28] <mup> Bug #1598049 opened: TestLogRecordForwarded fails on non-ubuntu <blocker> <centos> <ci> <test-failure> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1598049>
[08:29] <mgz> cherylj, ericsnow: ^ bug A
[08:56] <mgz> ericsnow, katco: also bug 1598063 (I have not assigned this one to milestone, we may be able to punt)
[08:56] <mup> Bug #1598063: Data race in apiserver/observer package <race-condition> <juju-core:Triaged> <https://launchpad.net/bugs/1598063>
[08:57] <mgz> wait, I did, but maybe we punt
[09:01] <mup> Bug #1598063 opened: Data race in apiserver/observer package <race-condition> <juju-core:Triaged> <https://launchpad.net/bugs/1598063>
[09:39] <jam> mgz: https://github.com/juju/juju/pull/5750
[09:41] <mgz> sinzui: this looks good to me ^
[09:42] <mgz> ericsnow:
[09:42] <mgz> ok  	github.com/juju/juju/featuretests	138.465s
[09:42] <mgz> http://reports.vapour.ws/releases/4108/job/run-unit-tests-win2012-amd64/attempt/2566
[09:43] <mgz> ericsnow: that's 54617e0a from wed
[09:44] <mgz> so, it passed (possibly with that junk in the logs, not shown), before your changes
[09:44] <ericsnow> yep
[09:47] <sinzui> mgz: jam: yes this looks good. but consider that run-unit-tests always calls "go test -i ./..." now
[09:48] <jam> sinzui: I missed that, but I see it now.
[09:49] <sinzui> jam: so gig mgz...who add it to the script :)
[09:51] <sinzui> jam: mgz: In the past, we varied the command line for testing. 2 or 3 of the variations are no longer used. I think the only case where we don't use the makefile is running with race
[09:52] <mgz> sinzui: yeah, we were looking at the gating job specifically
[09:53] <sinzui> mgz: yeah that is one we flip-flop from makefile's test to --race
[09:53] <sinzui> mgz: and the goal for xenial-amd64 is to use race next week
[09:53] <sinzui> or today even
[10:14] <perrito666> would anyone kindly review https://github.com/juju/juju/pull/5747 that lacks a reviewboard link for reasons escape my control?
[10:15] <perrito666> axw: I addressed your comments for register, please re-check
[10:22] <axw> perrito666: reviewed
[10:22] <perrito666> axw: tx
[10:22] <perrito666> did you really go through the checklist?
[10:27] <anastasiamac> perrito666: did u keep checklist in mind while coding? :D
[10:27]  * axw looks shamefaced
[10:27] <axw> we're not starting till next week :)
[10:27] <perrito666> anastasiamac: evidently, I always do, even before it existed :p
[10:30] <perrito666> thumper: as if it had head you https://twitter.com/4BringingFire/status/748265855398576128
[11:11] <mup> Bug #1598113 opened: resource-get should not download if not necessary <resources> <juju-core:New> <https://launchpad.net/bugs/1598113>
[11:24] <perrito666> cherylj: got a fail from featuretest, which I little suspected where testing this
[11:25] <cherylj> :/
[11:28] <mgz> perrito666: bug 1598049?
[11:28] <mup> Bug #1598049: TestLogRecordForwarded fails on non-ubuntu <blocker> <centos> <ci> <test-failure> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1598049>
[11:28] <mgz> or a new one?
[11:30] <perrito666> mgz: a new one, I made a change in juju register and it would seem that the same functionality is being tested in the cmd tests and in feature tests
[11:53] <mup> Bug #1316223 changed: specifying juju deploy --networks=vlan:42 causes a panic <deploy> <juju-core:Invalid> <https://launchpad.net/bugs/1316223>
[11:53] <mup> Bug #1584805 changed: Timeout in github.com/juju/juju/apiserver/service on windows <bitesize> <ci> <regression> <test-failure> <timeout> <unit-tests> <windows> <juju-core:Fix Released> <juju-core 1.25:New> <https://launchpad.net/bugs/1584805>
[11:53] <mup> Bug #1595276 changed: TestDestroyControllerErrors failure with out of order errors <azure-provider> <ci> <intermittent-failure> <test-failure> <juju-core:Fix Released by fwereade> <https://launchpad.net/bugs/1595276>
[11:53] <mup> Bug #1598118 opened: log-forwarder worker bounces endlessly when forwarding is not configured <2.0> <debug-log> <log-forwarding> <logging> <juju-core:Triaged> <https://launchpad.net/bugs/1598118>
[12:02] <mup> Bug #1598127 opened: lxdbr0 spam in log file <logging> <juju-core:Triaged> <https://launchpad.net/bugs/1598127>
[12:10] <perrito666> mgz: this is a flaky run  right? http://juju-ci.vapour.ws:8080/job/github-merge-juju/8340/console
[12:14] <mup> Bug #1598127 changed: lxdbr0 spam in log file <logging> <juju-core:Triaged> <https://launchpad.net/bugs/1598127>
[12:23] <mup> Bug #1598127 opened: lxdbr0 spam in log file <logging> <juju-core:Triaged> <https://launchpad.net/bugs/1598127>
[13:32] <dimitern> cherylj: here it is - bug 1598164
[13:32] <mup> Bug #1598164: [aws] adding a machine post-bootstrap on the controller model closes of api port in controller security group <add-machine> <addressability> <ec2-provider> <tech-debt> <juju-core:New> <https://launchpad.net/bugs/1598164>
[13:32] <mup> Bug #1598164 opened: [aws] adding a machine post-bootstrap on the controller model closes of api port in controller security group <add-machine> <addressability> <ec2-provider> <tech-debt> <juju-core:New> <https://launchpad.net/bugs/1598164>
[13:34] <cherylj> thanks, dimitern!
[13:44] <katco> mgz: thanks, i'll pick up bug 1598063 shouldn't be a hard dx/fix
[13:44] <mup> Bug #1598063: Data race in apiserver/observer package <race-condition> <juju-core:Triaged by cox-katherine-e> <https://launchpad.net/bugs/1598063>
[13:47] <mgz> katco: thank you!
[15:03] <mgz> perrito666: bug 1598049
[15:03] <mup> Bug #1598049: TestLogRecordForwarded fails on non-ubuntu <blocker> <centos> <ci> <test-failure> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1598049>
[15:05] <perrito666> mgz: tx
[15:19] <katco> very simple review for someone: http://reviews.vapour.ws/r/5201/
[15:20] <katco> mgz: perhaps you are interested
[15:26] <mup> Bug #1598206 opened: lxc/lxd/shared/util_linux.go sys/types.h: No such file or directory <blocker> <ci> <ppc64el> <regression> <test-failure> <unit-tests> <juju-core:Triaged> <https://launchpad.net/bugs/1598206>
[15:28] <mgz> katco: ta
[16:22] <katco> wallyworld: ping
[16:23] <wallyworld> katco: hey, how'd the house go?
[16:23] <katco> wallyworld: fine, boring
[16:23] <wallyworld> boring is good
[16:23] <katco> wallyworld: yep :)
[16:23] <katco> wallyworld: re. your comment on my review. are you referring to the observer multiplexer?
[16:24] <katco> wallyworld: e.g. this? https://github.com/juju/juju/blob/master/apiserver/observer/observer.go#L102-L115
[16:24] <katco> i.e. rather
[16:24] <wallyworld> katco: https://github.com/juju/juju/blob/master/apiserver/observer/observer.go#L109
[16:25] <wallyworld> the use of a go routine inside the loop
[16:25] <katco> wallyworld: i don't think that's what's causing the issue
[16:25] <katco> wallyworld: the race is between a call to ServerRequest and ServerReply
[16:25] <wallyworld> the race output seemd to implicate that aspect; i was going by advice from williamn
[16:26] <wallyworld> ie the race output specically talks about calls from inside those go eoutines
[16:26] <katco> wallyworld: that loop will just call a single observer method on multiple observers concurrently.
[16:26] <katco> wallyworld: yeah, but not at the top of the call-stack...
[16:27] <katco> wallyworld: Previous write by goroutine 56:
[16:27] <katco>   github.com/juju/juju/apiserver/observer.(*RequestNotifier).ServerRequest()
[16:27] <katco> wallyworld: Read by goroutine 245:
[16:27] <katco>   github.com/juju/juju/apiserver/observer.(*RequestNotifier).ServerReply()
[16:27] <katco> wallyworld: the issue is that there are two calls coming into the same observer concurrently... i could remove the multiplexer entirely and this race would still occur
[16:28] <wallyworld> ok, my brain hasn't yet delved into the full detail, so i can't confirm mentally one way or the other
[16:29] <wallyworld> but we didn;t see this race before right
[16:29] <katco> wallyworld: this observer had mutexes before
[16:29] <wallyworld> the observer stuff seems to have introduced it
[16:30] <wallyworld> where were the mutexes? why were they removed?
[16:30] <wallyworld> did their removal intorduce the race?
[16:30] <katco> wallyworld: https://github.com/juju/juju/blob/bbc4a902fe44ee6effdd5e0216b3e0b8216643ef/apiserver/apiserver.go#L248
[16:30] <katco> wallyworld: because of what i said in the PR... i incorrectly assumed that requests/replies would happen synchronously
[16:31] <katco> wallyworld: the rpc server does not guarantee that. it has nothing to do with the multiplexer
[16:32] <katco> wallyworld: here's where ServerReply is kicked off on a new goroutine: https://github.com/juju/juju/blob/master/rpc/server.go#L465
[16:32] <wallyworld> katco: i'm slow today (or always) - so the above mutex on line 248 avoided races before this new work?
[16:32] <katco> wallyworld: apparently so
[16:32] <wallyworld> so why does this new work introduce the recaes then? is that mutex removed?
[16:33] <wallyworld> the observer stuff should not need extra locking?
[16:33] <katco> wallyworld: yes, because the rpc server will call ServerRequest, and then spawn another goroutine and call ServerReply
[16:34] <katco> wallyworld: if it's not in the observer, we need to touch rpc.Conn to lock there
[16:34] <katco> wallyworld: i.e. synchronize the reply to the request so it's synchronous
[16:36] <katco> wallyworld: i can tell you don't believe me; i'll just inject the RequestNotifier and show that the problem doesn't reside in the multiplexer
[16:40] <thumper> wallyworld: yarp
[16:44] <perrito666> sinzui: 2 things, 1) how will I know when I have my new mongo 2) how did you find a mongo for windows with ssl?
[16:45] <sinzui> perrito666: we rarely update the db. the choice to switch to 2.6 was driven by the case it is supported.
[16:45] <sinzui> perrito666: When we want to test only with 3.2, we will put 3.2 on the host
[17:18] <katco> wallyworld: ok, justification posted to the review
[17:32] <perrito666> mgz: https://github.com/juju/juju/pull/5759
[18:18] <mup> Bug # changed: 1575940, 1588403, 1594415, 1596967, 1597372, 1597516, 1597519
[18:42] <wallyworld> katco: hey, sorry, have been smashed trying to get log forwarding working using the actual code for the build
[18:43] <katco> wallyworld: no worries at all
[18:45] <wallyworld> katco: one comment given to me in additon was the use of a RWMutex instead of just a sync.Mutex - the added complexity is not justified
[18:46] <katco> wallyworld: i'm sorry, i don't understand how that's more complex?
[18:48] <mup> Bug #1598272 opened: LogStreamIntSuite.TestFullRequest sometimes fails <ci> <intermittent-failure> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1598272>
[18:56] <wallyworld> katco: it's cognitive overhead - when do i use read vs write lock, as opposed to just lock
[18:57] <mup> Bug #1585825 changed: Takes too long to download a resource from a controller to unit <ci> <resources> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1585825>
[18:57] <katco> wallyworld: shouldn't we be thinking about that?
[18:58] <wallyworld> katco: not prematurely
[18:58] <wallyworld> not unless it has been shown to be an issue
[18:58] <katco> wallyworld: it's our RPC mechanism, we want to limit our critical section as much as possible
[19:09] <mup> Bug #1585825 opened: Takes too long to download a resource from a controller to unit <ci> <resources> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1585825>
[19:18] <mup> Bug #1585825 changed: Takes too long to download a resource from a controller to unit <ci> <resources> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1585825>
[20:03] <mup> Bug # opened: 1598286, 1598289, 1598290, 1598291
[20:24] <mup> Bug # opened: 1598286, 1598289, 1598290, 1598291, 1598292, 1598293
[20:30] <mup> Bug #1598292 changed: log forwarding subject to clock skew <juju-core:Triaged> <https://launchpad.net/bugs/1598292>
[20:30] <mup> Bug #1598293 changed: log forwarding feature does not use updated config <juju-core:Triaged> <https://launchpad.net/bugs/1598293>
[20:36] <mup> Bug #1598292 opened: log forwarding subject to clock skew <juju-core:Triaged> <https://launchpad.net/bugs/1598292>
[20:36] <mup> Bug #1598293 opened: log forwarding feature does not use updated config <juju-core:Triaged> <https://launchpad.net/bugs/1598293>
[23:46] <mup> Bug #1598319 opened: Openstack Provider - No way to use multiple images <juju-core:New> <https://launchpad.net/bugs/1598319>