[00:01] <wallyworld> agprado: you dropped out?
[01:11] <axw> thumper: I don't know if you already saw, but wgrant is still seeing performance issues with 2.2.9 - still seing saslStart
[01:11] <axw> see backscroll
[01:42] <axw> thumper: also, I couldn't repro that intermittent test failure. tried it a couple hundred times
[01:42] <thumper> :(
[01:42] <thumper> ok
[01:46] <wallyworld> axw: if you had 5, just wanted a quick chat RE: the status PR. easier than typing into a PR
[01:47] <axw> wallyworld: sure
[01:47] <wallyworld> standup
[02:11] <wallyworld> axw: thumper has a theory that api rate limiting is broken - that could explain the SASl overhead
[02:11] <wallyworld> as the server / mongo is getting hammered
[02:11] <axw> okey dokey
[02:11] <wallyworld> he says prodstack retstart showed a thundering herd
[02:12] <wallyworld> all agents reconnected within a few seconds
[02:12] <wallyworld> maybe if we address rate limiting issue that will help alleviate the SASL symptoms
[02:42] <axw> thumper: do you have some logs to share?
[02:42] <axw> or were you looking at a screenshare
[02:42] <thumper> axw: no
[02:42] <thumper> I was watching the prodstack grafana instance
[02:43] <thumper> just noticed that the connection count for the controllers jumped back to full very quickly
[02:45] <axw> thumper: looks to me like the connection count is incremented before rate-limiting kicks in
[02:46] <axw> rate limiting happens in the Login handler, which is after we update teh conn count (which is what feeds the prometheus metric)
[02:55]  * thumper nods
[10:46] <tasdomas> balloons, ping?
[11:43] <tasdomas> how do I trigger a test build on a juju PR?
[11:54] <anastasiamac> tasdomas: !!build!!
[11:55] <tasdomas> anastasiamac, thanks
[11:55] <anastasiamac> nws :)
[11:58] <tasdomas> anastasiamac, could I get a review of https://github.com/juju/juju/pull/8262 ?
[12:20] <anastasiamac> tasdomas: it's past my eod but tomorrow for sure ;)
[12:20] <tasdomas> anastasiamac, ah, sorry - enjoy your evening
[14:45] <tasdomas> balloons, ping?
[14:45] <balloons> good day tasdomas
[14:45] <tasdomas> balloons, are juju acceptance tests actually being run?
[14:46] <balloons> tasdomas, not during the PR. They run post-commit
[14:47] <tasdomas> balloons, why is it looking into .juju/environments.yaml ( https://github.com/juju/juju/blob/develop/acceptancetests/jujupy/configuration.py#L44 ) ?
[14:48] <balloons> tasdomas, it looks for JUJU_HOME first as you can see
[14:49] <balloons> tasdomas, that should be JUJU_DATA, but JUJU_HOME is a juju 1ish that is still present
[14:51] <tasdomas> balloons, still, on l67 it looks for a file called environments.yaml
[14:52] <balloons> tasdomas, yes that file defines the cloud credentials
[14:53] <balloons> tasdomas, are you wanting to run your test?
[14:54] <balloons> tasdomas, typically we export JUJU_HOME which has the environments.yaml with our CI creds in it, and the tests go from there
[14:54] <balloons> that means we don't have to juju add-credentials everytime
[14:54] <tasdomas> balloons, so that is not the old-style environments.yaml that juju used to use
[14:55] <balloons> tasdomas, no.. it's a weird combination of juju1 and juju2
[14:55] <balloons> tasdomas, we have a plan to eliminate all the juju 1ism's from the test framework, but the work is only about halfway done
[14:56] <balloons> the use of JUJU_HOME and environments.yaml is some of the last holdouts
[15:44] <balloons> tasdomas, if you like I can run your test changes through jenkins manually to ensure it passes
[15:44] <tasdomas> balloons, is there a way for me to do that myself?
[23:22] <axw> thumper: with the "too many open files" thing, forget what I said about state objects being open during upgrade. that's still a problem for upgrades from 2.1, but 2.2.8 to 2.2.9 wouldn't have run any upgrade steps
[23:22] <thumper> right