[00:01] agprado: you dropped out? [01:11] thumper: I don't know if you already saw, but wgrant is still seeing performance issues with 2.2.9 - still seing saslStart [01:11] see backscroll [01:42] thumper: also, I couldn't repro that intermittent test failure. tried it a couple hundred times [01:42] :( [01:42] ok [01:46] axw: if you had 5, just wanted a quick chat RE: the status PR. easier than typing into a PR [01:47] wallyworld: sure [01:47] standup [02:11] axw: thumper has a theory that api rate limiting is broken - that could explain the SASl overhead [02:11] as the server / mongo is getting hammered [02:11] okey dokey [02:11] he says prodstack retstart showed a thundering herd [02:12] all agents reconnected within a few seconds [02:12] maybe if we address rate limiting issue that will help alleviate the SASL symptoms [02:42] thumper: do you have some logs to share? [02:42] or were you looking at a screenshare [02:42] axw: no [02:42] I was watching the prodstack grafana instance [02:43] just noticed that the connection count for the controllers jumped back to full very quickly [02:45] thumper: looks to me like the connection count is incremented before rate-limiting kicks in [02:46] rate limiting happens in the Login handler, which is after we update teh conn count (which is what feeds the prometheus metric) [02:55] * thumper nods === frankban|afk is now known as frankban [10:46] balloons, ping? [11:43] how do I trigger a test build on a juju PR? [11:54] tasdomas: !!build!! [11:55] anastasiamac, thanks [11:55] nws :) [11:58] anastasiamac, could I get a review of https://github.com/juju/juju/pull/8262 ? [12:20] tasdomas: it's past my eod but tomorrow for sure ;) [12:20] anastasiamac, ah, sorry - enjoy your evening === salmankhan1 is now known as salmankhan [14:45] balloons, ping? [14:45] good day tasdomas [14:45] balloons, are juju acceptance tests actually being run? [14:46] tasdomas, not during the PR. They run post-commit [14:47] balloons, why is it looking into .juju/environments.yaml ( https://github.com/juju/juju/blob/develop/acceptancetests/jujupy/configuration.py#L44 ) ? [14:48] tasdomas, it looks for JUJU_HOME first as you can see [14:49] tasdomas, that should be JUJU_DATA, but JUJU_HOME is a juju 1ish that is still present [14:51] balloons, still, on l67 it looks for a file called environments.yaml [14:52] tasdomas, yes that file defines the cloud credentials [14:53] tasdomas, are you wanting to run your test? [14:54] tasdomas, typically we export JUJU_HOME which has the environments.yaml with our CI creds in it, and the tests go from there [14:54] that means we don't have to juju add-credentials everytime [14:54] balloons, so that is not the old-style environments.yaml that juju used to use [14:55] tasdomas, no.. it's a weird combination of juju1 and juju2 [14:55] tasdomas, we have a plan to eliminate all the juju 1ism's from the test framework, but the work is only about halfway done [14:56] the use of JUJU_HOME and environments.yaml is some of the last holdouts [15:44] tasdomas, if you like I can run your test changes through jenkins manually to ensure it passes [15:44] balloons, is there a way for me to do that myself? === frankban is now known as frankban|afk === hml_ is now known as hml [23:22] thumper: with the "too many open files" thing, forget what I said about state objects being open during upgrade. that's still a problem for upgrades from 2.1, but 2.2.8 to 2.2.9 wouldn't have run any upgrade steps [23:22] right === agprado is now known as agprado|afk === agprado|afk is now known as agprado