| wallyworld | agprado: you dropped out? | 00:01 |
|---|---|---|
| axw | thumper: I don't know if you already saw, but wgrant is still seeing performance issues with 2.2.9 - still seing saslStart | 01:11 |
| axw | see backscroll | 01:11 |
| axw | thumper: also, I couldn't repro that intermittent test failure. tried it a couple hundred times | 01:42 |
| thumper | :( | 01:42 |
| thumper | ok | 01:42 |
| wallyworld | axw: if you had 5, just wanted a quick chat RE: the status PR. easier than typing into a PR | 01:46 |
| axw | wallyworld: sure | 01:47 |
| wallyworld | standup | 01:47 |
| wallyworld | axw: thumper has a theory that api rate limiting is broken - that could explain the SASl overhead | 02:11 |
| wallyworld | as the server / mongo is getting hammered | 02:11 |
| axw | okey dokey | 02:11 |
| wallyworld | he says prodstack retstart showed a thundering herd | 02:11 |
| wallyworld | all agents reconnected within a few seconds | 02:12 |
| wallyworld | maybe if we address rate limiting issue that will help alleviate the SASL symptoms | 02:12 |
| axw | thumper: do you have some logs to share? | 02:42 |
| axw | or were you looking at a screenshare | 02:42 |
| thumper | axw: no | 02:42 |
| thumper | I was watching the prodstack grafana instance | 02:42 |
| thumper | just noticed that the connection count for the controllers jumped back to full very quickly | 02:43 |
| axw | thumper: looks to me like the connection count is incremented before rate-limiting kicks in | 02:45 |
| axw | rate limiting happens in the Login handler, which is after we update teh conn count (which is what feeds the prometheus metric) | 02:46 |
| * thumper nods | 02:55 | |
| === frankban|afk is now known as frankban | ||
| tasdomas | balloons, ping? | 10:46 |
| tasdomas | how do I trigger a test build on a juju PR? | 11:43 |
| anastasiamac | tasdomas: !!build!! | 11:54 |
| tasdomas | anastasiamac, thanks | 11:55 |
| anastasiamac | nws :) | 11:55 |
| tasdomas | anastasiamac, could I get a review of https://github.com/juju/juju/pull/8262 ? | 11:58 |
| anastasiamac | tasdomas: it's past my eod but tomorrow for sure ;) | 12:20 |
| tasdomas | anastasiamac, ah, sorry - enjoy your evening | 12:20 |
| === salmankhan1 is now known as salmankhan | ||
| tasdomas | balloons, ping? | 14:45 |
| balloons | good day tasdomas | 14:45 |
| tasdomas | balloons, are juju acceptance tests actually being run? | 14:45 |
| balloons | tasdomas, not during the PR. They run post-commit | 14:46 |
| tasdomas | balloons, why is it looking into .juju/environments.yaml ( https://github.com/juju/juju/blob/develop/acceptancetests/jujupy/configuration.py#L44 ) ? | 14:47 |
| balloons | tasdomas, it looks for JUJU_HOME first as you can see | 14:48 |
| balloons | tasdomas, that should be JUJU_DATA, but JUJU_HOME is a juju 1ish that is still present | 14:49 |
| tasdomas | balloons, still, on l67 it looks for a file called environments.yaml | 14:51 |
| balloons | tasdomas, yes that file defines the cloud credentials | 14:52 |
| balloons | tasdomas, are you wanting to run your test? | 14:53 |
| balloons | tasdomas, typically we export JUJU_HOME which has the environments.yaml with our CI creds in it, and the tests go from there | 14:54 |
| balloons | that means we don't have to juju add-credentials everytime | 14:54 |
| tasdomas | balloons, so that is not the old-style environments.yaml that juju used to use | 14:54 |
| balloons | tasdomas, no.. it's a weird combination of juju1 and juju2 | 14:55 |
| balloons | tasdomas, we have a plan to eliminate all the juju 1ism's from the test framework, but the work is only about halfway done | 14:55 |
| balloons | the use of JUJU_HOME and environments.yaml is some of the last holdouts | 14:56 |
| balloons | tasdomas, if you like I can run your test changes through jenkins manually to ensure it passes | 15:44 |
| tasdomas | balloons, is there a way for me to do that myself? | 15:44 |
| === frankban is now known as frankban|afk | ||
| === hml_ is now known as hml | ||
| axw | thumper: with the "too many open files" thing, forget what I said about state objects being open during upgrade. that's still a problem for upgrades from 2.1, but 2.2.8 to 2.2.9 wouldn't have run any upgrade steps | 23:22 |
| thumper | right | 23:22 |
| === agprado is now known as agprado|afk | ||
| === agprado|afk is now known as agprado | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!