| rick_h | axw: howdy | 01:16 |
|---|---|---|
| axw | rick_h: heya! | 01:16 |
| rick_h | axw: question for you, I've gotten your prometheus work going on a controller and I'm pairing it with my repeated/parallel deploy thing I'm hacking on | 01:17 |
| rick_h | axw: and I'm looking to put together a kind of "ootb stuff to watch" setup and curious if you have feedback on the most important/basic items to track out of the available prometheus data points | 01:17 |
| rick_h | axw: especially stuff that might cause things to go boom? or things to keep an eye on? | 01:18 |
| * axw thinking | 01:19 | |
| rick_h | axw: all good, and maybe I should do this over email and such | 01:19 |
| rick_h | axw: but was sitting here and figured I'd try to catch you | 01:19 |
| axw | rick_h: I guess I don't really know, because if I knew what to look for we wouldn't have the problem :) I'd probably focus on mongo-related metrics to start | 01:20 |
| axw | rick_h: so you might want to set up the mongodb exporter on your controller as well, if you haven't already | 01:21 |
| axw | rick_h: we do show the txn ops executed by juju, but that doesn't tell you about growth of mem/cpu etc. in mongo | 01:21 |
| rick_h | axw: no, haven't set that up. Didn't run across that in any of the emails/notes/docs | 01:21 |
| rick_h | axw: is that baked in or something extra to install/setup? | 01:21 |
| axw | rick_h: it's a separate thing altogether | 01:22 |
| axw | rick_h: https://github.com/dcu/mongodb_exporter | 01:22 |
| rick_h | axw: ok cool ty. I'll take a look at that | 01:23 |
| axw | rick_h: another thing that would be helpful is periodic CPU and memory profiles | 01:25 |
| axw | rick_h: have you done that before? | 01:26 |
| rick_h | axw: yea, in my first test I setup munin and used that to generate graphs across the testing time. https://goo.gl/photos/7Cj1FuSRgQeAt79Y8 | 01:26 |
| axw | rick_h: sorry, I mean pprof profiles | 01:27 |
| rick_h | axw: but not done that with prometheus. This was my first foray into it | 01:27 |
| axw | rick_h: pprof will give us source-code level information | 01:27 |
| rick_h | oic, pprof no, but recall some notes/docs around those in the past | 01:27 |
| * rick_h has to go for boy reading time | 01:27 | |
| rick_h | ty axw, lots to think over there | 01:27 |
| axw | rick_h: on the controller machine, just use the command "juju-heap-profile" to get a heap profile | 01:27 |
| axw | rick_h: periodically. that'll be good enough to start with | 01:28 |
| axw | rick_h: enjoy :) | 01:28 |
| wallyworld_ | menn0: have you seen an issue where after running upa juju model with squid-deb-proxy in place, the squid process sits on 6-10% CPU and you need to restart squid-deb-proxy to reset it? happens everytime for me | 03:38 |
| menn0 | wallyworld_: no I haven't seen that and I use squid-deb-proxy | 03:39 |
| menn0 | wallyworld_: have you checked the squid-deb-proxy logs? | 03:39 |
| wallyworld_ | nothing obvious | 03:39 |
| wallyworld_ | menn0: do you use an apt proxy at all then? | 03:40 |
| wallyworld_ | oh | 03:40 |
| wallyworld_ | never mind | 03:40 |
| wallyworld_ | i see your answer | 03:40 |
| menn0 | wallyworld_: strace of the squid process? | 03:41 |
| menn0 | something is obviously keeping it busy | 03:41 |
| wallyworld_ | yeah, i'll look into it | 03:42 |
| wallyworld_ | after another deployment blows it up | 03:42 |
| anastasiamac | wallyworld_: the work to bring in new aws regions on 2.1, did it include brining in new instance types as sizes too? do we pull it from AWS file or have our "copy" still? | 03:51 |
| wallyworld_ | we doesnload the aws json file and process it | 03:51 |
| wallyworld_ | we should have used the latest info | 03:51 |
| wallyworld_ | at the time | 03:51 |
| wallyworld_ | are there any instances types that are missing? | 03:52 |
| anastasiamac | wallyworld_: https://bugs.launchpad.net/juju/+bug/1668307 | 03:52 |
| mup | Bug #1668307: support latest AWS instance sizes and types <juju:Incomplete> <https://launchpad.net/bugs/1668307> | 03:52 |
| anastasiamac | wallyworld_: just checking what to say | 03:53 |
| anastasiamac | wallyworld_: thnx | 03:53 |
| anastasiamac | wallyworld_: and since u've already answered, ty x2 | 03:53 |
| wallyworld_ | anastasiamac: i don't see that we support i3 as he wants | 03:55 |
| wallyworld_ | maybe those were a very recent additon to aws, would need to check | 03:55 |
| wallyworld_ | yeah, 4 days ago | 03:56 |
| wallyworld_ | according to google | 03:56 |
| wallyworld_ | we should aim to fix that for 2.1.1 | 03:56 |
| anastasiamac | wallyworld_: u blow my mind \o/ I should not ask u anything at release call :D must b lack of coffee - ur answer back then was 2.2 | 03:59 |
| anastasiamac | wallyworld_: if u can take it on for 2.1.1, feel free to update the bug and milestone plz | 03:59 |
| wallyworld_ | when? | 03:59 |
| wallyworld_ | today? | 03:59 |
| anastasiamac | wallyworld_: this morning. it's even in the minutes! :D | 03:59 |
| wallyworld_ | lol, i don't even recall | 04:00 |
| anastasiamac | wallyworld_: yep, m blaming it on coffee :D | 04:00 |
| wallyworld_ | did we really discuss that? i guess we must have | 04:01 |
| axw | jam: I am here, but need to eat, so will not be at the standup | 04:50 |
| jam | axw: how goes? | 06:38 |
| jam | You're right that we can create the test, I just have to understand how to plug all the test stuff together. It can give errors so I'll give it a go | 06:38 |
| jam | unfortunately, while most attributes of ServerError are public the actual *error* attribute is private.. | 06:39 |
| axw | jam: sorry was afk. goes ok, still working on tests for the storage regression | 06:59 |
| babbageclunk | wallyworld_: sorry, never got back to you. Finding the deserialization stuff pretty slow going - will hit you up for next tasks tomorrow. | 07:34 |
| wallyworld_ | sgtm | 07:34 |
| jam | axw: ping about checking the error | 08:07 |
| jam | If we're going to check, 404 seems a very generic error | 08:07 |
| axw | jam: it means not found, isn't that what we care about? that the resource does not exist? | 08:08 |
| axw | jam: I'm not *too* fussed, I just feel that looking into the message is a bit brittle. do we have a guarantee that that won't change? I guess it's unlikely to, since it's only for old versions of MAAS | 08:10 |
| jam | so, we may start calling a different api, and an old server would give us a new URL that it doesn't support | 08:10 |
| jam | axw: what about "Unknown API Endpoint ... /static-routes/" ? | 08:14 |
| jam | just check for those two strings? | 08:14 |
| jam | at this point, I don't really care, mostly thinking about "what should our error checking actually look like" | 08:14 |
| jam | and not really wanting to treat 'any error' as the error we think we might get | 08:14 |
| jam | and how specific should we be around that | 08:14 |
| axw | jam: I'm probably being overly optimistic about what URLs we would try and how well services adhere to HTTP status code meanings. "Unknown API Endpoint ... /static-routes/" sounds fine | 08:15 |
| jam | I can hypothetically see us getting a 404 from not having a *specific* route, rather than not implementing the API at all, for example | 08:16 |
| axw | jam: sure, but that should be a different URL though? /static-routes/<foo>, rather than /static-routes/ | 08:17 |
| jam | axw: hence my point of checking the URL rather than just the 404 | 08:18 |
| axw | though to be fair, lack of an endpoint resulting in 404... | 08:18 |
| axw | jam: anyway, I don't really care that much. your proposal is fine | 08:18 |
| jam | axw: sure. it feels like the typo of tech-board weigh in, more from a "how specific should we be about errors from outside" | 08:18 |
| axw | jam: sure, we can discuss tomorrow | 08:19 |
| axw | jam: btw why did the meeting move? | 08:19 |
| axw | bit of a crappy time for you? | 08:19 |
| jam | sorry, I meant to chat with you specifically, but yeah. I talked to tim/ian/menno on Monday | 08:19 |
| axw | jam: no worries, just curious | 08:20 |
| jam | I can't ever make exactly 7:00 cause that's when I'm downstairs taking my son to the bus | 08:20 |
| jam | so 30min later means I won't be late every week | 08:20 |
| * axw nods | 08:20 | |
| === tinwood_swap is now known as tinwood | ||
| rick_h | jam: want to chat today? | 11:35 |
| jam | rick_h: sure, brt | 11:35 |
| === hml_ is now known as hml | ||
| redir | jam: yt? | 17:22 |
| redir | jam: lemme know if you want to chat about kvm at all | 17:22 |
| jam | hi redir.. not really :) | 17:22 |
| jam | sure | 17:22 |
| redir | jam: at your leisure | 17:22 |
| jam | give me a few to post this | 17:23 |
| redir | jam no worries I'm here all day | 17:26 |
| redir | well your night | 17:26 |
| jam | though only just today, right? | 17:26 |
| redir | jam: correct | 17:27 |
| jam | sinzui: https://github.com/juju/juju/pull/7048 tested on finfolk | 17:53 |
| sinzui | thank you jam | 17:54 |
| cholcombe | Is aws bootstrapping having issues currently? https://juju-dist.s3.amazonaws.com/tools/streams/v1/com.ubuntu.juju-released-tools.sjson not found | 18:00 |
| rick_h | cholcombe: aws is having s3 outages in us-east atm | 18:05 |
| rick_h | cholcombe: twitter screams in outrage when us-east is effected by something heh | 18:06 |
| cholcombe | rick_h: ok | 18:07 |
| cholcombe | rick_h: that's odd because i'm trying to bootstrap us-west-2. does it always pull from us-east? | 18:07 |
| rick_h | cholcombe: not sure, just noticed your s3 url and I've just been looking into the outage notes so far | 18:07 |
| cholcombe | ok i'll hold off | 18:08 |
| * redir lunches | 20:00 | |
| babbageclunk | Morning thumper - could you take another look at https://github.com/juju/description/pull/1 | 20:17 |
| thumper | babbageclunk: morning, and yes | 20:17 |
| babbageclunk | thumper: thanks! | 20:17 |
| thumper | babbageclunk: all good | 20:21 |
| babbageclunk | thumper: cool cool, thanks | 20:21 |
| * thumper dives into code for 35 minutes before first call | 20:21 | |
| babbageclunk | thumper: ooh, does $$merge$$ work on that repo? | 20:22 |
| thumper | probably not... | 20:22 |
| thumper | hmm... | 20:22 |
| thumper | babbageclunk: do you have merge access? | 20:23 |
| thumper | babbageclunk: I'm assuming all the tests pass :) | 20:23 |
| thumper | if so, I can just click the merge button | 20:23 |
| babbageclunk | thumper: no merge access - yes, all the tests pass :) | 20:26 |
| thumper | babbageclunk: I've merged it, and I'll look to get automated landing setup | 20:28 |
| babbageclunk | thumper: cheers | 20:29 |
| babbageclunk | wallyworld_: review plz? https://github.com/juju/juju/pull/7049 | 22:13 |
| wallyworld_ | sure | 22:13 |
| babbageclunk | wallyworld_: I'm looking at 7041 now | 22:18 |
| wallyworld_ | ty | 22:19 |
| thumper | what is that github git helper folk use? | 22:19 |
| babbageclunk | thumper hub | 22:26 |
| babbageclunk | thumper: it's annoyingly underdocumented | 22:26 |
| babbageclunk | https://github.com/github/hub | 22:27 |
| thumper | babbageclunk: found it | 22:27 |
| thumper | babbageclunk: how do I set up my creds for it? | 22:27 |
| babbageclunk | thumper: huh, they have a man page for it, but I guess that doesn't get installed when you `go get` it. | 22:29 |
| babbageclunk | https://github.com/github/hub/blob/master/share/man/man1/hub.1.ronn | 22:29 |
| * redir has trouble focusing today | 23:00 | |
| anastasiamac | redir: :( | 23:01 |
| thumper | redir: don't blame you | 23:05 |
| === thumper is now known as thumper-dogwalk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!