fatgooseanyone knows how to find k8s logs from the juju charm canonical distribution of kubernetes? specifically the controller-manager log. where is it on the master node?00:31
fatgoose(assuming this is the right place to ask such question)00:31
anastasiamacfatgoose: best place for these kind of questions would be #juju rather than #juju-dev00:36
thumperaxw: I noticed an intermittent failure in develop featuretests, to do with upgrade testing: upgrade_test.go:140: c.Assert(waitForUpgradeToStart(upgradeCh), jc.IsTrue), ... obtained bool = false04:17
thumperaxw: I know you love intermittent failures :)04:17
axwthumper: I love them so much. I'll take a look after lunch04:21
thumperaxw: awesome, thanks04:21
wgrantaxw: Do you have a moment to talk about statuseshistory?05:43
axwwgrant: sure05:43
wgrantJan 10 05:43:21 juju-189c77-controller-0 mongod.37017[1098]: [conn96] remove juju.statuseshistory query: { _id: ObjectId('5a53ff76f582e9a9395ed438') } ndeleted:1 keyUpdates:0 writeConflicts:0 numYields:1 locks:{ Global: { acquireCount: { r: 762, w: 762 } }, Database: { acquireCount: { w: 762 } }, Collection: { acquireCount: { w: 393 } }, Metadata: { acquireCount: { w: 369 } }, oplog: { acquireCount: { w:05:43
wgrant369 } } } 120ms05:43
wgrantaxw: Does that mean it deleted only one row at a time?05:43
wgrantI just ran remove-application on 2.2.9 on a unit with lots of statuseshistory, and mongo and juju are now each eating a core and there's a LOT of that in the logs05:44
wgrantAh, got some of this toward the end, which looks more reasonable:05:44
wgrantJan 10 05:43:54 juju-189c77-controller-0 mongod.37017[1098]: [conn49] command juju.$cmd command: delete { delete: "statuseshistory", deletes: 1000, writeConcern: { getLastError: 1, j: true }, ordered: true } keyUpdates:0 writeConflicts:0 numYields:0 reslen:100 locks:{ Global: { acquireCount: { r: 2002, w: 2002 } }, Database: { acquireCount: { w: 2002 } }, Collection: { acquireCount: { w: 1002 } }, Metadata:05:44
wgrant{ acquireCount: { w: 1000 } }, oplog: { acquireCount: { w: 1000 } } } protocol:op_query 1051ms05:44
axwwgrant: the first one does look like that, yes05:45
wgrantaxw: Interestingly, the batching does not totally fix the problem. I've just accidentally reproduced the saslStart storm outside the big controller for the first time...05:52
wgrantI wonder what information I should gather.05:52
axwwgrant: syslog and jujud log for a start, goroutine profile and cpu profile of jujud might also be helpful05:53
wgrantaxw: "perf top" shows that almost all CPU time is spent calculating SHA-1s.05:54
wgrantFor SASL05:54
axwwgrant: the profile might still give a clue as to why05:55
wgrantaxw: How do I do the goroutine thingy?05:55
axw1 sec, need to refresh my own memory05:55
axwwgrant: on the controller, "juju-goroutines"06:00
axwand "juju-cpu-profile" will give you a cpu profile06:00
wgrantaxw: Wow that's easier than I expected06:00
wgrantaxw: https://private-fileshare.canonical.com/~wgrant/juju-prod-ols-snap-store-20180110-saslStart-storm.tar.gz06:07
wgrantI'll add it to the bug as well, just wanting to gather as much as possible before I restart it.06:07
axwwgrant: thanks06:10
axwwgrant: no smoking gun in that data AFAICS :(06:24
wgrantaxw: :(06:30
=== frankban|afk is now known as frankban
=== rogpeppe1 is now known as rogpeppe
tasdomashow do I run juju acceptance tests on my machine?13:22
balloonstasdomas, you should be able to just run most of them14:19
tasdomasballoons, I thought so too14:19
balloonsSome may require arguments but the tests try and be smart in most cases and make default values14:19
balloonstasdomas, I assume you are looking at the assess_budget tests?14:34
tasdomasballoons, yeah14:34
tasdomasballoons, however, I'm getting this when I try to run tests: https://pastebin.ubuntu.com/26360414/14:35
tasdomasballoons, that's with a virtualenv, with requirements.txt installed14:35
balloonstasdomas, well that's simple enough. You are missing python-fixtures looks like14:36
balloonstasdomas, the requirements.txt is a bit of a lie atm, as you also need the packages in juju-ci-common14:36
balloonstasdomas, these packages: https://github.com/juju/juju/blob/develop/acceptancetests/juju-ci-tools-common14:37
tasdomasballoons, thanks - I'll keep looking into this14:39
balloonstasdomas, sorry for the trouble. But thanks for updating the tests as well. If you continue to have trouble let me know14:40
tasdomasballoons, I will14:40
=== frankban is now known as frankban|afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!