[11:31] <BlackDex> hello there, i have a juju that has gone out of controll
[11:31] <BlackDex> it's cpu load i 6 and counting
[11:31] <BlackDex> 300% cpu usage
[11:31] <BlackDex> wich is all cpu's
[11:32] <BlackDex> how can i check what is the issue?
[11:33] <BlackDex> I tried restarting the bootstrap node, but that didn't resovled the issue
[12:05] <jam> BlackDex: do you see anything informative in either "juju debug-log" or by ssh'ing to the bootstrap node and tailing /var/log/juju/machine-0.log ?
[12:06] <BlackDex> not someting i can figure out what is wrong
[12:06] <BlackDex> the juju debug-log is not responding
[12:09] <BlackDex> jam: http://paste.openstack.org/show/yw0NXzEMEyXW8rjuZiCu/
[12:10] <jam> BlackDex: something like "fortress operation aborted" makes me think something internally is failing, causing lots of things to restart, but I think we're missing the root failure in your traceback.
[12:10] <jam> you could try changing the logging config, with: juju model-config -m controller logging-config="<root>=DEBUG;juju.apiserver=INFO"
[12:10] <jam> which might make it a bit less noisy
[12:10] <BlackDex> watcher loop failed: read tcp 127.0.0.1:41456->127.0.0.1:37017: i/o timeout
[12:11] <BlackDex> oke
[12:11] <BlackDex> one moments
[12:11] <jam> BlackDex: so i/o timeout to 37017 is saying that Mongo stopped talking to us
[12:11] <jam> are you running in HA, or just a single bootstrap node?
[12:11] <BlackDex> just a single node
[12:13] <jam> BlackDex: so the other thing to consider is juju version (should be in the top of "juju status -m controller")
[12:14] <BlackDex> 2.2.4
[12:14] <BlackDex> it states an upgrade is available
[12:14] <jam> BlackDex: I know there were several load-related fixes in 2.2.9
[12:14] <BlackDex> ah
[12:14] <jam> and even more for 2.3.* but you may want to go 2.2.9 first
[12:14] <BlackDex> but i can't install that version as a client
[12:14] <BlackDex> is that a problem?
[12:15] <jam> BlackDex: you should be able to "juju upgrade-juju -m controller --agent-version=2.2.9" even if your client is older.
[12:15] <BlackDex> normally i install the updated client first and then the server
[12:15] <BlackDex> oke
[12:15] <BlackDex> cool
[12:15] <BlackDex> lets see what that does
[12:16] <jam> BlackDex: one caveat is if you're uploading the agent version from the client, rather than downloading it from streams.canonical.com, but that would depend on whether you have internet access, etc.
[12:16] <jam> Sorry I need to go for a bit, will check back in later
[12:16] <BlackDex> npt
[12:16] <BlackDex> np, thx
[12:16] <BlackDex> this is a good start
[13:07] <rick_h> magicaltrout: ping, was thinking of showing the updated ssl terminitation charm as a juju show demo Wed and wanted to see if that's ok
[13:08] <magicaltrout> rick_h: not mine, do what you like :P
[13:08] <rick_h> oh bah
[13:08] <rick_h> why do I always confuse you folks in irc?
[13:08] <rick_h> need a real nickname...like "rick" :P
[13:09] <magicaltrout> i'm not named after a harry potter movie
[13:09] <rick_h> how goes magicaltrout? having fun?
[13:09] <rick_h> magicaltrout: no, just a fish in the zelda game :P like voltfin trout and such
[13:09] <magicaltrout> busy away, getting some bundle stuff cleared away for our store page the ux folks are working on
[13:09] <rick_h> magicaltrout: yea, was working with them last week at the sprint. Fun times
[13:09] <magicaltrout> along with 2 new contracts on random stuff
[13:10] <magicaltrout> and not being paid since December because of a screw up from the US govt.
[13:10] <magicaltrout> you know
[13:10] <magicaltrout> the usual ;)
[13:10] <rick_h> ouch
[13:10] <rick_h> anyone home in that us govt? :P
[13:11] <magicaltrout> they tend to like shutting down then leaving the contractors to keep stuff working, then feel like not paying them for doing so :P
[13:11] <magicaltrout> have just kicked off an interesting project on cancer early detection with a hospital in upstate NY
[13:11] <magicaltrout> thats some interesting stuff
[13:12] <rick_h> magicaltrout: https://youtu.be/UnPI4xKvf0c?t=449
[13:12] <rick_h> magicaltrout: that's cool
[13:12] <magicaltrout> https://cancer.jpl.nasa.gov/ a spin off from that
[13:13] <rick_h> oh man, what was that 80s flick where they take a space ship into a body
[13:13] <magicaltrout> ha, it is true, trump doesn't like giving jobs to people
[13:13] <magicaltrout> when all he does it tout employment ;)
[13:36] <zeestrat> Anyone got some tips on how to further troubleshoot https://bugs.launchpad.net/juju/+bug/1755155?
[13:36] <mup> Bug #1755155: charm hook failures after controller model upgrade from 2.1.2 to 2.3.4 <juju:New> <https://launchpad.net/bugs/1755155>
[13:40] <zeestrat> No fun getting stuck after only upgrading the controller model to 2.3.4 when I want to try out the new features!
[14:23] <BlackDex> jam: It seems that upgrading to juju v2.2.9 did the trick mostly. After upgrading the agents also to 2.2.9 it seemed to settle a little bit.
[23:38] <vern> does anyone know how I can make juju run an arbitrary command on the controller as part of the bootstrap process? I need to configure the host with a new ca cert so that it can talk to some ssl urls that have certs signed by a company certificate authority