=== davdunc_ is now known as davdunc === BlackDex_ is now known as BlackDex [11:31] hello there, i have a juju that has gone out of controll [11:31] it's cpu load i 6 and counting [11:31] 300% cpu usage [11:31] wich is all cpu's [11:32] how can i check what is the issue? [11:33] I tried restarting the bootstrap node, but that didn't resovled the issue [12:05] BlackDex: do you see anything informative in either "juju debug-log" or by ssh'ing to the bootstrap node and tailing /var/log/juju/machine-0.log ? [12:06] not someting i can figure out what is wrong [12:06] the juju debug-log is not responding [12:09] jam: http://paste.openstack.org/show/yw0NXzEMEyXW8rjuZiCu/ [12:10] BlackDex: something like "fortress operation aborted" makes me think something internally is failing, causing lots of things to restart, but I think we're missing the root failure in your traceback. [12:10] you could try changing the logging config, with: juju model-config -m controller logging-config="=DEBUG;juju.apiserver=INFO" [12:10] which might make it a bit less noisy [12:10] watcher loop failed: read tcp 127.0.0.1:41456->127.0.0.1:37017: i/o timeout [12:11] oke [12:11] one moments [12:11] BlackDex: so i/o timeout to 37017 is saying that Mongo stopped talking to us [12:11] are you running in HA, or just a single bootstrap node? [12:11] just a single node [12:13] BlackDex: so the other thing to consider is juju version (should be in the top of "juju status -m controller") [12:14] 2.2.4 [12:14] it states an upgrade is available [12:14] BlackDex: I know there were several load-related fixes in 2.2.9 [12:14] ah [12:14] and even more for 2.3.* but you may want to go 2.2.9 first [12:14] but i can't install that version as a client [12:14] is that a problem? [12:15] BlackDex: you should be able to "juju upgrade-juju -m controller --agent-version=2.2.9" even if your client is older. [12:15] normally i install the updated client first and then the server [12:15] oke [12:15] cool [12:15] lets see what that does [12:16] BlackDex: one caveat is if you're uploading the agent version from the client, rather than downloading it from streams.canonical.com, but that would depend on whether you have internet access, etc. [12:16] Sorry I need to go for a bit, will check back in later [12:16] npt [12:16] np, thx [12:16] this is a good start [13:07] magicaltrout: ping, was thinking of showing the updated ssl terminitation charm as a juju show demo Wed and wanted to see if that's ok [13:08] rick_h: not mine, do what you like :P [13:08] oh bah [13:08] why do I always confuse you folks in irc? [13:08] need a real nickname...like "rick" :P [13:09] i'm not named after a harry potter movie [13:09] how goes magicaltrout? having fun? [13:09] magicaltrout: no, just a fish in the zelda game :P like voltfin trout and such [13:09] busy away, getting some bundle stuff cleared away for our store page the ux folks are working on [13:09] magicaltrout: yea, was working with them last week at the sprint. Fun times [13:09] along with 2 new contracts on random stuff [13:10] and not being paid since December because of a screw up from the US govt. [13:10] you know [13:10] the usual ;) [13:10] ouch [13:10] anyone home in that us govt? :P [13:11] they tend to like shutting down then leaving the contractors to keep stuff working, then feel like not paying them for doing so :P [13:11] have just kicked off an interesting project on cancer early detection with a hospital in upstate NY [13:11] thats some interesting stuff [13:12] magicaltrout: https://youtu.be/UnPI4xKvf0c?t=449 [13:12] magicaltrout: that's cool [13:12] https://cancer.jpl.nasa.gov/ a spin off from that [13:13] oh man, what was that 80s flick where they take a space ship into a body [13:13] ha, it is true, trump doesn't like giving jobs to people [13:13] when all he does it tout employment ;) [13:36] Anyone got some tips on how to further troubleshoot https://bugs.launchpad.net/juju/+bug/1755155? [13:36] Bug #1755155: charm hook failures after controller model upgrade from 2.1.2 to 2.3.4 [13:40] No fun getting stuck after only upgrading the controller model to 2.3.4 when I want to try out the new features! [14:23] jam: It seems that upgrading to juju v2.2.9 did the trick mostly. After upgrading the agents also to 2.2.9 it seemed to settle a little bit. [23:38] does anyone know how I can make juju run an arbitrary command on the controller as part of the bootstrap process? I need to configure the host with a new ca cert so that it can talk to some ssl urls that have certs signed by a company certificate authority