| === davdunc_ is now known as davdunc | ||
| === BlackDex_ is now known as BlackDex | ||
| BlackDex | hello there, i have a juju that has gone out of controll | 11:31 |
|---|---|---|
| BlackDex | it's cpu load i 6 and counting | 11:31 |
| BlackDex | 300% cpu usage | 11:31 |
| BlackDex | wich is all cpu's | 11:31 |
| BlackDex | how can i check what is the issue? | 11:32 |
| BlackDex | I tried restarting the bootstrap node, but that didn't resovled the issue | 11:33 |
| jam | BlackDex: do you see anything informative in either "juju debug-log" or by ssh'ing to the bootstrap node and tailing /var/log/juju/machine-0.log ? | 12:05 |
| BlackDex | not someting i can figure out what is wrong | 12:06 |
| BlackDex | the juju debug-log is not responding | 12:06 |
| BlackDex | jam: http://paste.openstack.org/show/yw0NXzEMEyXW8rjuZiCu/ | 12:09 |
| jam | BlackDex: something like "fortress operation aborted" makes me think something internally is failing, causing lots of things to restart, but I think we're missing the root failure in your traceback. | 12:10 |
| jam | you could try changing the logging config, with: juju model-config -m controller logging-config="<root>=DEBUG;juju.apiserver=INFO" | 12:10 |
| jam | which might make it a bit less noisy | 12:10 |
| BlackDex | watcher loop failed: read tcp 127.0.0.1:41456->127.0.0.1:37017: i/o timeout | 12:10 |
| BlackDex | oke | 12:11 |
| BlackDex | one moments | 12:11 |
| jam | BlackDex: so i/o timeout to 37017 is saying that Mongo stopped talking to us | 12:11 |
| jam | are you running in HA, or just a single bootstrap node? | 12:11 |
| BlackDex | just a single node | 12:11 |
| jam | BlackDex: so the other thing to consider is juju version (should be in the top of "juju status -m controller") | 12:13 |
| BlackDex | 2.2.4 | 12:14 |
| BlackDex | it states an upgrade is available | 12:14 |
| jam | BlackDex: I know there were several load-related fixes in 2.2.9 | 12:14 |
| BlackDex | ah | 12:14 |
| jam | and even more for 2.3.* but you may want to go 2.2.9 first | 12:14 |
| BlackDex | but i can't install that version as a client | 12:14 |
| BlackDex | is that a problem? | 12:14 |
| jam | BlackDex: you should be able to "juju upgrade-juju -m controller --agent-version=2.2.9" even if your client is older. | 12:15 |
| BlackDex | normally i install the updated client first and then the server | 12:15 |
| BlackDex | oke | 12:15 |
| BlackDex | cool | 12:15 |
| BlackDex | lets see what that does | 12:15 |
| jam | BlackDex: one caveat is if you're uploading the agent version from the client, rather than downloading it from streams.canonical.com, but that would depend on whether you have internet access, etc. | 12:16 |
| jam | Sorry I need to go for a bit, will check back in later | 12:16 |
| BlackDex | npt | 12:16 |
| BlackDex | np, thx | 12:16 |
| BlackDex | this is a good start | 12:16 |
| rick_h | magicaltrout: ping, was thinking of showing the updated ssl terminitation charm as a juju show demo Wed and wanted to see if that's ok | 13:07 |
| magicaltrout | rick_h: not mine, do what you like :P | 13:08 |
| rick_h | oh bah | 13:08 |
| rick_h | why do I always confuse you folks in irc? | 13:08 |
| rick_h | need a real nickname...like "rick" :P | 13:08 |
| magicaltrout | i'm not named after a harry potter movie | 13:09 |
| rick_h | how goes magicaltrout? having fun? | 13:09 |
| rick_h | magicaltrout: no, just a fish in the zelda game :P like voltfin trout and such | 13:09 |
| magicaltrout | busy away, getting some bundle stuff cleared away for our store page the ux folks are working on | 13:09 |
| rick_h | magicaltrout: yea, was working with them last week at the sprint. Fun times | 13:09 |
| magicaltrout | along with 2 new contracts on random stuff | 13:09 |
| magicaltrout | and not being paid since December because of a screw up from the US govt. | 13:10 |
| magicaltrout | you know | 13:10 |
| magicaltrout | the usual ;) | 13:10 |
| rick_h | ouch | 13:10 |
| rick_h | anyone home in that us govt? :P | 13:10 |
| magicaltrout | they tend to like shutting down then leaving the contractors to keep stuff working, then feel like not paying them for doing so :P | 13:11 |
| magicaltrout | have just kicked off an interesting project on cancer early detection with a hospital in upstate NY | 13:11 |
| magicaltrout | thats some interesting stuff | 13:11 |
| rick_h | magicaltrout: https://youtu.be/UnPI4xKvf0c?t=449 | 13:12 |
| rick_h | magicaltrout: that's cool | 13:12 |
| magicaltrout | https://cancer.jpl.nasa.gov/ a spin off from that | 13:12 |
| rick_h | oh man, what was that 80s flick where they take a space ship into a body | 13:13 |
| magicaltrout | ha, it is true, trump doesn't like giving jobs to people | 13:13 |
| magicaltrout | when all he does it tout employment ;) | 13:13 |
| zeestrat | Anyone got some tips on how to further troubleshoot https://bugs.launchpad.net/juju/+bug/1755155? | 13:36 |
| mup | Bug #1755155: charm hook failures after controller model upgrade from 2.1.2 to 2.3.4 <juju:New> <https://launchpad.net/bugs/1755155> | 13:36 |
| zeestrat | No fun getting stuck after only upgrading the controller model to 2.3.4 when I want to try out the new features! | 13:40 |
| BlackDex | jam: It seems that upgrading to juju v2.2.9 did the trick mostly. After upgrading the agents also to 2.2.9 it seemed to settle a little bit. | 14:23 |
| vern | does anyone know how I can make juju run an arbitrary command on the controller as part of the bootstrap process? I need to configure the host with a new ca cert so that it can talk to some ssl urls that have certs signed by a company certificate authority | 23:38 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!