| c01nwarr10r[HD] | VOISE, the largest music exchange on the planet in the next few years, based on the latest ethereum smart contract technolgy, VOISE offers a free decentralized market for artists to promote and sell 100% of their work on their own terms | 02:28 |
|---|---|---|
| c01nwarr10r[HD] | going to be a huge market | 02:28 |
| c01nwarr10r[HD] | anyone and everyone can participate, even the transactions are processed by anyone and everyone as it's decentralized | 02:28 |
| c01nwarr10r[HD] | has all the right ingredients to take over | 02:28 |
| c01nwarr10r[HD] | it's not going to look like the largest music exchange in the world early on, it's going to look like what VOISE looks like now, with the potential to be the largest music exchange in the world | 02:29 |
| c01nwarr10r[HD] | it would be to easy otherwise | 02:29 |
| c01nwarr10r[HD] | but if you'd prefer to play it safe, no big deal you can always see the glory, just don't expect to be the glory | 02:30 |
| c01nwarr10r[HD] | livecoin.net | 02:30 |
| === frankban|afk is now known as frankban | ||
| === markthomas_ is now known as markthomas | ||
| === hazmat_ is now known as hazmat | ||
| === frankban is now known as frankban|afk | ||
| === frankban|afk is now known as frankban | ||
| andreas_s | Hi, does anybody know if it's possible to have a single node lxd cloud and then to add the localhost as bare metal machine via ssh? | 14:15 |
| andreas_s | I tried it, the add-machine command succeeds, but the agent on the localhost (the lxd host) is stuck in pending | 14:16 |
| andreas_s | seems like it does not have the api ip set for t | 14:16 |
| andreas_s | some reason | 14:16 |
| === mhall119_ is now known as mhall119 | ||
| fallenour | o/ | 14:55 |
| fallenour | got an issue thats pretty weird, haproxy working fine, giving UP status, /var/log/haproxy.log showing things are good, initial landing page loads just fine, generic http currently, go to log in, processes request, returns with 502 bad gateway error. Only recent changes from last known good was updates to pike from ocata, openstack-charmers model | 14:56 |
| rick_h | bdx: https://photos.app.goo.gl/WAD6pZtkjt74AqsP2 cmr telegraf -> prometheus with network_get finally working | 14:56 |
| fallenour | any thoughts rick_h bdx jamespage | 14:57 |
| rick_h | fallenour: nothing in the haproxy logs about the 502? what's it hitting behind haproxy? which api endpoints? | 14:57 |
| bdx | rick_h: no way!!! I was just going to ping you and see how that was going | 14:57 |
| fallenour | rick_h: no, thats the crazy thing, /var/log/haproxy.log shows everything fine, and that it can reach the openstack dashboard no issues. | 14:58 |
| fallenour | rick_h: I checked all three haproxy instances just to ensure I was crazy. | 14:58 |
| fallenour | rick_h: also, keystone and mysql systems both show as fine. | 14:59 |
| Fallenour | Sorry rick_h what did I miss? Am afk but still listening | 15:16 |
| Fallenour | . | 15:27 |
| Fallenour | . | 15:35 |
| bdx | fallenour: its hard for anyone to say without diving into your setup | 15:43 |
| === frankban is now known as frankban|afk | ||
| === catbus is now known as catbus-afk | ||
| fallenour | o/ | 20:38 |
| fallenour | having a weird issue with juju xenial openstack model, I cant log in after applying pike updates. HAproxy loads landing dashboard login, but I cant log in after submitting creds, getting a 502 bad gateway error. Chceked haproxy, haproxy is loading portal on its side properly. | 20:39 |
| magicaltrout | fallenour: I can't help, but more importantantly do you own a kettle? | 20:46 |
| fallenour | magicaltrout: actually I do, but alas, I too, cannot help. I have no teleporter in which to send my kettle :9 | 20:49 |
| rick_h | magicaltrout: but does it need to be electric or stovetop? | 20:50 |
| magicaltrout | rick_h: you go camping | 20:50 |
| magicaltrout | we all know you own a kettle | 20:50 |
| rick_h | magicaltrout: yea, electric in the house but stovetop in the camper so you can heat it over a fire/gas grill | 20:50 |
| rick_h | only way to make pour over coffee while camping :) | 20:50 |
| fallenour | is there a way to test functionality of keystone? | 20:50 |
| magicaltrout | i do love a stovetop kettle, sadly i have electic hobs which suck | 20:51 |
| magicaltrout | you tried summoning any of the openstackers fallenour ? | 20:51 |
| rick_h | fallenour: sorry, will need a more openstack expert than myself on diagnosing the openstack bits | 20:51 |
| fallenour | magicaltrout: If I only had the spellpoints my good friend, if only. I am but a meager lvl 14 wizard. Openstackers require Summon Monster lvl 7 :( | 20:52 |
| magicaltrout | try prodding blindly in #openstack-charms | 20:53 |
| magicaltrout | see if anyone is alive | 20:53 |
| magicaltrout | or thedac he was complaining about the time difference when i was sat in the openstack room | 20:54 |
| magicaltrout | so logic dictates he's working somewhere | 20:54 |
| * thedac waves | 20:54 | |
| fallenour | thedac: 8D | 20:54 |
| fallenour | any ideas my friend? | 20:54 |
| thedac | reading backscroll now | 20:55 |
| magicaltrout | don't trust thedac though | 20:55 |
| magicaltrout | he doesn't own a kettle | 20:55 |
| thedac | :) | 20:55 |
| fallenour | Ill build a CA then, as I can be implicitly trusted, as I own a kettle | 20:55 |
| magicaltrout | there you go | 20:56 |
| fallenour | Im sure Mozilla will accept this sound logic | 20:56 |
| magicaltrout | damn right | 20:56 |
| fallenour | thedac: The gist of the current situation, prior to updating the xenial-charmers openstack model, system works fine. I update to pike packages, horizon loads dashboard login screen ,but after login, I get a 502 bad gateway error. I chcekd Haproxy systems, all say UP. I chcek keystone, keystone says up. | 20:58 |
| thedac | fallenour: oh, did you manually install the packages or did you let the charms update to pike? | 20:58 |
| fallenour | thedac: I used juju to run the update command on the system | 20:58 |
| thedac | ok | 20:59 |
| fallenour | juju run --unit <all the units 1 by 1> 'sudo apt-get update' | 20:59 |
| fallenour | thedac: do I need to update the model? | 20:59 |
| thedac | ah, ok, so that is manual | 20:59 |
| fallenour | thedac: screwed the pooch I did, didnt I? | 20:59 |
| thedac | fallenour: so, the charms to a lot of work to make sure the whole stack works on upgrade. | 20:59 |
| thedac | The charm is looking at the openstack-origin config value and will get very confused if that does not match what packages are installed | 21:01 |
| thedac | So the way to upgrade is to change that value. | 21:01 |
| thedac | One sec let me get you some docs | 21:01 |
| fallenour | thedac: ahh I see, so future lesson, update model? not packages manually? | 21:01 |
| thedac | Exactly | 21:01 |
| fallenour | ooh shit, I pulled form teh wrong repos didnt I? | 21:02 |
| thedac | fallenour: See https://jujucharms.com/keystone/269#charm-config-openstack-origin and https://jujucharms.com/keystone/269#charm-config-action-managed-upgrade | 21:03 |
| thedac | fallenour: the problem is any future hook that runs will get conflicting information about what release of OpenStack is in play | 21:04 |
| fallenour | thedac: so I guess my question is, what is my course of action to fix the issue? I have a feelign its a juju config command? | 21:06 |
| fallenour | thedac: right now it reads as cloud:xenial-pike | 21:07 |
| thedac | Oh, well that is correct | 21:07 |
| fallenour | thedac: do I need to confirm that for every system in question? | 21:07 |
| thedac | For each charm 'juju config $CHARM openstack-origin=cloud:xenial-pike' | 21:08 |
| fallenour | thedac: openstack-dashboard (horizon) also reads as cloud:xenial-pike | 21:08 |
| thedac | You can check it by not adding the =cloud:xenial-pike | 21:08 |
| fallenour | thedac: im not, im simply typing in juju config <application> openstack-origin and pressing [ENTER] | 21:09 |
| thedac | ok, yes, that shows what it currently set to. | 21:09 |
| fallenour | thedac: do you think it might be mysql that is the issue by chance? | 21:10 |
| fallenour | I dont think it is, but at this point, im at a loss. | 21:10 |
| thedac | fallenour: I guess I need to know what steps you took and in what order to attempt to help | 21:10 |
| thedac | at some point you did set openstack-origin was that before or after running apt commands? | 21:11 |
| fallenour | thedac: no, I just ran the update commands. Im actually interested myself how its set to pike myself. I built it ocata initially | 21:11 |
| thedac | hmm, so, I suspect this will take a fair amount of manual restaring of services. It is very difficult to know what state things are in | 21:14 |
| fallenour | thedac: Do I need to restart every service manually? | 21:15 |
| thedac | that is only a guess. What you might want to do is check each service one by one from cli. | 21:16 |
| fallenour | thedac: what should I check for? | 21:16 |
| fallenour | thedac: and shoudl I reset in any particular order? | 21:16 |
| === catbus-afk is now known as catbus | ||
| thedac | fallenour: I would start by checking the services. openstack catalog list, openstack image list, openstack server list. etc and see if any of those fail | 21:17 |
| fallenour | thedac: um...odd question, but where would I run that? Normally when im working with openstack id run that on a controller, a nova system, or a neutron box, but I dont have openstackpythonclient installed anywhere. Im assuming ill need to install that first and grab the adminrc file? | 21:20 |
| thedac | fallenour: yes, you'll need the openstack client, and adminrc file and a host that has access to all the APIs | 21:21 |
| fallenour | thedac: ok, next challenge. This has actually been an issue for me for a while. How do I get my ssh key added to the systems? ive tried juju add-ssh-key with no success | 21:26 |
| thedac | fallenour: I don't think there is an automated way. You can juju ssh into everything and manually add an ssh authorized key | 21:27 |
| fallenour | thedac: I try: juju add-ssh-key admin/conjure-openstack-base-29c /dir/to/id_rsa.pub ; but all I get is a invalid ssh key error | 21:28 |
| fallenour | thedac: cant do that. I already tried to juju ssh , but i get a permission denied (public key) error when I try the juju ssh command | 21:28 |
| thedac | I am afraid I have never used add-ssh-key command. Any other juju peeps want to run with that? | 21:30 |
| fallenour | thedac: what user do you normally log in with? | 21:30 |
| fallenour | thedac: I tried the generic "juju" and "ubuntu" as well, since I use MAAS, neither work | 21:30 |
| thedac | Usually juju ssh $APP/$UNIT logs in as the ubuntu user | 21:31 |
| fallenour | thedac: ideas on using scp to get the key there, then cat it to authorized file? | 21:32 |
| thedac | I am looking on how to use the add-ssh-key now. One sec | 21:32 |
| thedac | looks like you need to send it as a string: juju add-ssh-key "$(cat ~/mykey.pub)" | 21:33 |
| fallenour | wuh? | 21:35 |
| fallenour | hang on, I just tried that | 21:35 |
| thedac | I just confirmed that works | 21:35 |
| thedac | You can list the ssh keys with: juju ssh-keys | 21:35 |
| fallenour | thedac: now im getting error : "duplicate key" when I use the -m option >.> | 21:36 |
| fallenour | thedac: I think it hates me Q___Q | 21:37 |
| thedac | That would imply your key is already there | 21:37 |
| fallenour | thedac: it says it has my key listed, but when I try to ssh, it says denied | 21:37 |
| thedac | fallenour: are you doing "juju ssh $APP/$UNIT" or trying to ssh directly? | 21:38 |
| fallenour | thedac: tried: juju ssh <$app/$unit> as well as: juju ssh <username>@<$app/$unit> , permission denied both times | 21:39 |
| thedac | fallenour: and this is on the host you performed the deploy? | 21:39 |
| fallenour | thedac: yeap. im on my juju controller now | 21:39 |
| thedac | hold on :) | 21:40 |
| fallenour | thedac: just confirmed the key in my /home/<username>/.ssh/id_rsa.pub was also the sam....permissions o.o | 21:40 |
| fallenour | thedac: permissios seem fine o.o | 21:41 |
| thedac | If you run 'juju status' where you are now does that work? | 21:41 |
| fallenour | thedac: yeap | 21:41 |
| thedac | I am very confused why you are not able to juju ssh then | 21:42 |
| fallenour | thedac: I know right?! | 21:42 |
| fallenour | thedac: item of interest; according to juju run --unit openstack-dashboard/0 'ls -lisa /home/<username>/.ssh' the directory doesnt exist for me | 21:43 |
| fallenour | thedac: but "ubuntu" does exist | 21:44 |
| thedac | right, it is not going to create a user for you. | 21:44 |
| thedac | juju ssh is going to log in as ubunut | 21:44 |
| thedac | ubuntu even | 21:44 |
| thedac | fallenour: so juju run works? | 21:44 |
| fallenour | thedac: I think I found the issue fo rthe ubuntu user as well, its only got 600 perms on the authorized_keys file | 21:45 |
| fallenour | thedac: yea | 21:45 |
| fallenour | thedac: gonna try to change the perms on ubuntu user ssh key, then try to log in as it | 21:45 |
| thedac | I think that is pretty normal | 21:45 |
| thedac | confirmed 600 is ok | 21:45 |
| fallenour | thedac: not to the best of my knowledge. if I understand correctly, you need it to have 644 perms, not 600. It would mean only the user, not the system, could read the file, making it unable to auth your key? | 21:46 |
| thedac | I am just saying that is what all mine are set to and I can juju ssh into them fine | 21:46 |
| fallenour | thedac: huh...thats never worked for me, but then again, im not a linux god, and shall not argue XD | 21:46 |
| fallenour | thedac: o.O | 21:47 |
| fallenour | thedac: it works with ubuntu o.O | 21:47 |
| thedac | ok | 21:47 |
| fallenour | thedac: I feel so lawst x.x | 21:47 |
| thedac | fallenour: do you have an .ssh/config that could be getting in the way? Mabye setting user or something | 21:48 |
| fallenour | thedac: not that im aware of. note: do NOT OPEN, the /var/log/juju/mysql files, not the droids you are looking for | 21:50 |
| fallenour | thedac: mysql error logs look clean, successfully syncing with group | 21:51 |
| fallenour | thedac: what systems aside from Haproxy, keystone, and horizon are used during login on the dashboard? | 21:52 |
| thedac | That is basically it. | 21:52 |
| fallenour | thedac: then what could the issue be? and where at? | 21:53 |
| thedac | Although it is going to make a bunch of api calls after that | 21:53 |
| thedac | so again, I would verify each api one at a time. Start with 'openstack catalog list' which will validate keystone | 21:53 |
| fallenour | thedac: yea, checking that now | 21:54 |
| fallenour | thedac: also, I found that the systems do a daily apt download, lesson learned the hardway :( | 21:54 |
| fallenour | thedac: idea, how to scp a file from a remote system to your system? | 22:02 |
| thedac | Do you have keys setup on the remote system? | 22:03 |
| fallenour | YAHTZEEE | 22:05 |
| fallenour | thedac: nope, password. always a saving grace, at least one password based system | 22:05 |
| fallenour | scp <username>@<ip address>:/path/to/target/file . | 22:06 |
| fallenour | worked like a charm | 22:06 |
| fallenour | thedac: /sigh | 22:08 |
| fallenour | thedac: now its telling me auth url error, even though in the admin rc file its included >.> | 22:08 |
| thedac | ok, and the correct creds for an admin user on the cloud? | 22:09 |
| fallenour | thedac: yea, keystone is still on v2 right? | 22:09 |
| fallenour | and its still OS_AUTH_URL= right? | 22:10 |
| thedac | Depends on how you configure it. juju config preferred-api-version | 22:10 |
| thedac | juju config keystone preferred-api-version | 22:10 |
| fallenour | thedac: yea, its 2 | 22:11 |
| thedac | ok | 22:11 |
| fallenour | thedac: the port wrong maybe? | 22:12 |
| thedac | what is the error exactly? | 22:13 |
| thedac | and RC looks similar to this? http://pastebin.ubuntu.com/25768571/ | 22:13 |
| fallenour | thedac: whooo, just spat a lot of angry at me, gimme a moment | 22:14 |
| fallenour | thedac: is openstack-dashboard a bad idea place to be running this from? maybe nova instead? | 22:14 |
| thedac | I would not install other packages on those units if you can avoid it. What about the machine where you ran juju commands. That is a natural fit. | 22:15 |
| fallenour | thedac: yea thats my hope as well | 22:16 |
| fallenour | thedac: Ive already also made that system my saltmaster. | 22:17 |
| fallenour | thedac: hold the pony! it just kicked me out of my container, and dropped me onto the machine? Did this seriously just do a container escape? | 22:18 |
| fallenour | thedac: holy shit batman, it sure did! | 22:19 |
| fallenour | thedac: bug issue for later. Current question, im getting python output issues, ideas? | 22:21 |
| thedac | We are all over the place. I have no idea where you are at at this point :) | 22:22 |
| fallenour | thedac: moved back to controller, side lined the container escape issue for later, moved the admin rc file to the controller, chmod +x on the file to make it executable, now getting same python traceback output when I run: . admin-openrc.sh && openstack catalog list | 22:23 |
| fallenour | thedac: juju controller, specifically | 22:23 |
| === med_ is now known as Guest99060 | ||
| thedac | what does the traceback look like? can you pastebin | 22:23 |
| fallenour | thedac: yeap, un momento | 22:24 |
| fallenour | thedac: https://paste.ngx.cc/ba | 22:25 |
| thedac | try 'env |grep OS_' | 22:26 |
| thedac | I don't think your RC is exporting | 22:26 |
| thedac | again, it should look something like http://pastebin.ubuntu.com/25768648/ | 22:27 |
| fallenour | thedac: it gave me a lot of OS_ info found in my admin openrc file | 22:27 |
| thedac | Then 'source RCFILE' | 22:27 |
| thedac | ok | 22:27 |
| fallenour | thedac: it took the source command | 22:27 |
| fallenour | thedac: same issue on openstack catalog list | 22:28 |
| fallenour | thedac: do you think there is an issue with keystone specifically? | 22:28 |
| fallenour | are there local commands we can try on keystone directly? | 22:29 |
| thedac | I don't think so. | 22:29 |
| fallenour | thedac: just an fyi, keystone/0 is refusing ssh connections | 22:29 |
| thedac | ? | 22:29 |
| thedac | How so? | 22:29 |
| fallenour | thedac: yea, getting a "cannot connect to any address: <keystone IP:22> | 22:30 |
| thedac | ok, let's take another step back. Can you pastebin 'juju status' for me | 22:30 |
| fallenour | thedac: sure | 22:31 |
| fallenour | thedac: https://paste.ngx.cc/11 | 22:32 |
| fallenour | thedac: just an fyi, ignore apache and nfs servers, currently experimenting with those. As for haproxy, ive directly checked each of them individually, according to all of them, they are working properly. | 22:33 |
| fallenour | more specifically, they are getting the openstack-dashboard-70, with UP status, on restart, and as current as last restart less than an hour ago. | 22:33 |
| thedac | so haproxy as an external charm is not a regular part of our deploy. The api charms themselves install haproxy (i.e. cinder) to use internally | 22:34 |
| fallenour | thedac: yea, I added an external one. its pointed at the current haproxy systems. | 22:34 |
| thedac | What address are you using to access horizon? | 22:34 |
| fallenour | horizon.eduarmor.com/horizon | 22:34 |
| thedac | right but what IP does that resolve to? an haproxy or openstack-dashboard? | 22:35 |
| fallenour | thedac: it resolves to openstack-dashboard | 22:35 |
| thedac | Have you tried going directly to the horizon ip? 10.0.0.51 from the pastebin | 22:35 |
| fallenour | thedac: on login though, its the same issue as with DNS, 502 bad gateway | 22:35 |
| thedac | ok, can you check the apache logs on the openstack-dashborad unit? | 22:37 |
| fallenour | thedac: sure. also, idea, odd one though. wget the api path? on keystone? | 22:37 |
| fallenour | thedac: oo | 22:37 |
| thedac | Sure, you will get an unathorized but that will tell us it is up | 22:37 |
| fallenour | thedac: actually, I take that last back, I got a slightly different error this time. | 22:38 |
| fallenour | thedac: actually, I take that last back, I got a slightly different error this time. "this page isnt working ; 10.0.0.51 didnt send any data" on chrome , "Connection reset" on firefox | 22:38 |
| fallenour | thedac: checking apache now | 22:38 |
| thedac | oh, from the browser that will be less useful. You also need to specify the port 5000 | 22:39 |
| fallenour | thedac: ok this is weird. check this out: 10.0.0.51 - - [18/Oct/2017:22:35:21 +0000] "POST /horizon/auth/login/ HTTP/1.1" 200 0 "http://10.0.0.51/horizon/auth/login/?next=/horizon/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0" 10.0.0.51 - - [18/Oct/2017:22:37:31 +0000] "GET / HTTP/1.0" 200 946 "-" "-" 10.0.0.51 - - [18/Oct/2017:22:37:34 +0000] "GET / HTTP/1.0" 200 946 "-" "-" | 22:39 |
| fallenour | thedac: https://paste.ngx.cc/b3 | 22:40 |
| thedac | hmm, that looks fairly normal | 22:41 |
| fallenour | thedac: except that thats my traffic right now | 22:41 |
| fallenour | thedac: with the connection reset error | 22:41 |
| fallenour | thedac: how can it be a connection reset error and a http 200 ok code? | 22:42 |
| thedac | The time stamp matches up with when you were trying to connect? | 22:42 |
| fallenour | thedac: looks to be the case | 22:44 |
| fallenour | thedac: yea, exactly. I just changed the page I was aiming at, matches with the logs, loading a http 200 946 "-" "-" | 22:45 |
| fallenour | thedac: in the logs. Im so confused. | 22:45 |
| thedac | But you get a connnection rest in firefox? | 22:45 |
| fallenour | thedac: because according to this, it loaded the entire page just fine | 22:45 |
| fallenour | thedac: yea. this is nuts | 22:46 |
| thedac | Let me ask some more questions. Before you upgrading things this was all working as is? | 22:46 |
| fallenour | thedac: yea, it was working perfectly | 22:46 |
| thedac | I have not used haproxy in front of openstack-dashboard. So that is the only thing I have some doubts about. | 22:48 |
| thedac | It *sounds* like you are hitting the haproxy and it doesn't think any back ends are valid | 22:48 |
| thedac | or something like that | 22:48 |
| fallenour | thedac: I have an idea, and it looks like you are on th esame page as me. I think it might be the haproxy to haproxy system, although it wasnt an issue before, and im curious if that cna be the case, specifically because I current....the router o.o | 22:49 |
| fallenour | thedac: its gonna go to the switch, on the same lan, or the router, which is the gateway, to load the horizon page? | 22:50 |
| fallenour | 10.0.0.0/24 network | 22:50 |
| fallenour | thedac: im thinking it might be dropping my connection witha connection reset on the 2nd haproxy system on the return, what are your thoughts? | 22:50 |
| fallenour | thedac: OOO JACKPOT! | 22:51 |
| thedac | my thought is let's remove complexity and try to go directly to horizion (without the external haproxy) | 22:51 |
| fallenour | thedac: theres a pile of errors in the error.log file, all for the keystone system, all failing to login | 22:51 |
| thedac | ok, so keystone may be in a bad state. | 22:52 |
| fallenour | thedac: https://paste.ngx.cc/56 | 22:52 |
| fallenour | thedac: what is the chance the system is refusing all connections, not just ssh? | 22:52 |
| fallenour | thedac: just tried a reboot command on the keystone system | 22:53 |
| fallenour | thedac: its not responding? o.O | 22:53 |
| thedac | how did you do that? | 22:53 |
| thedac | Let's try 'juju status keystone' | 22:54 |
| fallenour | thedac: yeap, already got it: Unit Workload Agent Machine Public address Ports Message keystone/0* active executing 2/lxd/2 10.0.0.4 5000/tcp (config-changed) Unit is ready | 22:54 |
| thedac | And can you ping 10.0.0.4? | 22:55 |
| thedac | How about 'nc -vz 10.0.0.4 5000' | 22:55 |
| fallenour | thedac: yea, ms just droped from 18+ down to 2-3. | 22:55 |
| thedac | fallenour: just fyi, I am running out road here | 22:56 |
| fallenour | thedac: yea, me too. ok. another idea. juju config keystone/0 ; and then just simply reset the api port to 5000 again? | 22:57 |
| thedac | ? | 22:57 |
| fallenour | im still getting the same, failed user login, connection timed out | 22:57 |
| thedac | did you try the nc command? | 22:57 |
| fallenour | thedac: didnt try that. not very familiar with syntax, cna you give me an example? | 22:58 |
| thedac | nc -vz 10.0.0.4 5000 | 22:58 |
| thedac | :) | 22:58 |
| thedac | That will check if that ip is listening on that port | 22:58 |
| fallenour | thedac: its non-responsive to the command. me thinks the system keystone/0 isnt repsonding to inquiries from other systems? | 22:59 |
| thedac | hmm, ok | 22:59 |
| thedac | I am really confused because juju status should be telling us it is down | 23:00 |
| fallenour | thedac: I know, this is insane | 23:00 |
| fallenour | even tried juju config keystone/0 service-port=5000 ; got a warning that it was already configured to 5000 | 23:01 |
| fallenour | still not working thedac | 23:01 |
| thedac | yeah, we are down at the machine level problems at this point. You cannot ssh to the unit at all? | 23:01 |
| fallenour | thedac: yea, cant use juju ssh keystone/0 | 23:02 |
| thedac | How about the host on machine 2. juju ssh 2 | 23:02 |
| fallenour | thedac: on it now | 23:02 |
| thedac | Try that nc command from on machine 2 | 23:02 |
| fallenour | thedac: lxc list shows 3 containers | 23:03 |
| fallenour | thedac: they are on the same machine together 8O | 23:03 |
| thedac | They are contained as lxcs | 23:03 |
| fallenour | thedac: keystone and openstack-dashboard, yea, foudn it with lxc list ; I didnt know they were on the same box together, never realized. | 23:04 |
| thedac | That should still work | 23:04 |
| fallenour | thedac: no response yet from the nc -vz 10.0.0.4 5000 command | 23:04 |
| thedac | ok | 23:04 |
| fallenour | thedac: ping works | 23:04 |
| fallenour | thedac: firewall issue on the system? | 23:04 |
| thedac | no | 23:04 |
| fallenour | thedac: yea, dont think so either, tons of inbound established from 10.0.0.46 on variety of ports. | 23:05 |
| thedac | seems that container is not responding. Again I can't explain why juju is unaware of this | 23:05 |
| fallenour | nothing for port 5000 though | 23:05 |
| fallenour | thedac: lxc restart syntax to restart container ? | 23:06 |
| thedac | We could try that. I was holding off yet | 23:06 |
| thedac | Try an lxc exec and see if we can get a shell | 23:06 |
| thedac | lxc exec juju-492c8a-2-lxd-2 bash | 23:07 |
| thedac | Try that | 23:07 |
| fallenour | thedac: we got shell | 23:08 |
| thedac | ok see if you have network connectivity from within the container | 23:08 |
| thedac | ping out or something | 23:08 |
| fallenour | thedac: no ping to google | 23:09 |
| fallenour | thedac: ping to dashboard good | 23:09 |
| thedac | try the db 10.0.0.66 | 23:09 |
| fallenour | thedac: ping to google working now | 23:09 |
| thedac | ok | 23:09 |
| fallenour | thedac: ping to db working | 23:10 |
| thedac | let's check keystone logs /var/log/keystone/keystone.log | 23:10 |
| fallenour | thedac: wait...its intermittent | 23:10 |
| thedac | That would be a problem | 23:10 |
| fallenour | the 78% packet loss? | 23:10 |
| fallenour | thedac: working fine now, really weird | 23:10 |
| thedac | Well that indicates we may have bigger problems. If the network is unreliable | 23:11 |
| fallenour | thedac: we have logs, massive amounts of logs, check this out: https://paste.ngx.cc/d7 | 23:12 |
| thedac | ok, but no ERRORS | 23:12 |
| thedac | So keystone thinks it is running fine | 23:12 |
| fallenour | thedac: yea, looks that way. | 23:13 |
| thedac | Any chance we have a duplicate IP for 10.0.0.4? That would explain some of the intermittent behavior | 23:13 |
| thedac | or for the DBs ip | 23:13 |
| fallenour | thedac: maybe? id have to check maas, but Im not even sure were to look really for that | 23:14 |
| thedac | If there are no manually configured hosts and maas controls everything then I am not worried abou it. It is if we have any manually configured hosts that it could be a problem | 23:14 |
| thedac | So as a couple of baby steps. We could restar apache on the keystone unit. Test. Maybe restart the keystone unit container. test. But if this is a network issue none of that will help | 23:16 |
| fallenour | thedac: its non responsive to the service apache2 restart command | 23:17 |
| thedac | It is just hanging? | 23:17 |
| fallenour | thedac: super slow, says active(running) now | 23:17 |
| fallenour | thedac: its taking a long time | 23:17 |
| fallenour | thedac: whats the command again for cpu util? | 23:17 |
| thedac | Can you check how many keystone processes are running? ps -ef |grep keystone |wc -l | 23:18 |
| fallenour | thedac: 7 | 23:18 |
| thedac | ok, that is reasonable | 23:18 |
| thedac | This feels like a performance issue somewhere | 23:19 |
| thedac | fallenour: I need to head out. My parting advice is just to check for system load on the physical servers and see if you can isolate where there is load | 23:19 |
| fallenour | thedac: alright, ill check it out. also just looked at the haproxy status on the container itself, its fine as well. | 23:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!