=== zz_CyberJacob is now known as CyberJacob === kadams54 is now known as kadams54-away === Guest87529 is now known as mwhudson === CyberJacob is now known as zz_CyberJacob === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === zz_CyberJacob is now known as CyberJacob === kadams54 is now known as kadams54-away [12:00] juju state server upgrade | http://askubuntu.com/q/609569 [12:02] marcoceppi_, can you pass this around in nuremberg? ^^ [12:26] jcastro: ack [12:32] Is there a way to import/export a MAAS configuration from/to a yaml file? [12:54] drbidwell: what do yoy mean maas configuration? [13:05] marcoceppi: the networks, cluster, and zones information. The stuff that I need to beat into by hand when I have to rebuild it. [13:06] marcoceppi: The Mirantis fuel software has a process for feeding this kind of information into the system from a yaml file. It was handy to have. [13:16] How many nodes to use Maas/Juju for Openstack (Juno)? | http://askubuntu.com/q/609589 [13:16] Not sure if this is really a juju question, but i just did an openstack install on ubuntu 14.04 using juju. I rebooted the machine after install. It's not clear what needs to be started? It's a single instance [13:19] bcsaller: https://insights.ubuntu.com/2015/04/15/using-the-services-framework-to-implement-your-charms-intent/ [13:19] et al. [13:20] cory_fu: tl;dr [13:20] jk [13:20] :) [13:22] cory_fu: that section on rewriting charms with the framework is gold [13:26] bcsaller: Agreed === kadams54 is now known as kadams54-away [13:55] bcsaller: thanks! [13:55] :) === kadams54-away is now known as kadams54 [13:56] Spads: thank you [13:59] If a charm relation is changed, can I determine exactly which fields in the relation changed? [14:03] lukasa: not without keeping state in the unit. Its common to be able to process the whole state and take what ever actions are possible given the data you have [14:06] bcsaller: Yeah, sadly I'm working on charm that has a particularly annoying component in it [14:06] I have a follow-on question, but it works best if you know about the config file infrastructure the openstack charms have === kadams54 is now known as kadams54-away [15:08] wwitzel3, i;ve got some pull requests into juju-pyramid for you, when you get a chance === scuttle|afk is now known as scuttlemonkey === kadams54 is now known as kadams54-away === FilipeCi_ is now known as FCS [15:52] Hi, im using bootstraped in remote maas [15:52] and "juju debug-log" [15:52] says: ERROR cannot open log file: open /var/log/juju/all-machines.log: no such file or directory [15:52] strange, why it is reading local file? [15:54] mh.. ok, i just realized that juju is telling me that is not all-machine.log in machine0 [15:54] lazyPower_, catbus1: I did a manual reset, put the password in that file, and run 'juju resolved mysql/0', but now I am getting all kinds of errors regarding authorization failures from horizon: "Unable to retrieve public images", "Unable to retrieve images for the current project" when I try to launch instances or create volumes. Is the mysql root password used by the other charms in some way? [16:14] ctlaugh: it may be that horizon is using an administrative credential, which would denote it needs the password update as well [16:15] beisner: Heyo, Can i tap your brain for a second with regard to ctlaugh's issue? ^ [16:19] lazyPower_, beisner: I looked in the horizon logs and they are full of the authorization errors from glance. So, then, I looked in the glance logs, and they are full of this: "WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token". [16:20] o/ hi lazyPower_ ctlaugh [16:20] beisner: the TLDR; on whats going on - ctlaugh joined yesterday and was missing /var/lib/mysql.passwd [16:20] So, I've been looking in the keystone configs and logs trying to see what might be the cause, but haven't found it yet. Keystone's config looks like it uses a 'keystone' user, which shouldn't have changed. [16:21] to get root access back, i linked to MySQL docs on how to do a single-user-admin reset, this appears to have exploded into dropping auth for the other openstack components (assuming they were consuming the db-admin role) [16:21] I just want to point out, run juju resolved with --retry so the hook script will be re-run. running it without --retry only sets the status to resolved. [16:24] could this be bug 1423153 ? [16:24] Bug #1423153: /var/lib/mysql/mysql.passwd no longer exist [16:25] ^ ctlaugh, lazyPower_ helps if I tag ya [16:28] beisner:, lazyPower_ - I saw that bug, but wasn't sure if it was related to my situtation or not. I deployed this cloud months ago, and it's been working fine since then... until I powered the whole system off and moved it. That's when I started getting the errors and the file was missing (well, not sure if it was ever there since I had never looked for it, but the charm was complaining that it was missing). [16:33] ctlaugh, lazyPower_ - the only deployment i have up at the moment to poke on uses percona cluster instead of mysql. on that, there is no /var/lib/mysql.passwd file. that is with the 15.01 charms i believe. [16:33] ctlaugh, so do the usual api commands fail? keystone user-list && keystone token-get && nova list && cinder list && glance image-list ...etc? [16:34] + keystone catalog && nova service-list [16:35] ctlaugh, lazyPower_ - i'd definitely start by looking at keystone, rabbit and mysql logs. if those three aren't happy, the rest of the stack will be full of errors. [16:38] beisner: thanks for taking a look [16:38] beisner: lazyPower_ : some work, some don't: http://paste.ubuntu.com/10827803/ [16:40] lazyPower_, sure welcome! [16:42] ctlaugh: which revision of the keystone charm are you using? [16:43] lazyPower_: charm: cs:trusty/keystone-14 [16:43] can-upgrade-to: cs:trusty/keystone-20 [16:45] It does appear to be keystone that's having the problem, but I see nothing that looks like an error in /var/log/keystone/keystone.log. [16:45] I take that back... [16:46] Well, I see a few errors related to not being able to connect to mysql, but I think that is from when it was restarting. [16:47] ctlaugh, basically, until keystone token-get works, everything else will be problematic. i'd restart the keystone service, whilst watching its log, to see if it is connecting. [16:47] connecting to mysql that is. [16:51] ctlaugh, gotta step out for ~30, but will return. let us know re: keystone:mysql [16:58] beisner: ok, thank you. I'm trying, but keystone.log is showing NO errors when token-get is failing. Just lots and lots of debug spam. [16:58] beisner: I'm about to leave for lunch as well, but will be back soon. [17:07] ctlaugh, keystone --debug token-get [17:21] hi === CyberJacob is now known as zz_CyberJacob [18:18] beisner: http://paste.ubuntu.com/10828233/ [18:20] ctlaugh, can you pastebin the keystone log? === zz_CyberJacob is now known as CyberJacob [18:37] beisner: working on it. it's choking on the massive file size. i am going to split it. [18:44] beisner: http://svartalfheim.laughlins.org/share/keystone.txt [19:06] ctlaugh, sorry, pulled in another direction. things to check: what openstack environment variables are set? it looks like you're using a token. i generally don't, i use something like this, where x.x.x.x is the keystone IP. you may get more meaningful output from keystone --debug token-get and keystone --debug catalog if you set your environment like so. [19:06] http://paste.ubuntu.com/10828404/ [19:12] beisner: I took the token out, made sure I have all the EVs set like in your example (I had them all there already, but also had the token one) and I get this: Expecting a token provided via either --os-token or env[OS_SERVICE_TOKEN] [19:12] (from keystone --debug token-get) [19:14] ctlaugh, what does this show?: env | grep OS ... be sure to mask your actual password. [19:14] beisner: http://paste.ubuntu.com/10828440/ [19:15] ctlaugh, dpkg-query --show python-keystoneclient ? [19:16] +: apt-cache policy python-keystoneclient ? [19:17] beisner: http://paste.ubuntu.com/10828453/ [19:18] OpenStack Juno and restarting it | http://askubuntu.com/q/609718 [19:19] ctlaugh - do ANY keystone cmds work? keystone catalog keystone user-list keystone service-list keystone endpoint-list [19:20] beisner: not after removing the token EV. with the token, these work: endpoint-list, user-list, service-list [19:21] beisner: keystone catalog fails with this, even with the token: 'NoneType' object has no attribute 'has_service_catalog' [19:22] ctlaugh, weird. i'm not sure what's going on. i was hoping to see some meaningful keystone errors. any hints from mysql logs? [19:22] failed auth, errors connecting, etc? [19:24] beisner: nothing that I see [19:24] ctlaugh, are these all deployed with juju? any changes made to the stack outside of using juju config options? [19:25] rather, charm config options [19:25] beisner: no, all with Juju -- no change to any charm config outside of that [19:26] beisner: I was hoping some kind of solution was going to jump out, but I'm thinking now it might be easier to ust blow it all away and start over. [19:27] ctlaugh, so did the mysql password end up getting manually reset? [19:28] Yes, I manually reset it (the root password), and verified I could log in using it. I put that password in the mysql.passwd file and run juju resolved --retry. [19:29] beisner: I looked in the config files of nova, keystone, glance, etc. etc and if there is a database URL, they all have separate users -- never root. So, I can't see how a root password change could have affected that. [19:29] ctlaugh, right, i would think the same. [19:29] beisner: Is there something to do to force the keystone charm to re-do whatever it does to see if it can correct itself? [19:32] ctlaugh, there is a debug hooks developer mode that can re-trigger individual hooks, but it's generally for charm development. [20:26] anyone available to kick an automated charm test for me on https://bugs.launchpad.net/charms/+bug/1441622 [20:26] Bug #1441622: Review queue: xCAT [20:26] beisner: I really appreciate all your time today and all your attempts to help. I am going to redeploy my openstack setup -- hopefully that'll be quicker to get back up and running. I didn't have anything critical in it... it was just used for an Openstack CI setup I was creating. [20:29] Can I use Juju / MaaS to provision barebone servers with any image (not just openstack) ? [20:35] ctlaugh, you're welcome - sorry i couldn't help dig deeper today. === kadams54 is now known as kadams54-away === CyberJacob is now known as zz_CyberJacob [20:44] How to edit machine hardware detail deployed by Juju? | http://askubuntu.com/q/609741 [22:20] beisner: I redeployed from scratch, and now all the keystone commands are working. The only thing I see that is still failing (both in horizon and command-line) is 'glance image-list' -- keeps returning an "Authorization failed for token". Very strange... still looking. === zz_CyberJacob is now known as CyberJacob === ming is now known as Guest25942 === CyberJacob is now known as zz_CyberJacob === \b is now known as benonsoftware === wolsen_ is now known as wolsen