[00:27] <stokachu> did the latest juju release fix the local ssh problems?
[00:31] <davecheney> stokachu: it shuld have
[00:32] <davecheney> what problem are you having ?
[00:32] <stokachu> Permission denied (publickey,password).
[00:32] <stokachu> ERROR exit status 255
[00:32] <stokachu> this was from a bootstrap, then juju deploy wordpress
[00:33] <stokachu> using juju debug-log
[00:34] <stokachu> ive also started seeing agent-state-info: 'hook failed: "install"'
[00:34] <stokachu> with the latest juju-core
[00:35] <davecheney> stokachu: what was the command you typed ?
[00:36] <stokachu> davecheney: sudo juju debug-log
[00:36] <stokachu> and sudo juju deploy wordpress
[00:36] <davecheney> stokachu: you don't need to use sudo
[00:36] <davecheney> for the local povider you need to sudo *ONLY* for juju bootstrap and juju destroy-environment
[00:36] <davecheney> this should be clearly marked in the documentation
[00:36] <davecheney> if it is not
[00:36] <davecheney> please let me know and I will raise an issue
[00:37] <davecheney> i believe debug-log does not work on the local provider
[00:37] <davecheney> but the logs are in ~/.juju/local/something/seomting
[00:37] <davecheney> we loopback mount them there
[00:38] <stokachu> re-running with just juju deploy wordpress
[00:45] <stokachu> davecheney: http://paste.ubuntu.com/6356219/
[00:45] <stokachu> ran with `sudo juju bootstrap --upload-tools; juju deploy wordpress`
[00:46] <rick_h_> stokachu: yea, that's the bug I hit, that's not really a bug.
[00:46] <davecheney> stokachu: le sigh
[00:47] <rick_h_> #1247299
[00:47] <_mup_> Bug #1247299: apparmor blocks cgroup-lite from mounting <cgroup-lite (Ubuntu):Invalid> <https://launchpad.net/bugs/1247299>
[00:47] <stokachu> ah
[00:47] <davecheney> rick_h_: is this the thing that slipped into 12.04.3 while we were in SF ?
[00:47] <rick_h_> stokachu: I hit that because I was installing build-essentials in my install hook, which grabs cgroups and that fails in lxc
[00:47] <rick_h_> davecheney: not sure when this hit. I've just started working on my own personal charm and hit that bug this weekend
[00:48] <stokachu> rick_h_: i guess wordpress does something similar i havent looked
[00:48] <rick_h_> but it's marked invalid since it's got a lxc workaround
[00:48] <rick_h_> davecheney: however, I'd say it's a bug against the juju lxc approach then perhaps?
[00:48] <rick_h_> davecheney: I was going to bring it up on Monday when more folks were around
[00:49] <stokachu> http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg03673.html
[00:49] <davecheney> rick_h_: it is monday ...
[00:49] <rick_h_> I know it's not everyone's approach, but I like to do my python stuff in a virtualenv and need build tools and -dev packages to be able to pip install some things into it
[00:49] <stokachu> thats what i used
[00:49] <rick_h_> davecheney: not here :P
[00:49] <stokachu> i ran into this when using vagrant-lxc and maas
[00:49] <stokachu> had to create a separate aa profile
[00:50] <rick_h_> stokachu: right, you turned off apparmor
[00:50] <davecheney> app armour has done more to improve the security of linux by scaring people away than any other 'security technology'
[00:50] <rick_h_> I think juju should work with our setup ootb better
[00:50] <stokachu> i didnt turn it off
[00:50] <davecheney> rick_h_: i agree
[00:50] <davecheney> and it did right til last week
[00:50] <rick_h_> stokachu: sorry, I thought you did what this therad mentions > Alternatively, you could set "lxc.aa_profile = unconfined" which would
[00:50] <stokachu> i added "mount -> /tmp/*/*," to /etc/apparmor.d/lxc/lxc-default-with-loops
[00:50] <rick_h_> > turn off apparmor entirely for the container.
[00:50] <rick_h_> stokachu: ah, ok. Sorry, missed which approach you took.
[00:50] <stokachu> :)
[00:51] <stokachu> i initially turned it off but decided against it
[00:51] <rick_h_> davecheney: so yea, suggestions on how best to move the bug from lxc to juju appreciated. I'm not really sure how/who best to bring up the point.
[00:51] <davecheney> rick_h_: thumper is around
[00:51] <rick_h_> but I was :( when I hit roadblocks trying to charm my own stuff this weekend
[00:51] <davecheney> he wrote the local provider
[00:51] <rick_h_> oooh, I like bugging thumper :)
[00:51] <davecheney> and I thought that he knew about this bug
[00:52]  * thumper isn't round
[00:52] <thumper> been exercising and everything
[00:52] <rick_h_> and can blame all things on him wahoo!
[00:52] <stokachu> lol
[00:52] <rick_h_> thumper: I'd like my stuff to work kthx :P
[00:53]  * davecheney hands thumper his rimshot
[00:53] <rick_h_> thumper: #1247299 is biting a bunch of us and lxc says it's not their problem. How do we make it juju lxc's problem?
[00:53] <_mup_> Bug #1247299: apparmor blocks cgroup-lite from mounting <cgroup-lite (Ubuntu):Invalid> <https://launchpad.net/bugs/1247299>
[00:53]  * thumper looks
[00:54] <stokachu> my problem showed up when trying to mount ephermeral images from /tmp
[00:54] <rick_h_> thumper: and mine from trying to install build-essentials
[00:55] <thumper> ugh
[00:55] <stokachu> i just want to deploy joomla :(
[00:55] <stokachu> lol
[00:55] <rick_h_> stokachu: we've got to start boxing up a cake to send to thumper :)
[00:56]  * thumper doesn't want a cake
[00:56] <thumper> but I'll accept a 24kg kettle bell
[00:56] <rick_h_> lol
[00:56] <stokachu> rick_h_: hah my girl can make gluten free cakes
[00:57] <thumper> rick_h_: so installing build-essential triggers this problem
[00:57] <thumper> ?
[00:57] <rick_h_> thumper: yea, did for me
[00:57] <rick_h_> thumper: and moving to aws caused no issues
[00:57] <thumper> stokachu, rick_h_: can I get you both to comment on the bug with issues and reproducability?
[00:57] <thumper> I'll put it on my short term todo list
[00:57] <stokachu> sure
[00:57] <thumper> ta
[00:57] <rick_h_> thumper: sure thing
[01:00] <stokachu> thumper: ok updated, thanks :)
[01:00] <thumper> sp
[01:00] <thumper> np
[01:01] <thumper> marcoceppi: what do I need to run to create the framework of a charm?
[01:01] <thumper> marcoceppi: I want to make a charm that installs build-essential and compiles hello_world.cpp
[01:01] <thumper> and logs the output
[01:02] <marcoceppi> thumper: install charm-tools from ppa:juju/stable then run `juju charm create <name-of-charm>`
[01:02] <rick_h_> thumper: working on pushing up my demo charm and getting a log copy for the bug
[01:02] <thumper> marcoceppi: ta
[01:02] <thumper> marcoceppi: why are you working sunday night?
[01:03] <marcoceppi> stokachu: juju debug-log and juju ssh <machine-#> still do not work for local provider. You can find the logs in ~/.juju/local/log and you can ssh with juju ssh <unit>
[01:03] <marcoceppi> thumper: because the charm queue is still backed up
[01:03] <thumper> those are more things on my todo list...
[01:03] <marcoceppi> it keeps me up at night in the fetal position
[01:03] <stokachu> marcoceppi: ok thanks that was my other problem
[01:04] <marcoceppi> stokachu: np!
[01:04] <marcoceppi> we really /really/ should document that in the docs
[01:04]  * marcoceppi creates a task
[01:04] <stokachu> marcoceppi: ah sweet, juju ssh wordpress/0 works
[01:04] <stokachu> that at least unblocks me for the time being
[01:05] <marcoceppi> stokachu: yeah, you can still do everything like the other providers, just have to do it in a slightly different way
[01:05] <marcoceppi> local provider is special ;)
[01:05] <stokachu> lol cool man
[01:09] <stokachu> so with cgroup's do we need to add a profile to allow `mount -> /sys/fs/cgroup/*`
[01:11] <rick_h_> thumper: bug updated with comment, link to the charm, and the unit log showing the error
[01:11] <thumper> rick_h_: ta
[01:12] <davecheney> https://github.com/davecheney/charms/tree/master/raring/docker
[01:12] <davecheney> ^ docker charm, awesome, or heresy ?
[01:14] <marcoceppi> davecheney: awesome heresy!
[01:14] <davecheney> (Y/Y) !
[01:14] <rick_h_> I don't know. I need to look up docker again. I don't get it, I thought it was just a pretty api around lxc. Everyone is going more gagga so I must have missed something
[01:15] <davecheney> marcoceppi: seriusly, the docker install process is a piece of shit
[01:15] <davecheney> curl some brogramming thing
[01:15] <davecheney> add some packages
[01:15] <rick_h_> lol
[01:15] <davecheney> recompile your modules
[01:15] <davecheney> etc
[01:15] <marcoceppi> davecheney: oh, you mean just like rvm and every other cool package out there?
[01:15] <davecheney> so I made it into a charm
[01:15] <marcoceppi> curl pipe to bash as root! What could do wrong?
[01:15] <davecheney> FFFFFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUU cached charm!
[01:16] <rick_h_> davecheney: yea, doing local charm dev needs a --no-cachy-cachy option
[01:16] <marcoceppi> davecheney: your charm lacks configuration and hooks. Also please don't make charmers the maintiner
[01:16] <stokachu> so i think the lxc issue has to be solved within the lxc-ubuntu-cloud template along with apparmor
[01:16] <marcoceppi> rick_h_ davecheney -u flag should do the trick?
[01:16] <rick_h_> marcoceppi: yea, but then I do a diff and see my revision went from 2 to 10
[01:17] <rick_h_> and before merging go back and change it to 3
[01:17] <stokachu> where the lxc-ubuntu-cloud template sets a aa_profile customize directive for an apparmor profile we'd write
[01:17] <marcoceppi> rick_h_: echo "revision" >> .bzrignore
[01:17] <marcoceppi> nobody cares about that file anymore
[01:17] <davecheney> rick_h_: to be fair, i should be yusing --upgrade-charm
[01:17] <rick_h_> marcoceppi: I thought that was the point of -u, to auto increment that
[01:17] <davecheney> but it didn't make sense in this instance
[01:18] <davecheney> marcoceppi: i forgot that as well
[01:18] <rick_h_> marcoceppi: I guess it doesn't make sense that it would work withuot the file since there's nothing to mark upgraded?
[01:18] <marcoceppi> rick_h_: it is, but it only matters for local deployments. Everything else doesn't care about revision
[01:18] <marcoceppi> rick_h_: if you delete the file juju deploy will just create it again
[01:18] <rick_h_> marcoceppi: right, so to work on the charm you need that file to exist to dev on it.
[01:18] <rick_h_> marcoceppi: ah, didn't realize that
[01:19] <marcoceppi> rick_h_: right, but as you may well know, outside of local dev, the revision file gets changed dynamically by the store
[01:19] <marcoceppi> I <3 the store sooo much
[01:19] <davecheney>  % juju ssh docker/0 -- sudo docker run -i -t ubuntu /bin/bash
[01:19] <davecheney> ^ is this to meta ?
[01:19] <davecheney> marcoceppi: please lets not talk about cs revisions
[01:19] <rick_h_> marcoceppi: also wanted to bug you about charm helpers sometime. I tried to use them this weekend. I noticed that in the juju-gui charm we've got a charmhelpers.py to use, but it's a ton larger now and has a bunch of files that are crazy (juju-gui file?) with their own deps/etc.
[01:19] <davecheney> it's only monday
[01:19] <davecheney> i don't need to cry on monday
[01:20] <rick_h_> lol
[01:20] <marcoceppi> davecheney: eh, it could work. I saw someone's charm actually allowed you to send a docker config file via config-changed
[01:20] <rick_h_> marcoceppi: I ended up giving up and just doing shell vs trying to get the helpers into the charm
[01:20] <marcoceppi> davecheney: it's still sunday here, I'm already crying ;)
[01:21] <marcoceppi> rick_h_: yeah, I have plans to reign in charm-helpers. Documenting what it does is first on my list, packaging is a close second
[01:21] <marcoceppi> rick_h_: there's a charm school video on it if you want to watch how it works for an hour
[01:21] <rick_h_> marcoceppi: k, I wasn't sure how that workd work. The install hook would need to do the install and you couldn't use it there then?
[01:22] <marcoceppi> rick_h_: well, currently charm-helpers is embedded in the charm
[01:22] <marcoceppi> rick_h_: so it would already be available
[01:22] <marcoceppi> if you're using lp:charm-helpers
[01:22] <rick_h_> marcoceppi: ok, so the current *correct* way is to download and embed it then? Keeping it up to date via?
[01:22] <marcoceppi> rick_h_: there's an update tool
[01:22] <rick_h_> marcoceppi: yea, I figured embedding was easy when it was a single file, but it has no upgrade path and such then
[01:23] <marcoceppi> rick_h_: you basically create a .charm-helpers file and there's an executable you can run when you want to sync stuff
[01:23] <rick_h_> marcoceppi: oh, must have missed that when going through the code
[01:23] <marcoceppi> It was born out of a need to keep remote deps down for IS. But now that the library is more mature it's time to properly package it in a ppa
[01:23] <marcoceppi> so people don't have to do this
[01:24] <marcoceppi> with a goal to get it in to the archive
[01:25] <rick_h_> cool, well I wanted to try it out this weekend. Heh, between that and the cgroups issue spent a couple of hours working on the charm and managed to get one config value in and working
[01:25] <rick_h_> wheeee
[01:25] <marcoceppi> woo who!
[01:25] <marcoceppi> rick_h_: http://www.youtube.com/watch?v=6kWfLujVwNI
[01:26] <marcoceppi> some light video watching for you ;)
[01:26] <rick_h_> marcoceppi: yea, I thought I was cool trying to go through the source code and looking at the gui charm since I knew it used it. :P
[01:26] <davecheney> marcoceppi: I wonder if I could make something like juju-docker plugin that would pipe the Dockerfile to your docker/0 unit ?
[01:26] <marcoceppi> davecheney: subordinate would be cool too
[01:27] <marcoceppi> davecheney: s/would be cool/might be another way to do that as well, I don't know though/
[01:27] <marcoceppi> I should probably just go use docker and get an idea for it
[01:27] <marcoceppi> It seems cool, but I don't see what it solves that vagrant doesn't already do
[01:32] <marcoceppi> jamespage: adam_g: Do you guys want eyes on your openstack charm in the queue? I know we've typically left those up to your review but I'd be happy to move it along
[03:27] <davecheney> marcoceppi: simplest explaiantion of docker
[03:27] <davecheney> checking your container into git
[03:27] <davecheney> want to move your containre to a new provider
[03:27] <davecheney> just check it out over there
[06:13] <jam> morning all
[06:13] <jam> hey wallyworld_, sorry I'm late, still want to chat?
[06:21] <wallyworld_> jam: hi, just got back from getting kid from school, he was meant to have cricket training but it got called off so i got a phone call to go get him. we can talk now if you want
[06:21] <jam> wallyworld_: sure, I'm there now
[06:29] <nesoi1> Hello, I just went through the juju demo, installing wordpress and mysql on an Amazon EC2 instance. There were no error messages, but when I go to that URL I get 502 Bad Gateway
[06:29] <nesoi1> anyone here have an idea of what I should do to correct this?
[06:32] <nesoi1> ah, it was just a delay
[06:32] <nesoi1> nevermind
[12:26] <ahasenack> hm, I have juju-core 1.16.2
[12:26] <ahasenack> yet bootstrap is *uploading* 1.17.0.1 tools
[12:26] <ahasenack> andreas@nsn7:~$ juju version
[12:26] <ahasenack> 1.17.0-raring-amd64
[12:26] <ahasenack> but
[12:26] <ahasenack> andreas@nsn7:~$ juju bootstrap -v --constraints mem=1500M
[12:26] <ahasenack> verbose is deprecated with the current meaning, use show-log
[12:26] <ahasenack> 2013-11-04 12:24:12 INFO juju.provider.openstack provider.go:127 opening environment "canonistack2"
[12:26] <ahasenack> 2013-11-04 12:24:15 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.17
[12:26] <ahasenack> ...
[12:27] <ahasenack> oh, wait
[12:27] <ahasenack> juju-core	1.16.2-0ubuntu1~ubuntu13.04.1~juju1
[12:27] <ahasenack> so the package is 1.16.2, but it calls itself internalle 1.17.0?
[12:27] <ahasenack> and there are no 1.17.0 tools in this cloud I suppose
[12:31] <ahasenack> oh, never mind, my mistake
[12:31] <marcoceppi> ahasenack: it should attempt to bootstrap with the same client minor version
[12:32] <ahasenack> marcoceppi: it's my mistake, I had another "juju" binary in the PATH :(
[12:32] <marcoceppi> ahasenack: ah
[13:12] <sidnei> marcoceppi: https://code.launchpad.net/~sidnei/charms/precise/haproxy/restore-services-from-relation/+merge/193773 sorry, the branch you merged had broken tests. i should have set it to 'work in progress'. also, probably good practice to run the tests before merging for those that have it now.
[13:12] <marcoceppi> sidnei: I typically do, didn't realize this has tests
[13:13] <marcoceppi> sidnei: let me roll back
[13:13] <sidnei> marcoceppi: wait, no
[13:13] <marcoceppi> sidnei: wait, tests pass
[13:13] <sidnei> marcoceppi: this is a follow up mp that fixes the tests
[13:14] <sidnei> marcoceppi: indeed, my bad. the uncommitted fix in my local branch broke the tests, i thought i had pushed it.
[13:14] <marcoceppi> sidnei: do you want me to roll back or not?
[13:14] <sidnei> marcoceppi: anyway, the previous fix was incomplete. please merge this additional mp
[13:15] <sidnei> it was tested by ahasenack
[13:52] <marcoceppi> sidnei: merged
[13:52] <sidnei> marcoceppi: thanks
[13:57] <mthaddon> anyone able to help troubleshoot an env that says it's bootstrapped and not bootstrapped at the same time? http://paste.ubuntu.com/6358869/
[14:00] <marcoceppi> mthaddon: if you ran juju bootstrap, and the instance didn't fire up, or went away, there's an "is_bootstrapped" file in the object store that tells juju that it's bootstrapped
[14:00] <marcoceppi> mthaddon: juju destroy-environment to clean that file up, then you can bootstrap again
[14:01] <mthaddon> marcoceppi: is there a bug about this do you know? this is pretty troublesome for mojo
[14:01] <marcoceppi> mthaddon: not that I know of
[14:01]  * marcoceppi looks
[14:02] <mthaddon> ah, the bootstrap node is in ERROR state
[14:02] <marcoceppi> mthaddon: https://bugs.launchpad.net/juju-core/+bug/1212177
[14:02] <_mup_> Bug #1212177: bootstrap detection stops too early <bootstrap> <ui> <juju-core:Triaged> <https://launchpad.net/bugs/1212177>
[14:02] <mthaddon> thx muchly
[16:01] <jcastro> bcsaller, did you end up getting the bundles working?
[16:01] <jcastro> with quickstart?
[16:31] <Azendale> marcoceppi: should the execution_environment() function in the charm helpers be valid to call when you are in an install hook?
[16:32] <marcoceppi> Azendale: which charm helpers?
[16:33] <Azendale> marcoceppi: I'm looking at the one in etherpad-lite. How can I know/find the version?
[16:36] <marcoceppi> Azendale: there's a charm-helpers1 and charm-helpers2. If there's a folder called "charmhelpers" that's embeded in the charm it's charm-helpers 2
[16:37] <Azendale> marcoceppi: ok, then, it's charm-helpers2
[16:37] <marcoceppi> execution_environment seems valid enough
[16:37] <marcoceppi> the relation_ commands would be problematic
[16:37] <marcoceppi> should probably be wrapped in a try: block for those specific relation calls and set to either False or an empty array
[16:38] <marcoceppi> actually
[16:38] <marcoceppi> these all look safe
[16:38] <marcoceppi> each relation method seems to do the right thing
[16:39] <Azendale> relation_get() is the one that is failing in the install hook
[16:40] <marcoceppi> relation_get should return None based on the code
[16:40] <marcoceppi> the try block catches ValueError to make it none
[16:41] <marcoceppi> Azendale: what error is being raised?
[16:41] <Azendale> marcoceppi: Not ValueError. Let me look
[16:42] <Azendale> marcoceppi: it raises CalledProcessError
[16:42] <marcoceppi> Azendale: could you show me the output again?
[16:42] <Azendale> https://bugs.launchpad.net/charms/+bug/1247636 has a copy of it
[16:42] <_mup_> Bug #1247636: etherpad-lite fails to deploy, install hook running get-relation <Juju Charms Collection:New> <https://launchpad.net/bugs/1247636>
[16:43] <marcoceppi> Azendale: hum, maybe the catch should be updated to catch subprocess.CalledProcessError
[16:44] <marcoceppi> ah yes, I thought this looked familiar
[16:44] <marcoceppi> Azendale: you brought this up earlier (or someone did)
[16:44] <Azendale> marcoceppi: yes, I did
[16:44] <marcoceppi> oh, there's our conversation!
[16:45] <Azendale> marcoceppi: yep. I got a chance to get back to this this morning
[16:45] <marcoceppi> I'm torn as for what the solution should be.
[16:46] <marcoceppi> I think there needs to be a try block in the execution_environment method
[16:46] <marcoceppi> I still think it erroring out at relation_get because no JUJU_REALTION_ID is provided is valid
[16:49] <Azendale> marcoceppi: So, it looks like execution_environment() is intended to get all the valid information it can about the environment. Maybe make it check?
[16:49] <marcoceppi> Azendale: yeah, it should try for the relation_get call and set the value to None during exception
[18:38] <kentb> hey all, I just hit this exact same problem on juju running saucy...after starting up dbus manually and a ton of headache, I can get the charm to deploy.  Is there anything I need to be doing differently with the saucy bits?
[18:38] <kentb> http://askubuntu.com/questions/364714/juju-deploy-of-charm-mysql-in-maas-provider-failing-after-successful-bootstrap
[19:44] <marcoceppi> jcastro: mmm, check this out http://rtomayko.github.io/ronn/ronn.1.html
[19:45] <jcastro> WHAT.
[19:46] <marcoceppi> wish it wasn't a ruby gem. I want to use this with charm tools, so you can juju charm info <charm> and get a man page output from the readme
[19:47] <marcoceppi> maybe I'll make like Ronn as a Service or something
[19:47] <jcastro> heh
[20:00] <Azendale> marcoceppi: Is there somewhere I should be branching charm-helpers2 from? I think I found a pretty good fix to the problem it was having with running get-relation in an install hook
[20:01] <marcoceppi> Azendale: lp:charm-helpers
[20:19] <Azendale> marcoceppi: I branched the charm helpers, but it looks like there is a change in the latest one that looks a lot like my change -- so I think a new version of the charm helpers might just fix the problem all together. What's the process for update the version of charm helpers a charm uses? (or is there somewhere to read about this?)
[20:23] <marcoceppi> Azendale: there's a charm-helpers-sync tool, it's in http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/tools/charm_helpers_sync/README
[20:23] <marcoceppi> it'll allow you to update the latest version of charm-helpers in the charm
[20:23] <marcoceppi> then test, bzr commit, push to a branch and propose for merging
[20:24] <Azendale> marcoceppi: ok, thanks :)
[22:35] <bic2k> oi, I'm sure I'm missing something simple, but I `juju upgrade-juju` from 1.14 to 1.16.2 and now all my agents are down... lol
[22:43] <marcoceppi> bic2k: how long ago did you start the upgrade?
[22:43] <marcoceppi> bic2k: it can take some time to get everything settled again
[22:43] <bic2k> marcoceppi: more than an hour
[22:44] <marcoceppi> bic2k: try sshing in to one of your nodes
[22:44] <bic2k> marcoceppi: syslog shows it cannot connect to the API locally?
[22:44] <bic2k> marcoceppi: I can SSH in to various nodes still, including machine-0
[22:44] <marcoceppi> bic2k: right, the nodes are up, are the agents all running?
[22:46] <bic2k> marcoceppi: looks like jujud is running on the nodes still, still using 1.14 from the looks of it
[22:46] <marcoceppi> that's probably why they're all reporting down
[22:46] <marcoceppi> bic2k: has machine 0 been upgraded to 1.16.2?
[22:47] <bic2k> marcoceppi: it has the updated jujus in /var/lib/juju/tools but is still running 1.16.2
[22:47] <marcoceppi> davecheney: who should I bug on core about upgrade-juju
[22:47] <bic2k> marcoceppi: I mean it is still running 1.14.1
[22:47] <marcoceppi> bic2k: do the other nodes have 1.16.2 agents in /var/lib/juju/tools?
[22:47] <marcoceppi> this might be a real simple update the init scripts by hand if so
[22:48] <marcoceppi> but I'd like to collect logs from you if that's the case
[22:48] <bic2k> marcoceppi: sure, just tell me what logs you want and I'll get them filled on a bug for you.
[22:48] <bic2k> 1.16.2 is on all nodes/machines
[22:48] <marcoceppi> bic2k: /var/log/juju/machine-*.log if you could pastebin that for me, I'd like to look at one of them
[22:53] <bic2k> marcoceppi: I PM'ed the log
[22:53] <bic2k> marcoceppi: lots of errors up top
[22:53] <bic2k> I'll get the machine-0 log as well
[22:53] <bic2k> I suspect there are some interesting points in there as well
[22:53] <marcoceppi> yeah it looks like it did the right thing, but then failed to connect to the API
[22:54] <marcoceppi> bic2k: can you pastebin one of the juju*machine* init files in /etc/init/ on a node?
[22:54] <bic2k> marcoceppi: coming right up
[22:55]  * marcoceppi rolls the wheel of ping
[22:56] <marcoceppi> bah, everyone is eod. thumper thoughts?
[22:57] <bic2k> actually, this one is safe for public anyways: http://pastebin.com/qEKMk3hK
[22:57] <marcoceppi> bic2k: what does /var/lib/juju/tools/machine-2/jujud point to? 1.16.2 or 1.14.0 ?
[23:00] <bic2k> marcoceppi: machine-2 directory is symlinked to the 1.14.1 and unit-xxxxx-0 points to 1.16.2
[23:01] <marcoceppi> bic2k: yeah, outside of stabbing in the dark, I'm not sure where to go. Could you open a bug about this against juju-core with the logs you sent me (seems you've already stripped sensative information) as well as a tree of /var/lib/juju/tools
[23:02] <bic2k> marcoceppi: sure, any ideas on what to do in the meantime?
[23:02] <bic2k> marcoceppi: kinda deadlocked on management right now
[23:02] <marcoceppi> bic2k: I can only think of things to poke and try
[23:03] <marcoceppi> I dont' want to dig too deep in to the "oh lets try this crack idea" on a "production" environment
[23:03] <marcoceppi> it'll make it hard to step out what I did.
[23:03] <marcoceppi> I'm not even sure if you could revert back to 1.14.2 - I was going ot say you could update the sym link on boostrap to point to 1.14 but with schema changes in mongodb I dont' think that'll work sanely
[23:04] <bic2k> marcoceppi: hmm, I feel like if I could fix that API connect error on machine-0 things would work themselves out... just not sure how to manage that service on that machine. Doesn't appear as a proper upstart job
[23:05] <marcoceppi> bic2k: oh, it totally shouuld be
[23:05] <marcoceppi> bic2k: there should be two upstart jobs, juju-db and juju-something-or-another
[23:06] <bic2k> marcoceppi: oh weird, `service jujus-machine-0 status` works now
[23:06] <marcoceppi> bic2k: maybe it was still in the process of churning over?
[23:06] <marcoceppi> bic2k: is your bootstrap a micro?
[23:07] <bic2k> marcoceppi: still on 1.14.1
[23:07] <marcoceppi> bleh
[23:07] <bic2k> marcoceppi: smalls
[23:10] <marcoceppi> bic2k: so jujud-machine-0 is still running on 1.14.1?
[23:10] <bic2k> marcoceppi: yup
[23:10] <marcoceppi> :\
[23:10] <bic2k> marcoceppi: logs show that it told the nodes to upgrade
[23:10] <bic2k> marcoceppi: and it has a 1.16.2 in tools, so it staged the update... just didn't pull the trigger
[23:11] <marcoceppi> bic2k: so, try running juju upgrade-juju again...
[23:11] <marcoceppi> curious what that will do
[23:18] <bic2k> marcoceppi: sure, let me finish filing this bug report and capturing some more logs first
[23:18] <marcoceppi> bic2k: awesome, thanks for your patience
[23:18] <bic2k> marcoceppi: thanks for walking me through this :-)
[23:26] <bic2k> marcoceppi: filed, https://bugs.launchpad.net/juju-core/+bug/1247993
[23:26] <_mup_> Bug #1247993: upgrade 1.14.1 to 1.16.2 leaves machine agents not upgraded <juju-core:New> <https://launchpad.net/bugs/1247993>
[23:26] <bic2k> marcoceppi: rerun of juju upgrade-juju with verbose shows it has no upgrades
[23:26] <bic2k> marcoceppi: will wait a few minutes to see if it triggered anything
[23:31] <marcoceppi> bic2k: any progress?
[23:31] <bic2k> marcoceppi: na, it things it is already upgraded from the looks of it
[23:32]  * bic2k "thinks" not things... damn happy fingers
[23:32] <marcoceppi> gotchya
[23:34] <zradmin> is jcastro on? I think i may have found the issue with the quantum-gateway charm. It gave me a debug message about not understanding the shared-db-relation-changed hook and then it never writes any of the credentials to the nuetron.conf file
[23:36] <bjf> when i do "juju bootstrap" why is one of my maas nodes allocated to me? what does juju bootstrap do?
[23:37] <marcoceppi> bjf: bootstrap creates a node in your cloud environment that does all the orchestration
[23:37] <marcoceppi> it's how juju is asyncronous. So when you type juju deploy, you queue that event on the bootstrap node and it does it when ready
[23:38] <bjf> marcoceppi, is it unreasonable to use the maas server for juju orchestration?
[23:38] <bjf> marcoceppi, and should that node have been started up and running ?
[23:38] <marcoceppi> bjf: no. Juju can use maas as well as lxc, aws, hp cloud (openstack) and others
[23:38] <marcoceppi> yes, it should have
[23:39] <bjf> marcoceppi, ok, thanks, i'll go see what's up with that