blrnoted today that $LANG is unset in a hook execution context, this can cause problems for some python libraries that (arguably incorrectly) rely on a default system encoding e.g. calling codecs.open()01:44
blrcommented on https://github.com/juju/juju/issues/133 marcoceppi, any thoughts on resolving that one?02:14
joseOdd_Bloke, rcj: ping02:45
jamespagegnuoy, cinder is sufferring from inadequet patching in its unit tests09:24
jamespagethey work ok on a real machine - but on a virt machine (like the test environment)09:24
jamespagevdb is a real device :-)09:24
jamespageI've worked around this for now by prefixing device names with 'fake'09:24
jamespagebut it will need a wider review - I'll raise a bug task for it09:24
jamespagefor 15.1009:24
jamespagegnuoy, https://code.launchpad.net/~james-page/charms/trusty/cinder/unit-test-fixes/+merge/266692 review please :-)09:35
jamespageresolves the cinder test failures for now09:35
gnuoyjamespage, thanks, merged09:40
Odd_Blokejose: Pong.09:50
jamespagegnuoy, anything else I can help with?09:59
gnuoyjamespage, well if you wanted to create a skeleton release note I wouldn't hold it against you ...10:09
jamespagegnuoy, ok10:11
jamespagegnuoy, dosaboy, beisner: https://wiki.ubuntu.com/ServerTeam/OpenStackCharms/ReleaseNotes150710:22
jamespagegnuoy, deploy from source needs to coreycb input - is he back today?10:25
* jamespage goes to look10:26
gnuoyjamespage, looks like we have charm upgrade breakage.10:43
gnuoyrabbit is refusing connections from neutron-gateway after the upgrade (invalid creds). investigating now10:43
gnuoyActually, it's refusing connections from everywhere by the looks of it10:45
jamespagegnuoy, that would indicate a crappy rabbitmq upgrade methinks11:03
jamespagegnuoy, is that on 1.24?11:03
jamespagegnuoy, I'll take a peek11:06
jamespagegnuoy, I think I see a potential bug in the migration code in peerstorage11:15
gnuoyjamespage, sorry, was stuffing food in my face11:19
gnuoyjamespage, yes, 1.2411:19
gnuoyjamespage, am all ears viz-a-viz peerstorage migration bug. I still have my environment so can supply additional debug if helpful11:20
jamespagegnuoy, L9711:20
jamespagethere is a relation_get without a rid11:20
jamespageso I think that may cause a regeneration of all passwords if called outside of the cluster relation context11:21
jamespagegnuoy, grrr - redeploying as dns foobar11:22
gnuoyjamespage, it definitely looks like the password has changed in rabbit, using "rabbitmqctl change_password" to set it back to what it was seems to fix things.11:28
jamespagegnuoy, the password changed in rabbit, or the password changed on all the relations?11:28
jamespagethat's my hypothesis11:28
gnuoyjamespage, the password changed in rabbit11:29
gnuoyjamespage, well actually, what I'm saying is, the password in the client config and the password advertised down the relations are the same but they don't seem to equal the actual password rabbit has for the user11:30
jamespagegnuoy, yah11:30
jamespagethat matches my theory - just trying to prove it11:30
jamespagegnuoy, there is no code in the charm that changes passwords in rabbit, but it would ignore a change triggered by a broken migration - that would propagate out to related services, but not reflect the actual password11:31
beisnero/ good morning11:32
beisnergnuoy, afaict, reverse dns is/was a-ok.  host entries are coming and going with instances.11:33
gnuoybeisner, it worked straight through on my bastion with the only error being trusty/git/icehouse. I've scheduled another run but it's been in the queue for ~4hours11:34
beisnergnuoy, however, i have observed that due to rmq-funk in serverstack, some messages are really delayed.  that is observable in that serverstack-dns may not always have the message back and the reverse dns record added by the time the instances is already booted an on its way.  :-/11:34
beisnergnuoy, saw this as well on ddellav's bastion as we were t-shooting a failed re-(re)deploy11:35
jamespagemy dns appears foobarred right now11:35
jamespageI thought I just fixed it up11:35
beisnergnuoy, throttle is way down.  if we turn it up to have more concurrency, serverstack gives us error instances.11:35
beisneri just removed 6 error instances from last night (which induced some job fails)11:36
jamespagebeisner, indeed - I have partial entries for my dns11:36
gnuoyjamespage, I don't follow the scenario you outlined. broken migration?11:36
gnuoyI assume you don't mean db migration11:36
jamespagegnuoy, yeah - the migration incorrectly missed the peer relation data, so generates a new password11:36
jamespagegnuoy, peer -> leader migration11:37
gnuoyoh, yes, of course that migration11:37
gnuoyjamespage, so rabbit is pushing out a new password to the clients without actually changing the password for the user to the new value?11:37
gnuoyoh /o\11:38
jamespageI think that's the case, but can't get an env up right now11:38
jamespagebeisner, gnuoy: it would appear notifications are going astray somewhere on serverstack11:40
beisnerjamespage, oh yeah ... also observable in not always getting an instance;  juju sits at "allocating..."11:40
beisnermeanwhile nova knows nothing of the situation11:41
beisnerbut, on the jobs ref'd in bugs, i've run, re-run, and re-confirmed that things went well for those runs, afaict.11:41
jamespagebeisner, hmm11:42
beisnerjamespage, gnuoy - mojo os-on-os deploy test combos all pass.  bear in mind, that just fires up an instance on the overcloud, checks it, and tears down.
beisnerso there's a \o/ !11:44
beisnerjamespage, gnuoy - the bare metal equivalent of that ^ is also almost all green.  re-running a T-K fail.
jamespagegnuoy, do we have a bug open for the rmq upgrade problem?12:03
jamespagethe password def gets missed during the migration12:03
gnuoyjamespage, nope, I'll create one now12:03
beisnerjamespage, gnuoy:  fyi just deployed T-I/next.  vgs and lvs come back "no volume groups found."   added to bug 148050412:07
mupBug #1480504: Volume group "cinder-volumes" not found <amulet> <openstack> <uosci> <cinder (Juju Charms Collection):New> <https://launchpad.net/bugs/1480504>12:07
gnuoyjamespage, Bug #148089312:08
mupBug #1480893: Upgrading from stable to devel charm breaks clients <rabbitmq-server (Juju Charms Collection):New> <https://launchpad.net/bugs/1480893>12:08
joseOdd_Bloke: hey, I'm getting some errors with the ubuntu-repository-cache charm, the start hook is failing12:13
joselet me run and do a pastebin of the output12:14
jamespagegnuoy, dosaboy: added some detail to that bug - I need to take an hour out - maybe dosaboy could look at a fix in the meantime?12:17
jamespageotherwise I'll pickup when I get back12:17
gnuoyjamespage, dosaboy, I can take a look12:18
jamespagegnuoy, ta - i think the migration code needs to switch to always resolving the using the rid for the cluster relation - or get passed that from high up the stack (its not currently)12:19
=== psivaa is now known as psivaa-lunch
joseOdd_Bloke: lmk once you're back around please12:28
Odd_Blokejose: o/12:28
joseOdd_Bloke: hey. I'm getting an error on the start hook of the ubuntu-repository-cache charm, says 'permission denied' for /srv/www/blahblah12:29
joseI'm having some issues with GCE right now so haven't been able to launch the instance12:29
Odd_BlokeOh, hmph.12:29
Odd_BlokeLet me see if I can reproduce.12:29
joseI'll try to run again12:30
Odd_Blokejose: Are you using any config, or just the defaults?12:30
joseOdd_Bloke: defaults here12:30
Odd_Blokejose: Cool, waiting for my instances now. :)12:40
joseI wish I could say the same...12:41
Odd_Blokejose: I'm seeing a failure in the start hook; let me dig in to it.12:49
Odd_BlokeSome of the charmhelpers bits changed how they do permissions, so it's probably an easy fix.12:49
josecool, I thought that but wasn't sure12:49
Odd_Blokejose: Do you have a recommendation for quickly testing new versions of charms?  Is there something I can do with containers, or something?12:50
joseOdd_Bloke: oh, definitely! wall of text incoming12:50
joseso, ssh into the failing instance. then do sudo su. cd /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/12:51
* Odd_Bloke braces for impact.12:51
joseedit start from there12:51
josethen save your changes and do a juju resolved --retry ubuntu-repository-cache/012:51
joseand if it goes well it should go out of error state12:51
josejust copy the exact same changes you did on the unit to your local charm and commit + push12:52
joseDHX should be a good tool too, but I can't give much insight on how it works and its usage12:54
coreycbjamespage, hey I'm back, need input for something?12:55
Odd_BlokeHmph, I'm sure we saw this problem before and I fixed it.12:58
Odd_BlokeI guess I did trash my old charm-helpers merge branch, which might have been where I fixed it.12:59
joseprobably missed that one bit :)13:00
jamespagecoreycb, yeah - could you check the deploy from source release notes pls?13:01
jamespagecoreycb, https://wiki.ubuntu.com/ServerTeam/OpenStackCharms/ReleaseNotes150713:01
jamespagegnuoy, how far did you get?13:06
gnuoyjamespage, so...13:06
gnuoyI don't think which specify rid any higher13:07
jamespagegnuoy, ?13:07
gnuoysince leader_get is supposed to mimic leader-get13:07
jamespagegnuoy, well in the scope of peerstorage, its whatever we make it :-)13:07
jamespageas we have a wrapper function there13:07
gnuoyjamespage, as for line 98, peer_setting = _relation_get(attribute=attribute, unit=local_unit(), rid=valid_rid)13:08
gnuoydoes fix it13:08
gnuoyjamespage, if you use  relation_get you get an inifinte loop which is fun13:08
jamespagegnuoy, I was thinking - http://paste.ubuntu.com/11993006/13:08
jamespageless the debug13:09
jamespagegnuoy, this has potential to impact of pxc and stuff right?13:09
gnuoyjamespage, yes the whole caboodle13:09
jamespagegnuoy, infact I'm surprise everything else is still working :-)13:10
gnuoyjamespage, +1 to your fix given the point you make about the scope of leader_get in peer storage13:10
jamespagegnuoy, ok working on that now13:10
coreycbjamespage, notes look good, I made a few minor tweaks.13:13
jamespagedosaboy, gnuoy: https://code.launchpad.net/~james-page/charm-helpers/lp-1480893/+merge/26671213:24
jamespagethat should sort-out the out-of-cluster context migration of peer data to leader storage13:26
gnuoyjamespage, I'm surprised lint isn't sad about rid being defined twice13:28
gnuoyjamespage, err ignore me13:30
* jamespage was already doing that :-013:31
jamespagegnuoy, lol13:31
dosaboyjamespage: reviewed13:36
jamespagedosaboy, gnuoy jumped you and landed that13:38
jamespagedosaboy, I actually think leader_get should not be exposed outside of peerstorage13:38
jamespageits an internal function imho13:38
jamespagethe api is peer_retrieve13:38
jamespagewhich deals with the complexity13:38
dosaboyjamespage: yup fair enough13:42
jamespagegnuoy, want me to deal with syncing that to rmq?13:45
gnuoyjamespage, well, we should sync it across the board13:50
jamespagegnuoy, +113:50
jamespagegnuoy, got that automated yet?13:50
gnuoybeisner, looks like it's time for another charmhelper sync across the charms. I'll do that now unless you have any objections?13:52
beisnergnuoy, +1 also ty13:52
Odd_Blokejose: My units seems to get stuck in 'agent-state: installing'; any idea how I can work out what's happening?13:56
joseOdd_Bloke: juju ssh ubuntu-repository-cache/0; sudo tail -f /var/log/juju/unit-ubuntu-repository-cache-0.log (-n 50)13:56
josethat gives you the output of your scripts13:57
Odd_Blokejose: I haven't even got the agent installed yet, so my scripts haven't started.13:58
axinoOdd_Bloke: you'll have to go on the GCE console13:58
axinoOdd_Bloke: and look at "events" (or something) there13:58
joseOdd_Bloke: oh, huh. if there's a machine error, then juju ssh 0; sudo tail /var/log/juju/all-machines.log13:58
joseaxino: it's probably best to take a look at all-machines.log, last time when I went to the gce console machines simply weren't there and I couldn't find a detailed answer on what was going on :)13:59
axinojose: there was nothing in all-machines.log last time I had issues :( just events in GCE console (which are a bit hard to find, I must say)14:00
Odd_BlokeOh, perhaps I misunderstand the status output.14:00
joseI'm still learning how to deal with GCE :)14:00
Odd_Blokejose: OK, looks like I've fixed it.14:01
Odd_BlokeLet me push up a MP.14:01
josewoohoo! \o/14:01
Odd_Blokejose: https://code.launchpad.net/~daniel-thewatkins/charms/trusty/ubuntu-repository-cache/fix-perms/+merge/26672414:02
josetaking a look14:03
Odd_Blokejose: So host.mkdir creates parents, so that line is unnecessary, and forces permissions to something that is broken.14:05
Odd_Blokejose: So we can just lose that line.14:05
joseas long as it works we're good :P14:06
Odd_Blokejose: It does. :)14:16
jose'running start hook'14:17
Odd_BlokeOh, you're _testing_ it?14:17
joseI am :)14:17
joseneed to14:17
Odd_BlokeAs well you should.14:18
stoIs anyone working on a charm to install designate? I want to try it on my openstack deployment and I'll be happy to test a charm instead of installing it by hand (I have no experience writing charms right now)14:18
josesto: I'm sorry, but I don't know what designate is. maybe you have a link to its website?14:19
stojose: it is an openstack service https://github.com/openstack/designate14:19
stoAnd it is already packaged14:20
joseunfortunately, I don't see a designate charm on the store. sorry :(14:20
josebut maybe an openstack charmer can work on it? :)14:21
jamespagegnuoy, I'm going to switch to liberty milestone 2 updates - pull me back if you need hands14:21
stoYes, I know that there is no charm on the store, thats why I was asking... ;)14:21
jamespageits not critical but would like to push it out soonish14:21
gnuoyok, np14:22
=== natefinch is now known as natefinch-afk
beisnergnuoy, just lmk when the c-h sync is all pushed, and i'll run metal tests.  probably with some sort of heavy metal playing.14:22
gnuoybeisner, crank up Slayer, c-h sync is all pushed14:22
beisnergnuoy, jamespage - wrt that cinder bug, it's with the default lvm-backed storage where i'm seeing breakage.  works fine with ceph-backed storage.  bug updated with that lil tidbit.14:23
gnuoysto I heard people talking about creating a charm but I'm not sure it ever got past the hot air stage14:23
beisnergnuoy, awesome thanks14:23
beisnergnuoy, isn't that on our list-o-stuff to add more official support for in the os-charms?14:24
gnuoyI think Barbican and Designate were high on the list14:24
beisneryep that sounds right.14:25
joseOdd_Bloke: woohoo! it looks like it deployed cool!14:29
joseI'm gonna give it a quick test ride and merge14:29
stognuoy: ok, thanks, I' guess I'll install it by hand on a container to see how it works14:32
joseOdd_Bloke: woot woot! works works works works!14:34
gnuoyjamespage, beisner Trusty Icehouse, stable -> next upgrade test ran through cleanly. thanks for the patch Mr P.14:34
beisnergnuoy, jamespage  \o/14:35
beisnergnuoy, do you have a modified upgrade spec to deal with qg:ng?14:35
gnuoybeisner, yes, I'm running from lp:~gnuoy/openstack-mojo-specs/mojo-openstack-specs-ha14:36
Odd_Blokejose: \o/14:36
joseOdd_Bloke: should be merged. thanks a bunch for the quick fix, really appreciated!14:36
Odd_Blokejose: No worries, thanks for the quick merge. :)14:37
beisnergnuoy, these guys didn't get a c-h sync, is that by design?:  n-g, pxc, n-ovs, hacluster, ceph-radosgw, ceph-osd14:42
gnuoybeisner, I'll check, they may not be using the module that changed (but I'd have thought pxc was tbh)14:42
gnuoybeisner, sorry about that, done now (no change for n-ovs)14:53
gnuoyoh, cause it did work the first time14:53
=== psivaa-lunch is now known as psivaa
=== JoshStrobl is now known as JoshStrobl|AFK
jcastromarcoceppi: oh hey I forgot to ask you if everything with rbasak/juju in distro is ok?15:50
jcastroanyone need anything from me?15:50
marcoceppijcastro: I have to fix something in the packaing and upload it, I'm about to do a cut of charm-tools and such so I'll fix those then15:51
jamespagebeisner, suspect that regex is causing the issue - reconfirmning now15:54
beisnerjamespage, ack ty15:54
beisnerbeh.  look out, gnuoy, jamespage - i just got 11 ERROR instances on serverstack ("Connection to neutron failed: Maximum attempts reached")16:12
jamespagebeisner, sniffs like rmq16:21
* beisner must eat, biab...16:22
=== Guest11873 is now known as zz_Guest11873
lazyPowerapuimedo: o/16:42
apuimedolazyPower: how are you doing?16:42
lazyPowerPretty good :) Hows things on your side of the pond?16:42
apuimedolazyPower: I have a charm that at deploy time needs to know the public ip it will have16:43
apuimedousually what I was doing was add a machine, and then knowing the ip change the deployment config file16:44
lazyPowerapuimedo: unit-get public-address should get you situated with that though16:44
apuimedobut I was wondering if it were possible in the install script to learn the public ip16:44
apuimedothat's what I thought16:44
apuimedoand it's the same the other machines in the deployment will see it with, right?16:45
apuimedolazyPower: so unit_public_ip should do the trick16:45
lazyPowerand looking at teh source, that wraps unit-get public-address :)16:47
=== JoshStrobl|AFK is now known as JoshStrobl
lazyPowernp apuimedo :)17:06
=== zz_Guest11873 is now known as CyberJacob
beisnergnuoy, jamespage - the heat charm does have a functional usability issue, though not a deployment blocker, nor a blocker for using heat with custom templates.  that is, the /etc/heat/templates/ dir is just awol.   bug 1431013   looks to have always been this way, so prob not crit for 1507/8 rls.17:43
mupBug #1431013: Resource type AWS::RDS::DBInstance errors <amulet> <canonical-bootstack> <openstack> <uosci> <heat (Juju Charms Collection):Confirmed> <https://launchpad.net/bugs/1431013>17:43
=== natefinch-afk is now known as natefinch
lazyPowerejat-: Hey, how did you get along this weekend? I wound up being AFK for a good majority.17:44
beisnergnuoy, jamespage, coreycb ... aka ... ^ our "one" remaining tempest failure to eek out  ;-)    http://paste.ubuntu.com/11994632/17:47
coreycbbeisner, would that fixup the rest of the failing smoke tests?17:49
beisnersee paste ... we are down to that 117:49
beisnerafter some merges and template tweaks today17:49
beisnercoreycb, i'm installing heat from package in a fresh instance just to see if the templates dir is awol there (ie. without a charm involved).17:50
beisnercoreycb, yeah so these files and this dir don't make it into /etc/heat/templates when installing on trusty.  http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/heat/trusty/files/head:/etc/heat/templates/17:54
coreycbbeisner, that's awesome, down to 117:54
coreycbbeisner, might be a packaging issue17:55
beisnercoreycb, yeah, woot!17:56
beisnercoreycb, and ok, bug updated, she's all yours ;-)17:56
coreycbbeisner, thanks yeah I'll dig deeper later, need to get moving on stable kilo17:57
beisnercoreycb, yep np.  thanks!17:57
beisnergnuoy, 1.24.4 is in ppa:juju/proposed    re: email, when you next exercise the ha wip spec(s), can you do that on 1.24.4?18:16
ddellavjamespage, your requested changes have been made and all tests updated: https://code.launchpad.net/~ddellav/charms/trusty/glance/upgrade-action/+merge/26559218:34
jamespagebeisner, reverting that regex change resolves the problem19:50
jamespagewith cinder19:50
beisnerjamespage, ah ok.  i can't find context on that original c-h commit @ http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/revision/40919:51
jamespagebeisner, [daniel-thewatkins] Detect full disk mounts correctly in is_device_mounted19:52
beisnerjamespage, yep, saw that, but was looking for a merge proposal or bug to tie it to.19:52
beisner(being that that fix breaks this charm)19:53
beisnerjamespage, i suspect context is:  http://bazaar.launchpad.net/~charmers/charms/trusty/ubuntu-repository-cache/trunk/view/head:/lib/ubuntu_repository_cache/storage.py#L13120:02
jamespagebeisner, I'm actually wondering whether that charm-helpers change has uncovered a bug20:03
jamespagebeisner, huh - yeah it does20:04
beisnerooo oo a cascading bug20:04
jamespagebeisner, /dev/vdb was getting missed on instances, so got added to the new devices list before20:04
jamespageno longer true20:04
* jamespage scratches his head for a fix20:04
jamespagebeisner, the charm does not have configuration semantics that support re-using a disk that's already mounted20:08
jamespagebeisner, overwrite specific excludes disks already in use - its a sorta failsafe20:09
jamespagebeisner, I could do a ceph type thing for testing20:09
beisnerjamespage, ok so vdb is mounted @ /mnt, and with that c-h fix, the is it mounted helper actually works.  whereas all along we've just  been clobbering vdb?  is that about right?20:12
jamespagebeisner, yup20:12
beisnerjamespage, ok i see it clearly now.20:13
jamespagebeisner, ok - testing something now20:19
jamespagebeisner, https://code.launchpad.net/~james-page/charms/trusty/cinder/umount-mnt/+merge/26680320:21
jamespagetesting now20:21
beisnersweet.  oh look you even updated the amulet test.  i was just thinking:  i'll need to update a config option there.20:22
beisnerjamespage, if this approach is what we stick with, i'll update o-c-t bundles20:23
jamespagebeisner, how else would I test my change? ;)20:23
beisnerjamespage, well that's the shortest path for sure!20:24
jamespagebeisner, longer term filesystem_mounted should go to charm-helpers20:24
jamespagebut for tomorrow here is fine imho20:24
beisnerjamespage, ack20:28
jamespagebeisner, passed its amulet test for me20:39
jamespagebeisner, https://code.launchpad.net/~james-page/charms/trusty/cinder/umount-mnt/+merge/26680320:39
jamespagegnuoy, ^^ or any other charmer20:39
jamespagebeisner, I've not written a unit test which makes me feel guilty20:39
jamespagebut I need to sleep20:39
marcoceppijamespage: idk, lgtm20:40
beisnerjamespage, lol20:40
jamespagemarcoceppi, ta20:40
beisnerjamespage, yes, i believe this will do the trick.   thanks a ton.  i've updated and linked the bug.20:40
marcoceppijamespage: maybe just default ephemeral-mount to /mnt ?20:40
jamespagemarcoceppi, meh - I'd prefer to keep it aligned to ceph20:41
marcoceppijamespage: and I really don't care enough either way20:41
jamespagemarcoceppi, just in case someone did have /mnt mounted as something else :-)20:41
jamespagemarcoceppi, and really did not want it unmounted20:41
* marcoceppi nods20:41
jamespagemarcoceppi, this is really a testing hack20:41
marcoceppijamespage: yeah, I see that in the amulet test you updated20:41
jamespagebeisner, ok - going to land that now20:42
beisnerjamespage, yep +120:42
jamespagebeisner, done - to bed with me!20:43
beisnerjamespage, thanks again.  and, Odd_Bloke thanks for fixing that bug in is_device_mounted.20:43
Odd_Blokebeisner: :)20:50
moqqhow do i deal with an environment that seems completely stalled? when i try ‘juju status’ it just hangs indefinitely21:00
marcoceppimoqq: is the bootstrap node running?21:10
marcoceppiwhat provider are you using?21:11
moqqmarcoceppi: yes, machine-0 service is up. manual provider21:12
marcoceppimoqq: can you ssh into the machine?21:12
marcoceppimoqq: `initclt list | grep juju`21:12
moqqmarcoceppi: http://pastebin.com/dUqwsTez21:13
marcoceppimoqq: sweet! VoltDB21:14
* marcoceppi gets undistracted21:14
marcoceppimoqq: try `sudo restart jujud-machine-0`21:14
marcoceppigive it a few mins21:14
marcoceppithen juju status21:14
marcoceppialso, are you out of disk space? `df -h`?21:15
moqqno plenty of space. and restarting the service to no avail, have cycled it a good handful of times21:15
marcoceppimoqq: have you ccycled the juju-db job as well?21:15
marcoceppithat's the next one21:15
marcoceppimoqq: time to dive into the logs21:15
marcoceppiwhat's the /var/log/juju/machine-0 saying?21:16
moqqmarcoceppi: http://pastebin.com/KWDXACvD21:16
marcoceppimoqq: were you running juju upgrade-juju ?21:17
moqqyeah at one point i tried to and it failed21:17
marcoceppimoqq: from what version?21:17
marcoceppimoqq: this may be a bug that was fixed recently, and if so there's a way to recover still21:18
moqq1.23.something -> 1.24.421:18
marcoceppimoqq: what does `ls -lah /var/lib/juju/tools` look like?21:19
moqqmarcoceppi: http://paste.ubuntu.com/11996021/21:20
marcoceppimoqq: this should help: https://github.com/juju/docs/issues/53921:20
marcoceppimoqq: you'll need to do that for all of the symlinks21:20
marcoceppimoqq: so, stop all the agents first21:21
marcoceppithen that21:21
marcoceppithen start them all up again, with juju-db and machine-0 being the first and second ones you bounce21:21
moqqok thanks. on it21:21
beisnercoreycb, around?  if so can you land this puppy?:  https://code.launchpad.net/~1chb1n/charms/trusty/hacluster/amulet-extend/+merge/26635521:22
coreycbbeisner, sure, but is that branch frozen for release?21:26
moqqthanks marcoceppi that did the trick!21:29
marcoceppimoqq: awesome, glad to hear that. It was only a bug that existed in 1.23, so going forward you shouldn't have an issue with upgrades *related to this*21:30
moqqok excellent21:30
moqqnow, its gotten me to 1.24.321:30
moqqbut it seems to be refusing to go to 1.24.421:30
moqqubuntu@staging-control:/var/lib/juju/tools$ juju upgrade-juju --version=1.24.4  >>> ERROR no matching tools available21:31
marcoceppimoqq: 1.24 is a proposed release21:31
marcoceppimoqq: you need to set your tools stream to proposed instead of released21:31
marcoceppimoqq: I'd honestly just wait until it's released (in a few days)21:31
moqqi’m pretty sure i already did. juju has been constently chewing up 100% of all of our cpus21:31
moqqso i was hoping the .4 upgrade would fix that21:32
moqqcuz if its not solved soon i have to rip out juju and switch to puppet or chef21:32
marcoceppimoqq: hum, juju using 100% shouldn't happen21:33
marcoceppiis there a bug already for this?21:33
moqqyeah https://bugs.launchpad.net/juju-core/+bug/147728121:33
mupBug #1477281: machine#0 jujud using ~100% cpu, slow to update units state <canonical-bootstack> <canonical-is> <performance> <juju-core:Triaged> <https://launchpad.net/bugs/1477281>21:33
marcoceppimoqq: looks like this was reported with 1.23, is it still chewing 100% cpu on 1.24.3?21:34
moqqit looks fine for the moment. but when i did this upgrade on the other env earlier it was fine for 20m then went back to spiking21:35
moqqgoing to watch it21:35
marcoceppimoqq: if it does spike up and start chewing 100% again, def ping me in here and update that bug saying it's still a problem, it's not targeted at a release so it's really not on the radar atm21:36
marcoceppimoqq: as to your other question about 1.24.4, what does `juju get-env agent-stream` say?21:37
moqqmarcoceppi: ok, will do21:38
moqqapparently >>> ERROR key "agent-stream" not found in "staging" environment21:38
marcoceppimoqq: haha, well that's not good21:39
marcoceppiwell, that's not bad either21:39
marcoceppijsut, interesting21:39
marcoceppimoqq: you could try `juju set-env agent-stream=proposed`, then an other upgrade-juju (per https://lists.ubuntu.com/archives/juju/2015-August/005540.html)21:40
marcoceppibut if there is no value currently it may not like that21:40
moqqjust gave a warning, but it set ok21:40
marcoceppimoqq: well, if you feel like being daring you can give it a go21:41
marcoceppiin the changelog I doin't see any reference to CPU consumption21:41
bdxcore, devs, charmers: Is there a method by which juju can be forced to not overwrite changes to config files on node reboot?22:13
beisnercoreycb, nope we can land passing tests any time.22:14

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!