[01:53] <hatch> huwshimi hey you there?
[01:57] <huwshimi> hatch: Hey
[01:58] <hatch> hey check out my msg on your PR
[01:59] <huwshimi> hatch: Oh, strange... why would they be there?
[02:00] <hatch> I have no idea...
[02:00] <hatch> maybe your fork of juju-gui is out of date
[02:01] <huwshimi> $ git pull juju develop
[02:01] <huwshimi> From github.com:juju/juju-gui
[02:01] <huwshimi>  * branch            develop    -> FETCH_HEAD
[02:01] <huwshimi> Already up-to-date.
[02:01] <hatch> then run git push
[02:01] <hatch> to update your fork
[02:04] <hatch> I really don't know why those old commits would be there anyways
[02:04] <hatch> it should have diff'd them out as far as I know
[02:06] <huwshimi> yeah, quite strange.
[02:07] <huwshimi> brb, lunch
[02:13] <hatch> Thanks for fixing that huwshimi  but it looks like it's still pushing all of those commits in, maybe it needs a new PR? 
[02:21] <huwshimi> hatch: Yeah, I'll delete and start again :)
[02:21] <hatch> sorry I wish I knew what the issue was so that we could avoid it in the future hah
[02:30] <hatch> huwshimi I was wondering if you play any games on your new machine?
[02:31] <huwshimi> hatch: Nah
[02:32] <hatch> You should go buy some and let me know how they run so I can decide on the computer to buy haha
[02:32] <huwshimi> :)
[02:52] <huwshimi> hatch: By the time you're ready the new Airs will be out anyway
[02:53] <hatch> lol!
[02:58] <huwshimi> hatch: https://github.com/juju/juju-gui/pull/17
[02:58] <huwshimi> hatch: Lots of strangeness
[13:22]  * rick_h__ goes to doc office. bbl
[13:32]  * gary_poster goes to son's prospective middle school. bbl
[13:53] <benji> bac: Spads is asking me if the prod charmworld charm is "jc-charmworld/0"; do you know if that is right?
[13:54] <benji> bac: he may mean "service name" instead of "charm"
[13:55] <bac> benji: oh, the charm lives at bzr+ssh://bazaar.launchpad.net/~juju-jitsu/charms/precise/charmworld/trunk/
[13:56] <benji> bac: do you know the service name is for the production charmwolrd?
[13:56] <bac> no.  on staging it is just 'charmworld'
[13:56] <bac> charmworld/0
[13:56] <frankban> hazmat: ping re: the deployer MP
[13:57] <bac> benji: if he has access to prod, a juju status should show.  should only be one similar to charmworld
[13:58] <bac> benji: but webops grab everything and deploy locally so it could be there is a renaming
[13:58] <bac> let me look at the docs
[13:58] <benji> bac: do we run ES as its own charm in prod?
[13:59] <benji> s/charm/service/
[13:59] <bac> benji: looking at https://wiki.canonical.com/InformationInfrastructure/WebOps/CDO/Charmworld it looks like jc-charmworld is correct
[14:00] <bac> benji: yes, i would think ES is its own service.  can't see how it'd be otherwise.
[14:01] <bac> benji: the problem with understanding production is that page refers to scripts that we don't have access to, so it's kind of hard to see what's happening.
[14:01] <benji> yeah, really
[14:22] <benji> bac: it seems the staging.jujucharms.com environment is in a bad way: 2013-12-12 14:17:53,234 ERROR juju environment not found: is the environment bootstrapped?
[14:22] <benji> have you seen that before?  Should I nuke it from orbit and rebuild from scratch?
[14:23] <bac> benji: but, but, it seems happy http://staging.jujucharms.com/heartbeat
[14:23] <bac> benji: where did that terrifying message come from?
[14:23] <benji> bac: happy except that it hasn't ingested since 2013-12-08
[14:23] <benji> bac: from Spads in #webops
[14:23] <bac> benji: oh, well
[14:24] <bac> benji: huh.  well it is definitely bootstrapped
[14:25] <benji> bac: I guess I need to figure out why my orangify script isn't working so I can poke on the staging env myself
[14:27] <bac> benji: i found some subdirectories in that branch that were created at some point and were messing up the connection.  once i deleted them i could connect.  if 'bzr st' shows anything weird (local, charms, etc) you may try the same.
[14:27] <bac> weird = unknown
[14:27] <benji> bac: staging.jujucharms.com is running on go-juju, right?
[14:27] <bac> yes
[14:28] <bac> benji: you may want to manually copy over the ssh credentials.  the install script is broken in that it doesn't copy if they exist even though new ones may be available
[14:28] <bac> i've fixed it but haven't yet merged
[14:35] <bac> benji: i'm look at the staging charmworld/0 logs and don't see where that message from spads comes from.  i wonder if his env is not right.  there definitely is a problem with ingesting, and it looks like the job hasn't even tried to run since 9-Dec
[14:35] <bac> s/look/looking/
[14:35] <benji> bac: I have determined that he was running py-juju
[14:35] <bac> oh
[14:35] <bac> that'll do it
[14:35] <benji> heh
[14:36] <bac> so, one red herring down
[14:36] <benji> bac: I almost have orangify working, when I run "nova list" I get juju-staging-machine-0 through juju-staging-machine-5, is that expected?
[14:36] <bac> curtis did fix a problem with mongo the other day
[14:36] <bac> benji: sounds about right
[14:36] <bac> benji: what does juju status do?
[14:37] <benji> bac: orangify still fails, this time with "client: fatal: failed to establish ssh session (2)"
[14:37] <benji> that is after printing:
[14:37] <benji> Found a compatible Juju.
[14:37] <benji> Orange credientials loaded
[14:37] <benji> JUJU_HOME set to /home/benji/workspace/staging-tools/staging-tools
[14:37] <benji> State server is at 10.55.32.48
[14:38] <bac> benji: there's the problem
[14:38] <bac> credentials is misspelled
[14:40] <benji> heh
[14:46] <bac> benji: prob won't help but try pulling a new version of that branch
[14:46]  * benji tries
[14:46] <bac> benji: so there are lots of problems from 9-Dec and before.  that was when curtis got mongo working again
[14:47] <bac> but it doesn't look like ingest has run since then.  it hasn't generated any errors since that date
[14:47] <benji> bac: the main thing I'm interested in is getting ingest to run again; my hypothesis is that it will then make the search results on staging match prod (which are wrong, but at least we'll have an idea of what is going on)
[14:47] <bac> right
[14:50] <benji> bac: same error after pull
[14:50] <bac> but no typo?
[14:56] <frankban> guihelp: I need two reviews for https://codereview.appspot.com/41350043 (quickstart). Is anyone available? 
[15:02] <sinzui> hi bac
[15:02] <bac> sinzui: when mongo failed on staging it caused ingest to fail repeatedly and supervisord marked it as FATAL. do you know how to get it going again?
[15:04] <bac> sinzui: we're engaging spads in #webops on the issue though he isn't sure
[15:04] <sinzui> for staging?
[15:04] <sinzui> bac, he doesn't have access
[15:05] <sinzui> bac https://docs.google.com/a/canonical.com/document/d/190xKLPpEPSrlVix4D9eX_aTciiKFo12h0u7MpzqI28I/edit#heading=h.g8pxm6cjaq69
[15:05] <sinzui> ^ we need to add a section about restarting the service, which I know but cannot type without doing it.
[15:06] <sinzui> per canonistack guidelines, there is a script in /srv that allows webops/devops to restart the services
[15:08] <bac> sinzui: can i just stop and restart supervisord?
[15:09] <sinzui> bac: I did this http://paste.ubuntu.com/6561801/
[15:10] <sinzui> sudo /srv/deploymgr/charmworld restart
[15:10] <bac> sinzui: could we have just used the 'start_worker' target in the makefile?
[15:11] <bac> which i just found
[15:12] <bac> sinzui: i'll update the doc unless you want to
[15:12] <sinzui> bac: please do update the doc.
[15:13] <bac> thanks sinzui
[15:30] <rogpeppe> bac: sorry, was at lunch
[15:30] <bac> rogpeppe: ?
[15:30] <bac> rogpeppe: i didn't ping you, perhaps someone else did
[15:30] <bac> but i hope you had a good lunch
[15:31] <rogpeppe> bac: oh, sorry, i misread a remark of yours
[15:31] <rogpeppe> bac: it was just below one addressed to me
[15:31] <rogpeppe> bac: ignore :-)
[15:31] <bac> rogpeppe: np
[15:32] <bac> rogpeppe: was that a really late lunch or a really long one?  :)
[15:32] <rogpeppe> bac: late
[15:33] <rogpeppe> bac: i often forget to have lunch until hunger prompts me :-)
[15:46] <benji> bac: the queue size is increasing, so it looks like a queue-clearing is in the future
[15:47] <bac> benji: ok
[15:52] <bac> benji: should be able to do: db['charm-queue'].remove()
[15:52] <bac> though it is kinda scary
[15:52] <benji> heh, yeah
[15:54] <bac> and db['basket-queue']
[15:57]  * benji wonders if it is about team call time.
[15:59] <hatch> jujugui call in 1
[15:59] <hatch> benji are you actually going to make it on time today? :P
[15:59] <benji> I changed my reminder so I can't snooze it.  That seem to help.
[16:00] <gary_poster> rick_h__, assuming you are still at dr
[16:05] <hazmat> hatch, Makyo how'd the rest of the meeting go
[16:05] <hatch> hazmat pretty good, hangout after standup to fill you in?
[16:06] <hazmat> sure
[16:06] <hazmat> hatch, sounds good
[16:06] <hatch> cool I'll ping when we're done
[16:09] <hatch> hazmat https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.j0rk5d371ph8331ijtf48t2uj0?authuser=1 just finishing up 
[16:15] <bac> benji: so shall we do the cleanup?
[16:16] <benji> bac: yeah, we don't have a choice at this point, and since I haven't gotten orangify to work I need to enlist you if you can help
[16:16] <bac> benji: hangout?
[16:16] <benji> bac: sure, I'll create one
[16:30] <Makyo> hazmat, do we have a sprint authorisation number?
[16:34] <rick_h__> gary_poster: back now, sorry for the delay. 
[16:35] <gary_poster> np rick_h__ .  how are you feeling?  dr give you any meds?
[16:35] <rick_h__> gary_poster: yep, shot in the $#@$ and meds to take. Wheeeeeee
[16:36] <gary_poster> heh, hope it gets you well fast
[16:36] <rick_h__> gary_poster: rgr, working on it. Will be afk today. Sorry for the MIA
[16:36] <gary_poster> np at all.  good, go rest
[17:01] <frankban> gary_poster: book expensed :-)
[17:01] <gary_poster> frankban, :-) cool
[17:01] <hatch> ll
[17:01] <hatch> bah
[17:03] <hazmat> Makyo  sent via msg
[17:05] <Makyo> hazmat, ty
[17:09] <hatch> Makyo did you ever qa my fullscreen removal branch?
[17:09] <Makyo> hatch, Was going to wait on your verdict re: stuff in trunk fixed in this branch or another, but I can now.
[17:09] <hatch> well....the issues are in trunk
[17:10] <hatch> so just trying to fix anything small now
[17:11] <Makyo> hatch, okay, will in a sec; let me finish code review of frankban's branch
[17:11] <hatch> yeah sure, no rush
[17:22] <hatch> Makyo so it looks like the issues rick_h__  raised are more serious than a few cleanup lines in this branch unfortunately so I'll have to do them as a follow-up
[17:22] <Makyo> hatch, np, will QA in a sec with that in mind
[17:23] <hatch> just so you are aware, for whatever reason the browser is being dispatched twice which is causing the issues...heh, more double dispatch....lol
[17:27] <Makyo> Siiiigh~
[17:27] <hatch> this issue may have always been there we just never noticed it for whatever reason
[17:27] <hatch> I can't really see how any of the changes that have been made would cause this
[17:56] <Makyo> hatch, lgtm
[17:56] <hatch> thanks
[17:57] <bac> hi benji, did the charms get ingested?
[18:02] <benji> bac: yep; I'm now curious about a bundle potentially "stuck" in the bundle queue on prod
[18:03] <benji> every round now staging has one fewer baskets queue than prod and after the queue is drained there is still one basket in the basket queue; I'm wondering if the stuck one is Jorge's
[18:03] <bac> benji: what have the queues on prod looked like?  1164/22 now -- have then been going down?
[18:04] <bac> s/then/they
[18:04] <bac> s/then/they/
[18:04] <bac> gah
[18:04] <benji> bac: yep they are occasionally drained to 0/1
[18:04] <bac> oh, i bet that 1 is the problem!  :)
[18:04] <benji> yep
[18:16] <bac> benji: oh, with orangify you can now say 'source orangify stop'.  pretty darn neat.
[18:16] <bac> no more fumbling around looking for that pid file
[18:16] <benji> I saw that, nice addition
[18:17] <bac> my proudest moment of the morning
[18:17] <benji> :)
[18:52] <hatch> has anyone flown Delta internationally? 
[19:03] <BradCrittenden> hatch: yes
[19:03] <bac> gary_poster: call
[19:04] <bac> i meant, gary_poster call?
[19:04] <hatch> bac and?
[19:04] <gary_poster> bac almost
[19:04] <bac> hatch: and it was fine
[19:06] <bac> hatch: i think delta is a fine U.S. carrier.  i mean they don't do long hauls like the good asian carriers.
[19:07] <bac> hatch: i've flown delta on detroit-tokyo and survived.  where you going?
[20:38] <hatch> Makyo so those issues that rick_h__  found are actually in release :/
[20:38] <hatch> I -think- it has to do with the new token stuff
[20:39] <hatch> just going to bisect to try and roll back to find it
[20:40] <Makyo> I thought we knew that b/c it was in trunk? 
[20:40] <hatch> well trunk has a lot of other stuff in it too
[20:40] <hatch> could have been from anything really
[20:40] <Makyo> Yeah.
[20:40] <Makyo> Maybe come up with a test that fails for one of them to make it easier to bisect?
[20:46] <hatch> somehow I broke my repo, trying again, doesn't look like it was the token code though
[20:46] <hatch> hypothesis, incorrect
[20:46] <hatch> :)
[20:53] <gary_poster> older, hatch?
[20:53] <gary_poster> jujugui, looking for small but important review and qa https://github.com/juju/juju-gui/pull/18
[20:53] <Makyo> gary_poster, on it
[20:53] <gary_poster> thanks Makyo 
[20:54] <gary_poster> hatch, I mean, the problem is older than the token code?
[20:54] <hatch> it appears so, still bisecting
[20:55] <gary_poster> k
[20:56] <hatch> the double dispatch is caused by the app routing, then the login comes in as successful and then it routes again
[20:57] <hatch> double dispatching, but that doesn't appear to be the root cause
[20:57] <hatch> just another bug which we haven't noticed
[20:59] <gary_poster> ack
[20:59] <hatch> having to keep killing and restarting the app sure slows down bisection hah
[21:00]  * hatch crosses fingers it wasn't him
[21:03] <hatch> gary_poster according to bisect this is the culprit which broke the onboarding stuff....oops we should qa that :)
[21:04] <hatch> https://github.com/juju/juju-gui/commit/dec7617b378034fd93e41b1c382bb075c8b4978a
[21:04] <hatch> not entirely sure what the issue is
[21:04] <hatch> but at least we know where it is
[21:04] <gary_poster> hatch, so that is the token stuff
[21:05] <gary_poster> I just sent a note to rick that the git history is flatted rather than collapsed
[21:05] <gary_poster> which obscures the fact that this is part of the token commit
[21:05] <hatch> ohhh that's why these old ones are confusing
[21:05] <gary_poster> yup
[21:05] <gary_poster> I asked him if we can fix that
[21:05] <gary_poster> we'll see
[21:06] <gary_poster> flattened
[21:06] <hatch> ok cool, well hey, we just moved to git, and just used bisect to find an issue.....yay for the conversion :D
[21:06] <gary_poster> :-)
[21:09] <gary_poster> hatch, would it be helpful for me to look at that with you, or don't bother?
[21:09] <hatch> umm I think I'll be ok - I'm going to see if this is also the cause of the other redirection issues that just didn't happen to be noticed until I tried to muck with it recently
[21:09] <Makyo> Whoa.
[21:10] <hatch> it's very odd though that this commit is the culprit I don't see anything on face value that would cause the onboarding issue
[21:12] <Makyo> gary_poster, +1
[21:12] <gary_poster> hatch, currentUrl and onLogin seem most likely culprit in there to me.  It does affect the post-login redirect.
[21:12] <gary_poster> Makyo, thanks.  A bit scary, eh?
[21:12] <Makyo> gary_poster, yeah, thus the 'whoa' :T  Good catch, thanks for the fix!
[21:13] <gary_poster> thank you
[21:17] <hatch> gary_poster haha nice catch with #18 :)
[21:17] <hatch> *phew*
[21:17] <gary_poster> hatch, yeah, we kinda need that out soon. :-(
[21:21] <hatch> gary_poster fyi - here is the issue https://github.com/juju/juju-gui/commit/dec7617b378034fd93e41b1c382bb075c8b4978a#diff-2de9eda1655d499e934e7465e87ca0bdL951
[21:21] <hatch> super trivial fix 
[21:22] <hatch> or maybe not....I haven't used the token stuff
[21:22] <hatch> wana have a quick chat?
[21:22] <gary_poster> hatch, how so?  you don't have originalPath anymore
[21:22] <gary_poster> yeah sure
[21:32] <rick_h__> hatch: the double dispatch I think is from the url replace and re-dispath. It'll hit that twice
[21:34] <rick_h__> hatch: by bybassing the viewstate, nothing in subapp is watching/protecting against double dispatch
[21:34] <rick_h__> hatch: let me know if a hangout to chat will help, and maybe I'm wrong on this, but I think it's a simple fix.
[21:51] <hatch> rick_h__ hey we found the issue, it was actually located in a race condition in the login system that has been there......forever
[21:52] <rick_h__> hatch: oh well crap
[21:52] <hatch> yeah - it's a pretty odd edge case that has just been slowly exposed as we refactored code over the past month
[21:56] <hatch> jshint for chrome: http://mrpotes.github.io/jshint-extension/
[22:04]  * Makyo dogwalks, getting stomped.
[22:10] <huwshimi> Morning
[22:10] <rick_h__> hatch: ok cool. Never mind then. 
[22:10] <rick_h__> glad you got it figured out
[22:14] <gary_poster> morning huwshimi.  you have time to talk for 15 min now?  I have family obligations now
[22:15] <gary_poster> I mean later
[22:15] <gary_poster> bah
[22:15] <huwshimi> gary_poster: Sure, happy to talk anytime
[22:15] <gary_poster> reboot: morning huwshimi.  you have time to talk for 15 min now?  I have family obligations later :-P
[22:15] <huwshimi> gary_poster: OK!
[22:16] <gary_poster> ok thanks huwshimi .  https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.87augjtfc2pm1q1ilfuhencae0
[22:16] <huwshimi> gary_poster: Wait, did you need to reboot now, or in 15 minutes?
[22:16] <gary_poster> huwshimi, neither, I am just exhibiting poor communication skillz :-P
[22:16] <huwshimi> gary_poster: I know, I'm just teasing :)
[22:17] <gary_poster> lol
[22:20] <huwshimi> gary_poster: https://codereview.appspot.com/20120044/diff/1/bin/merge-files#newcode119
[22:32]  * gary_poster runs
[22:32] <gary_poster> bye all