[01:53] huwshimi hey you there? [01:57] hatch: Hey [01:58] hey check out my msg on your PR [01:59] hatch: Oh, strange... why would they be there? [02:00] I have no idea... [02:00] maybe your fork of juju-gui is out of date [02:01] $ git pull juju develop [02:01] From github.com:juju/juju-gui [02:01] * branch develop -> FETCH_HEAD [02:01] Already up-to-date. [02:01] then run git push [02:01] to update your fork [02:04] I really don't know why those old commits would be there anyways [02:04] it should have diff'd them out as far as I know [02:06] yeah, quite strange. [02:07] brb, lunch [02:13] Thanks for fixing that huwshimi but it looks like it's still pushing all of those commits in, maybe it needs a new PR? [02:21] hatch: Yeah, I'll delete and start again :) [02:21] sorry I wish I knew what the issue was so that we could avoid it in the future hah [02:30] huwshimi I was wondering if you play any games on your new machine? [02:31] hatch: Nah [02:32] You should go buy some and let me know how they run so I can decide on the computer to buy haha [02:32] :) [02:52] hatch: By the time you're ready the new Airs will be out anyway [02:53] lol! [02:58] hatch: https://github.com/juju/juju-gui/pull/17 [02:58] hatch: Lots of strangeness === gary_poster|away is now known as gary_poster [13:22] * rick_h__ goes to doc office. bbl [13:32] * gary_poster goes to son's prospective middle school. bbl [13:53] bac: Spads is asking me if the prod charmworld charm is "jc-charmworld/0"; do you know if that is right? [13:54] bac: he may mean "service name" instead of "charm" [13:55] benji: oh, the charm lives at bzr+ssh://bazaar.launchpad.net/~juju-jitsu/charms/precise/charmworld/trunk/ [13:56] bac: do you know the service name is for the production charmwolrd? [13:56] no. on staging it is just 'charmworld' [13:56] charmworld/0 [13:56] hazmat: ping re: the deployer MP [13:57] benji: if he has access to prod, a juju status should show. should only be one similar to charmworld [13:58] benji: but webops grab everything and deploy locally so it could be there is a renaming [13:58] let me look at the docs [13:58] bac: do we run ES as its own charm in prod? [13:59] s/charm/service/ [13:59] benji: looking at https://wiki.canonical.com/InformationInfrastructure/WebOps/CDO/Charmworld it looks like jc-charmworld is correct [14:00] benji: yes, i would think ES is its own service. can't see how it'd be otherwise. [14:01] benji: the problem with understanding production is that page refers to scripts that we don't have access to, so it's kind of hard to see what's happening. [14:01] yeah, really [14:22] bac: it seems the staging.jujucharms.com environment is in a bad way: 2013-12-12 14:17:53,234 ERROR juju environment not found: is the environment bootstrapped? [14:22] have you seen that before? Should I nuke it from orbit and rebuild from scratch? [14:23] benji: but, but, it seems happy http://staging.jujucharms.com/heartbeat [14:23] benji: where did that terrifying message come from? [14:23] bac: happy except that it hasn't ingested since 2013-12-08 [14:23] bac: from Spads in #webops [14:23] benji: oh, well [14:24] benji: huh. well it is definitely bootstrapped [14:25] bac: I guess I need to figure out why my orangify script isn't working so I can poke on the staging env myself [14:27] benji: i found some subdirectories in that branch that were created at some point and were messing up the connection. once i deleted them i could connect. if 'bzr st' shows anything weird (local, charms, etc) you may try the same. [14:27] weird = unknown [14:27] bac: staging.jujucharms.com is running on go-juju, right? [14:27] yes [14:28] benji: you may want to manually copy over the ssh credentials. the install script is broken in that it doesn't copy if they exist even though new ones may be available [14:28] i've fixed it but haven't yet merged [14:35] benji: i'm look at the staging charmworld/0 logs and don't see where that message from spads comes from. i wonder if his env is not right. there definitely is a problem with ingesting, and it looks like the job hasn't even tried to run since 9-Dec [14:35] s/look/looking/ [14:35] bac: I have determined that he was running py-juju [14:35] oh [14:35] that'll do it [14:35] heh [14:36] so, one red herring down [14:36] bac: I almost have orangify working, when I run "nova list" I get juju-staging-machine-0 through juju-staging-machine-5, is that expected? [14:36] curtis did fix a problem with mongo the other day [14:36] benji: sounds about right [14:36] benji: what does juju status do? [14:37] bac: orangify still fails, this time with "client: fatal: failed to establish ssh session (2)" [14:37] that is after printing: [14:37] Found a compatible Juju. [14:37] Orange credientials loaded [14:37] JUJU_HOME set to /home/benji/workspace/staging-tools/staging-tools [14:37] State server is at 10.55.32.48 [14:38] benji: there's the problem [14:38] credentials is misspelled [14:40] heh [14:46] benji: prob won't help but try pulling a new version of that branch [14:46] * benji tries [14:46] benji: so there are lots of problems from 9-Dec and before. that was when curtis got mongo working again [14:47] but it doesn't look like ingest has run since then. it hasn't generated any errors since that date [14:47] bac: the main thing I'm interested in is getting ingest to run again; my hypothesis is that it will then make the search results on staging match prod (which are wrong, but at least we'll have an idea of what is going on) [14:47] right [14:50] bac: same error after pull [14:50] but no typo? [14:56] guihelp: I need two reviews for https://codereview.appspot.com/41350043 (quickstart). Is anyone available? [15:02] hi bac [15:02] sinzui: when mongo failed on staging it caused ingest to fail repeatedly and supervisord marked it as FATAL. do you know how to get it going again? [15:04] sinzui: we're engaging spads in #webops on the issue though he isn't sure [15:04] for staging? [15:04] bac, he doesn't have access [15:05] bac https://docs.google.com/a/canonical.com/document/d/190xKLPpEPSrlVix4D9eX_aTciiKFo12h0u7MpzqI28I/edit#heading=h.g8pxm6cjaq69 [15:05] ^ we need to add a section about restarting the service, which I know but cannot type without doing it. [15:06] per canonistack guidelines, there is a script in /srv that allows webops/devops to restart the services [15:08] sinzui: can i just stop and restart supervisord? [15:09] bac: I did this http://paste.ubuntu.com/6561801/ [15:10] sudo /srv/deploymgr/charmworld restart [15:10] sinzui: could we have just used the 'start_worker' target in the makefile? [15:11] which i just found [15:12] sinzui: i'll update the doc unless you want to [15:12] bac: please do update the doc. [15:13] thanks sinzui [15:30] bac: sorry, was at lunch [15:30] rogpeppe: ? [15:30] rogpeppe: i didn't ping you, perhaps someone else did [15:30] but i hope you had a good lunch [15:31] bac: oh, sorry, i misread a remark of yours [15:31] bac: it was just below one addressed to me [15:31] bac: ignore :-) [15:31] rogpeppe: np [15:32] rogpeppe: was that a really late lunch or a really long one? :) [15:32] bac: late [15:33] bac: i often forget to have lunch until hunger prompts me :-) [15:46] bac: the queue size is increasing, so it looks like a queue-clearing is in the future [15:47] benji: ok [15:52] benji: should be able to do: db['charm-queue'].remove() [15:52] though it is kinda scary [15:52] heh, yeah [15:54] and db['basket-queue'] [15:57] * benji wonders if it is about team call time. [15:59] jujugui call in 1 [15:59] benji are you actually going to make it on time today? :P [15:59] I changed my reminder so I can't snooze it. That seem to help. [16:00] rick_h__, assuming you are still at dr [16:05] hatch, Makyo how'd the rest of the meeting go [16:05] hazmat pretty good, hangout after standup to fill you in? [16:06] sure [16:06] hatch, sounds good [16:06] cool I'll ping when we're done [16:09] hazmat https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.j0rk5d371ph8331ijtf48t2uj0?authuser=1 just finishing up [16:15] benji: so shall we do the cleanup? [16:16] bac: yeah, we don't have a choice at this point, and since I haven't gotten orangify to work I need to enlist you if you can help [16:16] benji: hangout? [16:16] bac: sure, I'll create one [16:30] hazmat, do we have a sprint authorisation number? [16:34] gary_poster: back now, sorry for the delay. [16:35] np rick_h__ . how are you feeling? dr give you any meds? [16:35] gary_poster: yep, shot in the $#@$ and meds to take. Wheeeeeee [16:36] heh, hope it gets you well fast [16:36] gary_poster: rgr, working on it. Will be afk today. Sorry for the MIA [16:36] np at all. good, go rest [17:01] gary_poster: book expensed :-) [17:01] frankban, :-) cool [17:01] ll [17:01] bah [17:03] Makyo sent via msg [17:05] hazmat, ty [17:09] Makyo did you ever qa my fullscreen removal branch? [17:09] hatch, Was going to wait on your verdict re: stuff in trunk fixed in this branch or another, but I can now. [17:09] well....the issues are in trunk [17:10] so just trying to fix anything small now [17:11] hatch, okay, will in a sec; let me finish code review of frankban's branch [17:11] yeah sure, no rush [17:22] Makyo so it looks like the issues rick_h__ raised are more serious than a few cleanup lines in this branch unfortunately so I'll have to do them as a follow-up [17:22] hatch, np, will QA in a sec with that in mind [17:23] just so you are aware, for whatever reason the browser is being dispatched twice which is causing the issues...heh, more double dispatch....lol [17:27] Siiiigh~ [17:27] this issue may have always been there we just never noticed it for whatever reason [17:27] I can't really see how any of the changes that have been made would cause this [17:56] hatch, lgtm [17:56] thanks [17:57] hi benji, did the charms get ingested? [18:02] bac: yep; I'm now curious about a bundle potentially "stuck" in the bundle queue on prod [18:03] every round now staging has one fewer baskets queue than prod and after the queue is drained there is still one basket in the basket queue; I'm wondering if the stuck one is Jorge's [18:03] benji: what have the queues on prod looked like? 1164/22 now -- have then been going down? [18:04] s/then/they [18:04] s/then/they/ [18:04] gah [18:04] bac: yep they are occasionally drained to 0/1 [18:04] oh, i bet that 1 is the problem! :) [18:04] yep [18:16] benji: oh, with orangify you can now say 'source orangify stop'. pretty darn neat. [18:16] no more fumbling around looking for that pid file [18:16] I saw that, nice addition [18:17] my proudest moment of the morning [18:17] :) [18:52] has anyone flown Delta internationally? [19:03] hatch: yes === BradCrittenden is now known as bac [19:03] gary_poster: call [19:04] i meant, gary_poster call? [19:04] bac and? [19:04] bac almost [19:04] hatch: and it was fine [19:06] hatch: i think delta is a fine U.S. carrier. i mean they don't do long hauls like the good asian carriers. [19:07] hatch: i've flown delta on detroit-tokyo and survived. where you going? [20:38] Makyo so those issues that rick_h__ found are actually in release :/ [20:38] I -think- it has to do with the new token stuff [20:39] just going to bisect to try and roll back to find it [20:40] I thought we knew that b/c it was in trunk? [20:40] well trunk has a lot of other stuff in it too [20:40] could have been from anything really [20:40] Yeah. [20:40] Maybe come up with a test that fails for one of them to make it easier to bisect? [20:46] somehow I broke my repo, trying again, doesn't look like it was the token code though [20:46] hypothesis, incorrect [20:46] :) [20:53] older, hatch? [20:53] jujugui, looking for small but important review and qa https://github.com/juju/juju-gui/pull/18 [20:53] gary_poster, on it [20:53] thanks Makyo [20:54] hatch, I mean, the problem is older than the token code? [20:54] it appears so, still bisecting [20:55] k [20:56] the double dispatch is caused by the app routing, then the login comes in as successful and then it routes again [20:57] double dispatching, but that doesn't appear to be the root cause [20:57] just another bug which we haven't noticed [20:59] ack [20:59] having to keep killing and restarting the app sure slows down bisection hah [21:00] * hatch crosses fingers it wasn't him [21:03] gary_poster according to bisect this is the culprit which broke the onboarding stuff....oops we should qa that :) [21:04] https://github.com/juju/juju-gui/commit/dec7617b378034fd93e41b1c382bb075c8b4978a [21:04] not entirely sure what the issue is [21:04] but at least we know where it is [21:04] hatch, so that is the token stuff [21:05] I just sent a note to rick that the git history is flatted rather than collapsed [21:05] which obscures the fact that this is part of the token commit [21:05] ohhh that's why these old ones are confusing [21:05] yup [21:05] I asked him if we can fix that [21:05] we'll see [21:06] flattened [21:06] ok cool, well hey, we just moved to git, and just used bisect to find an issue.....yay for the conversion :D [21:06] :-) [21:09] hatch, would it be helpful for me to look at that with you, or don't bother? [21:09] umm I think I'll be ok - I'm going to see if this is also the cause of the other redirection issues that just didn't happen to be noticed until I tried to muck with it recently [21:09] Whoa. [21:10] it's very odd though that this commit is the culprit I don't see anything on face value that would cause the onboarding issue [21:12] gary_poster, +1 [21:12] hatch, currentUrl and onLogin seem most likely culprit in there to me. It does affect the post-login redirect. [21:12] Makyo, thanks. A bit scary, eh? [21:12] gary_poster, yeah, thus the 'whoa' :T Good catch, thanks for the fix! [21:13] thank you [21:17] gary_poster haha nice catch with #18 :) [21:17] *phew* [21:17] hatch, yeah, we kinda need that out soon. :-( [21:21] gary_poster fyi - here is the issue https://github.com/juju/juju-gui/commit/dec7617b378034fd93e41b1c382bb075c8b4978a#diff-2de9eda1655d499e934e7465e87ca0bdL951 [21:21] super trivial fix [21:22] or maybe not....I haven't used the token stuff [21:22] wana have a quick chat? [21:22] hatch, how so? you don't have originalPath anymore [21:22] yeah sure [21:32] hatch: the double dispatch I think is from the url replace and re-dispath. It'll hit that twice [21:34] hatch: by bybassing the viewstate, nothing in subapp is watching/protecting against double dispatch [21:34] hatch: let me know if a hangout to chat will help, and maybe I'm wrong on this, but I think it's a simple fix. [21:51] rick_h__ hey we found the issue, it was actually located in a race condition in the login system that has been there......forever [21:52] hatch: oh well crap [21:52] yeah - it's a pretty odd edge case that has just been slowly exposed as we refactored code over the past month [21:56] jshint for chrome: http://mrpotes.github.io/jshint-extension/ [22:04] * Makyo dogwalks, getting stomped. [22:10] Morning [22:10] hatch: ok cool. Never mind then. [22:10] glad you got it figured out [22:14] morning huwshimi. you have time to talk for 15 min now? I have family obligations now [22:15] I mean later [22:15] bah [22:15] gary_poster: Sure, happy to talk anytime [22:15] reboot: morning huwshimi. you have time to talk for 15 min now? I have family obligations later :-P [22:15] gary_poster: OK! [22:16] ok thanks huwshimi . https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.87augjtfc2pm1q1ilfuhencae0 [22:16] gary_poster: Wait, did you need to reboot now, or in 15 minutes? [22:16] huwshimi, neither, I am just exhibiting poor communication skillz :-P [22:16] gary_poster: I know, I'm just teasing :) [22:17] lol [22:20] gary_poster: https://codereview.appspot.com/20120044/diff/1/bin/merge-files#newcode119 [22:32] * gary_poster runs [22:32] bye all === gary_poster is now known as gary_poster|away