| hatch | huwshimi hey you there? | 01:53 |
|---|---|---|
| huwshimi | hatch: Hey | 01:57 |
| hatch | hey check out my msg on your PR | 01:58 |
| huwshimi | hatch: Oh, strange... why would they be there? | 01:59 |
| hatch | I have no idea... | 02:00 |
| hatch | maybe your fork of juju-gui is out of date | 02:00 |
| huwshimi | $ git pull juju develop | 02:01 |
| huwshimi | From github.com:juju/juju-gui | 02:01 |
| huwshimi | * branch develop -> FETCH_HEAD | 02:01 |
| huwshimi | Already up-to-date. | 02:01 |
| hatch | then run git push | 02:01 |
| hatch | to update your fork | 02:01 |
| hatch | I really don't know why those old commits would be there anyways | 02:04 |
| hatch | it should have diff'd them out as far as I know | 02:04 |
| huwshimi | yeah, quite strange. | 02:06 |
| huwshimi | brb, lunch | 02:07 |
| hatch | Thanks for fixing that huwshimi but it looks like it's still pushing all of those commits in, maybe it needs a new PR? | 02:13 |
| huwshimi | hatch: Yeah, I'll delete and start again :) | 02:21 |
| hatch | sorry I wish I knew what the issue was so that we could avoid it in the future hah | 02:21 |
| hatch | huwshimi I was wondering if you play any games on your new machine? | 02:30 |
| huwshimi | hatch: Nah | 02:31 |
| hatch | You should go buy some and let me know how they run so I can decide on the computer to buy haha | 02:32 |
| huwshimi | :) | 02:32 |
| huwshimi | hatch: By the time you're ready the new Airs will be out anyway | 02:52 |
| hatch | lol! | 02:53 |
| huwshimi | hatch: https://github.com/juju/juju-gui/pull/17 | 02:58 |
| huwshimi | hatch: Lots of strangeness | 02:58 |
| === gary_poster|away is now known as gary_poster | ||
| * rick_h__ goes to doc office. bbl | 13:22 | |
| * gary_poster goes to son's prospective middle school. bbl | 13:32 | |
| benji | bac: Spads is asking me if the prod charmworld charm is "jc-charmworld/0"; do you know if that is right? | 13:53 |
| benji | bac: he may mean "service name" instead of "charm" | 13:54 |
| bac | benji: oh, the charm lives at bzr+ssh://bazaar.launchpad.net/~juju-jitsu/charms/precise/charmworld/trunk/ | 13:55 |
| benji | bac: do you know the service name is for the production charmwolrd? | 13:56 |
| bac | no. on staging it is just 'charmworld' | 13:56 |
| bac | charmworld/0 | 13:56 |
| frankban | hazmat: ping re: the deployer MP | 13:56 |
| bac | benji: if he has access to prod, a juju status should show. should only be one similar to charmworld | 13:57 |
| bac | benji: but webops grab everything and deploy locally so it could be there is a renaming | 13:58 |
| bac | let me look at the docs | 13:58 |
| benji | bac: do we run ES as its own charm in prod? | 13:58 |
| benji | s/charm/service/ | 13:59 |
| bac | benji: looking at https://wiki.canonical.com/InformationInfrastructure/WebOps/CDO/Charmworld it looks like jc-charmworld is correct | 13:59 |
| bac | benji: yes, i would think ES is its own service. can't see how it'd be otherwise. | 14:00 |
| bac | benji: the problem with understanding production is that page refers to scripts that we don't have access to, so it's kind of hard to see what's happening. | 14:01 |
| benji | yeah, really | 14:01 |
| benji | bac: it seems the staging.jujucharms.com environment is in a bad way: 2013-12-12 14:17:53,234 ERROR juju environment not found: is the environment bootstrapped? | 14:22 |
| benji | have you seen that before? Should I nuke it from orbit and rebuild from scratch? | 14:22 |
| bac | benji: but, but, it seems happy http://staging.jujucharms.com/heartbeat | 14:23 |
| bac | benji: where did that terrifying message come from? | 14:23 |
| benji | bac: happy except that it hasn't ingested since 2013-12-08 | 14:23 |
| benji | bac: from Spads in #webops | 14:23 |
| bac | benji: oh, well | 14:23 |
| bac | benji: huh. well it is definitely bootstrapped | 14:24 |
| benji | bac: I guess I need to figure out why my orangify script isn't working so I can poke on the staging env myself | 14:25 |
| bac | benji: i found some subdirectories in that branch that were created at some point and were messing up the connection. once i deleted them i could connect. if 'bzr st' shows anything weird (local, charms, etc) you may try the same. | 14:27 |
| bac | weird = unknown | 14:27 |
| benji | bac: staging.jujucharms.com is running on go-juju, right? | 14:27 |
| bac | yes | 14:27 |
| bac | benji: you may want to manually copy over the ssh credentials. the install script is broken in that it doesn't copy if they exist even though new ones may be available | 14:28 |
| bac | i've fixed it but haven't yet merged | 14:28 |
| bac | benji: i'm look at the staging charmworld/0 logs and don't see where that message from spads comes from. i wonder if his env is not right. there definitely is a problem with ingesting, and it looks like the job hasn't even tried to run since 9-Dec | 14:35 |
| bac | s/look/looking/ | 14:35 |
| benji | bac: I have determined that he was running py-juju | 14:35 |
| bac | oh | 14:35 |
| bac | that'll do it | 14:35 |
| benji | heh | 14:35 |
| bac | so, one red herring down | 14:36 |
| benji | bac: I almost have orangify working, when I run "nova list" I get juju-staging-machine-0 through juju-staging-machine-5, is that expected? | 14:36 |
| bac | curtis did fix a problem with mongo the other day | 14:36 |
| bac | benji: sounds about right | 14:36 |
| bac | benji: what does juju status do? | 14:36 |
| benji | bac: orangify still fails, this time with "client: fatal: failed to establish ssh session (2)" | 14:37 |
| benji | that is after printing: | 14:37 |
| benji | Found a compatible Juju. | 14:37 |
| benji | Orange credientials loaded | 14:37 |
| benji | JUJU_HOME set to /home/benji/workspace/staging-tools/staging-tools | 14:37 |
| benji | State server is at 10.55.32.48 | 14:37 |
| bac | benji: there's the problem | 14:38 |
| bac | credentials is misspelled | 14:38 |
| benji | heh | 14:40 |
| bac | benji: prob won't help but try pulling a new version of that branch | 14:46 |
| * benji tries | 14:46 | |
| bac | benji: so there are lots of problems from 9-Dec and before. that was when curtis got mongo working again | 14:46 |
| bac | but it doesn't look like ingest has run since then. it hasn't generated any errors since that date | 14:47 |
| benji | bac: the main thing I'm interested in is getting ingest to run again; my hypothesis is that it will then make the search results on staging match prod (which are wrong, but at least we'll have an idea of what is going on) | 14:47 |
| bac | right | 14:47 |
| benji | bac: same error after pull | 14:50 |
| bac | but no typo? | 14:50 |
| frankban | guihelp: I need two reviews for https://codereview.appspot.com/41350043 (quickstart). Is anyone available? | 14:56 |
| sinzui | hi bac | 15:02 |
| bac | sinzui: when mongo failed on staging it caused ingest to fail repeatedly and supervisord marked it as FATAL. do you know how to get it going again? | 15:02 |
| bac | sinzui: we're engaging spads in #webops on the issue though he isn't sure | 15:04 |
| sinzui | for staging? | 15:04 |
| sinzui | bac, he doesn't have access | 15:04 |
| sinzui | bac https://docs.google.com/a/canonical.com/document/d/190xKLPpEPSrlVix4D9eX_aTciiKFo12h0u7MpzqI28I/edit#heading=h.g8pxm6cjaq69 | 15:05 |
| sinzui | ^ we need to add a section about restarting the service, which I know but cannot type without doing it. | 15:05 |
| sinzui | per canonistack guidelines, there is a script in /srv that allows webops/devops to restart the services | 15:06 |
| bac | sinzui: can i just stop and restart supervisord? | 15:08 |
| sinzui | bac: I did this http://paste.ubuntu.com/6561801/ | 15:09 |
| sinzui | sudo /srv/deploymgr/charmworld restart | 15:10 |
| bac | sinzui: could we have just used the 'start_worker' target in the makefile? | 15:10 |
| bac | which i just found | 15:11 |
| bac | sinzui: i'll update the doc unless you want to | 15:12 |
| sinzui | bac: please do update the doc. | 15:12 |
| bac | thanks sinzui | 15:13 |
| rogpeppe | bac: sorry, was at lunch | 15:30 |
| bac | rogpeppe: ? | 15:30 |
| bac | rogpeppe: i didn't ping you, perhaps someone else did | 15:30 |
| bac | but i hope you had a good lunch | 15:30 |
| rogpeppe | bac: oh, sorry, i misread a remark of yours | 15:31 |
| rogpeppe | bac: it was just below one addressed to me | 15:31 |
| rogpeppe | bac: ignore :-) | 15:31 |
| bac | rogpeppe: np | 15:31 |
| bac | rogpeppe: was that a really late lunch or a really long one? :) | 15:32 |
| rogpeppe | bac: late | 15:32 |
| rogpeppe | bac: i often forget to have lunch until hunger prompts me :-) | 15:33 |
| benji | bac: the queue size is increasing, so it looks like a queue-clearing is in the future | 15:46 |
| bac | benji: ok | 15:47 |
| bac | benji: should be able to do: db['charm-queue'].remove() | 15:52 |
| bac | though it is kinda scary | 15:52 |
| benji | heh, yeah | 15:52 |
| bac | and db['basket-queue'] | 15:54 |
| * benji wonders if it is about team call time. | 15:57 | |
| hatch | jujugui call in 1 | 15:59 |
| hatch | benji are you actually going to make it on time today? :P | 15:59 |
| benji | I changed my reminder so I can't snooze it. That seem to help. | 15:59 |
| gary_poster | rick_h__, assuming you are still at dr | 16:00 |
| hazmat | hatch, Makyo how'd the rest of the meeting go | 16:05 |
| hatch | hazmat pretty good, hangout after standup to fill you in? | 16:05 |
| hazmat | sure | 16:06 |
| hazmat | hatch, sounds good | 16:06 |
| hatch | cool I'll ping when we're done | 16:06 |
| hatch | hazmat https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.j0rk5d371ph8331ijtf48t2uj0?authuser=1 just finishing up | 16:09 |
| bac | benji: so shall we do the cleanup? | 16:15 |
| benji | bac: yeah, we don't have a choice at this point, and since I haven't gotten orangify to work I need to enlist you if you can help | 16:16 |
| bac | benji: hangout? | 16:16 |
| benji | bac: sure, I'll create one | 16:16 |
| Makyo | hazmat, do we have a sprint authorisation number? | 16:30 |
| rick_h__ | gary_poster: back now, sorry for the delay. | 16:34 |
| gary_poster | np rick_h__ . how are you feeling? dr give you any meds? | 16:35 |
| rick_h__ | gary_poster: yep, shot in the $#@$ and meds to take. Wheeeeeee | 16:35 |
| gary_poster | heh, hope it gets you well fast | 16:36 |
| rick_h__ | gary_poster: rgr, working on it. Will be afk today. Sorry for the MIA | 16:36 |
| gary_poster | np at all. good, go rest | 16:36 |
| frankban | gary_poster: book expensed :-) | 17:01 |
| gary_poster | frankban, :-) cool | 17:01 |
| hatch | ll | 17:01 |
| hatch | bah | 17:01 |
| hazmat | Makyo sent via msg | 17:03 |
| Makyo | hazmat, ty | 17:05 |
| hatch | Makyo did you ever qa my fullscreen removal branch? | 17:09 |
| Makyo | hatch, Was going to wait on your verdict re: stuff in trunk fixed in this branch or another, but I can now. | 17:09 |
| hatch | well....the issues are in trunk | 17:09 |
| hatch | so just trying to fix anything small now | 17:10 |
| Makyo | hatch, okay, will in a sec; let me finish code review of frankban's branch | 17:11 |
| hatch | yeah sure, no rush | 17:11 |
| hatch | Makyo so it looks like the issues rick_h__ raised are more serious than a few cleanup lines in this branch unfortunately so I'll have to do them as a follow-up | 17:22 |
| Makyo | hatch, np, will QA in a sec with that in mind | 17:22 |
| hatch | just so you are aware, for whatever reason the browser is being dispatched twice which is causing the issues...heh, more double dispatch....lol | 17:23 |
| Makyo | Siiiigh~ | 17:27 |
| hatch | this issue may have always been there we just never noticed it for whatever reason | 17:27 |
| hatch | I can't really see how any of the changes that have been made would cause this | 17:27 |
| Makyo | hatch, lgtm | 17:56 |
| hatch | thanks | 17:56 |
| bac | hi benji, did the charms get ingested? | 17:57 |
| benji | bac: yep; I'm now curious about a bundle potentially "stuck" in the bundle queue on prod | 18:02 |
| benji | every round now staging has one fewer baskets queue than prod and after the queue is drained there is still one basket in the basket queue; I'm wondering if the stuck one is Jorge's | 18:03 |
| bac | benji: what have the queues on prod looked like? 1164/22 now -- have then been going down? | 18:03 |
| bac | s/then/they | 18:04 |
| bac | s/then/they/ | 18:04 |
| bac | gah | 18:04 |
| benji | bac: yep they are occasionally drained to 0/1 | 18:04 |
| bac | oh, i bet that 1 is the problem! :) | 18:04 |
| benji | yep | 18:04 |
| bac | benji: oh, with orangify you can now say 'source orangify stop'. pretty darn neat. | 18:16 |
| bac | no more fumbling around looking for that pid file | 18:16 |
| benji | I saw that, nice addition | 18:16 |
| bac | my proudest moment of the morning | 18:17 |
| benji | :) | 18:17 |
| hatch | has anyone flown Delta internationally? | 18:52 |
| BradCrittenden | hatch: yes | 19:03 |
| === BradCrittenden is now known as bac | ||
| bac | gary_poster: call | 19:03 |
| bac | i meant, gary_poster call? | 19:04 |
| hatch | bac and? | 19:04 |
| gary_poster | bac almost | 19:04 |
| bac | hatch: and it was fine | 19:04 |
| bac | hatch: i think delta is a fine U.S. carrier. i mean they don't do long hauls like the good asian carriers. | 19:06 |
| bac | hatch: i've flown delta on detroit-tokyo and survived. where you going? | 19:07 |
| hatch | Makyo so those issues that rick_h__ found are actually in release :/ | 20:38 |
| hatch | I -think- it has to do with the new token stuff | 20:38 |
| hatch | just going to bisect to try and roll back to find it | 20:39 |
| Makyo | I thought we knew that b/c it was in trunk? | 20:40 |
| hatch | well trunk has a lot of other stuff in it too | 20:40 |
| hatch | could have been from anything really | 20:40 |
| Makyo | Yeah. | 20:40 |
| Makyo | Maybe come up with a test that fails for one of them to make it easier to bisect? | 20:40 |
| hatch | somehow I broke my repo, trying again, doesn't look like it was the token code though | 20:46 |
| hatch | hypothesis, incorrect | 20:46 |
| hatch | :) | 20:46 |
| gary_poster | older, hatch? | 20:53 |
| gary_poster | jujugui, looking for small but important review and qa https://github.com/juju/juju-gui/pull/18 | 20:53 |
| Makyo | gary_poster, on it | 20:53 |
| gary_poster | thanks Makyo | 20:53 |
| gary_poster | hatch, I mean, the problem is older than the token code? | 20:54 |
| hatch | it appears so, still bisecting | 20:54 |
| gary_poster | k | 20:55 |
| hatch | the double dispatch is caused by the app routing, then the login comes in as successful and then it routes again | 20:56 |
| hatch | double dispatching, but that doesn't appear to be the root cause | 20:57 |
| hatch | just another bug which we haven't noticed | 20:57 |
| gary_poster | ack | 20:59 |
| hatch | having to keep killing and restarting the app sure slows down bisection hah | 20:59 |
| * hatch crosses fingers it wasn't him | 21:00 | |
| hatch | gary_poster according to bisect this is the culprit which broke the onboarding stuff....oops we should qa that :) | 21:03 |
| hatch | https://github.com/juju/juju-gui/commit/dec7617b378034fd93e41b1c382bb075c8b4978a | 21:04 |
| hatch | not entirely sure what the issue is | 21:04 |
| hatch | but at least we know where it is | 21:04 |
| gary_poster | hatch, so that is the token stuff | 21:04 |
| gary_poster | I just sent a note to rick that the git history is flatted rather than collapsed | 21:05 |
| gary_poster | which obscures the fact that this is part of the token commit | 21:05 |
| hatch | ohhh that's why these old ones are confusing | 21:05 |
| gary_poster | yup | 21:05 |
| gary_poster | I asked him if we can fix that | 21:05 |
| gary_poster | we'll see | 21:05 |
| gary_poster | flattened | 21:06 |
| hatch | ok cool, well hey, we just moved to git, and just used bisect to find an issue.....yay for the conversion :D | 21:06 |
| gary_poster | :-) | 21:06 |
| gary_poster | hatch, would it be helpful for me to look at that with you, or don't bother? | 21:09 |
| hatch | umm I think I'll be ok - I'm going to see if this is also the cause of the other redirection issues that just didn't happen to be noticed until I tried to muck with it recently | 21:09 |
| Makyo | Whoa. | 21:09 |
| hatch | it's very odd though that this commit is the culprit I don't see anything on face value that would cause the onboarding issue | 21:10 |
| Makyo | gary_poster, +1 | 21:12 |
| gary_poster | hatch, currentUrl and onLogin seem most likely culprit in there to me. It does affect the post-login redirect. | 21:12 |
| gary_poster | Makyo, thanks. A bit scary, eh? | 21:12 |
| Makyo | gary_poster, yeah, thus the 'whoa' :T Good catch, thanks for the fix! | 21:12 |
| gary_poster | thank you | 21:13 |
| hatch | gary_poster haha nice catch with #18 :) | 21:17 |
| hatch | *phew* | 21:17 |
| gary_poster | hatch, yeah, we kinda need that out soon. :-( | 21:17 |
| hatch | gary_poster fyi - here is the issue https://github.com/juju/juju-gui/commit/dec7617b378034fd93e41b1c382bb075c8b4978a#diff-2de9eda1655d499e934e7465e87ca0bdL951 | 21:21 |
| hatch | super trivial fix | 21:21 |
| hatch | or maybe not....I haven't used the token stuff | 21:22 |
| hatch | wana have a quick chat? | 21:22 |
| gary_poster | hatch, how so? you don't have originalPath anymore | 21:22 |
| gary_poster | yeah sure | 21:22 |
| rick_h__ | hatch: the double dispatch I think is from the url replace and re-dispath. It'll hit that twice | 21:32 |
| rick_h__ | hatch: by bybassing the viewstate, nothing in subapp is watching/protecting against double dispatch | 21:34 |
| rick_h__ | hatch: let me know if a hangout to chat will help, and maybe I'm wrong on this, but I think it's a simple fix. | 21:34 |
| hatch | rick_h__ hey we found the issue, it was actually located in a race condition in the login system that has been there......forever | 21:51 |
| rick_h__ | hatch: oh well crap | 21:52 |
| hatch | yeah - it's a pretty odd edge case that has just been slowly exposed as we refactored code over the past month | 21:52 |
| hatch | jshint for chrome: http://mrpotes.github.io/jshint-extension/ | 21:56 |
| * Makyo dogwalks, getting stomped. | 22:04 | |
| huwshimi | Morning | 22:10 |
| rick_h__ | hatch: ok cool. Never mind then. | 22:10 |
| rick_h__ | glad you got it figured out | 22:10 |
| gary_poster | morning huwshimi. you have time to talk for 15 min now? I have family obligations now | 22:14 |
| gary_poster | I mean later | 22:15 |
| gary_poster | bah | 22:15 |
| huwshimi | gary_poster: Sure, happy to talk anytime | 22:15 |
| gary_poster | reboot: morning huwshimi. you have time to talk for 15 min now? I have family obligations later :-P | 22:15 |
| huwshimi | gary_poster: OK! | 22:15 |
| gary_poster | ok thanks huwshimi . https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.87augjtfc2pm1q1ilfuhencae0 | 22:16 |
| huwshimi | gary_poster: Wait, did you need to reboot now, or in 15 minutes? | 22:16 |
| gary_poster | huwshimi, neither, I am just exhibiting poor communication skillz :-P | 22:16 |
| huwshimi | gary_poster: I know, I'm just teasing :) | 22:16 |
| gary_poster | lol | 22:17 |
| huwshimi | gary_poster: https://codereview.appspot.com/20120044/diff/1/bin/merge-files#newcode119 | 22:20 |
| * gary_poster runs | 22:32 | |
| gary_poster | bye all | 22:32 |
| === gary_poster is now known as gary_poster|away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!