[09:09] not sure if I was disconnected earlier [09:09] morning fwereade: I am investigating the path to remove store code from juju-core, in order to create its own git repo. given http://pastebin.ubuntu.com/7492017/ , it seems we have to do that in many incremental steps. at first sight, it seems the utils and testing packages should be migrated first, then schema, then charm and finally store. What do you think? I am not sure how this kind of work has been handled be [09:09] fore. I imagine something like 1) lock the package (email?), 2) create the git repo and 3) update core (remove the package, fix imports, fix dependencies file). [09:37] frankban, it's a bit of a perfect storm of cleaners, decorators, furniture deliveries and cleaners here atm, and I'm just making a sandwich; I will be with you shortly [09:37] fwereade: :-) np and thank you [10:25] frankban, ok, sorry, I think I can work again now [10:25] frankban, are you free for a g+? [10:25] fwereade: cool [10:26] fwereade: sure [10:27] frankban, https://plus.google.com/hangouts/_/g5z557wyte4tlcapwfeqetgyzya?hl=en [10:28] fwereade: This party is over... https://plus.google.com/hangouts/_/gusnuo5mlb37av6rppczm72heea?hl=en ? [10:29] frankban, I'm the first one there apparently [11:36] frankban: you guys make good progress there? [11:36] rick_h_: yes, I'll send an email about current state [11:37] frankban: awesome thanks [11:46] rick_h_, frankban: can either of you take a look at https://github.com/juju/juju-gui/pull/330 [11:48] jcsackett: sure thing, I can in a couple of min [11:49] rick_h_: thanks. [11:52] jcsackett: can you peek at huw's again this morning in trade? [11:55] rick_h_: already looking. [11:59] * rick_h_ needs coffee for review time [12:00] morning [12:02] rick_h_: email sent [12:03] * frankban lunches [12:07] rick_h_: don't know if you started looking at mine; there was a merge conflict with develop that i've resolved. [12:07] jcsackett: awesome [12:20] rick_h_: what time does huw start? [12:21] jcsackett: he's usually around 7pm or so [12:21] ok. i have questions about his branch that keep me from just stamping it, but don't want to block him--i'll try to be on IRC for a bit then. [12:22] we need another JS coder on the team in his timezone. :p [12:22] jcsackett: sounds good [12:23] heh, well I can help look as well if needed. I'll be around [12:35] bac, rick_h_, jcsackett review s'il vous plait http://bit.ly/1h3Sw5i :) [12:36] if someone has time [12:36] redir: already on it [12:37] bac: great. I just pushed up a commit trimming out ununsed helper bits in the tests [12:38] doh [12:38] ok [13:09] frankban: replied and passed along with one request to pass that through the juju-dev list. [13:10] frankban: I've got a card of size 10 "Pull our deps from core into sharable libraries" to use to create cards for starting that work [13:10] frankban: I'll create a doc from your email and we can start to create work items out of it from there. [13:11] redir: review done, and approved with changes. please be sure to see the in-line comments, which may be a bit funky since you landed another rev. there is a drop down to show the different diffs. [13:11] QA was good too [13:12] bac just reading through it in email. updates forthcoming. [13:12] think i have a couple unused imports too. [13:12] redir: lint should catch those [13:12] it did, but I forgot about them in the 5 minutes it took to run the tests...:( [13:15] redir: that is one benefit of lbox, it runs 'make check' before letting you proceed. [13:17] so maybe I can use it [13:17] as @not canonical [13:17] I see wayne using it and asked him. He said he was using non canonical account to do it:( [13:18] bac: did we just do a deploy of charmworld on production? [13:19] jcsackett: i hope not [13:19] * redir blinks [13:19] jcsackett: no, still on 508 http://manage.jujucharms.com/heartbeat [13:19] bac: mbruzek is reporting intermittent oddities with the API. he looked for manage.jujucharms.com/api/3/charm/precise/openerp-server, and got no data. but it's showing data now. [13:20] http://manage.jujucharms.com/heartbeat is showing r508 [13:20] jcsackett: we may be having data center problems. i'm getting intermittent failures from the LP api [13:20] bac: ok. [13:20] i shall mozy over to webops and mention it [13:20] bac: oh, huh, i think a server went down? start time on heartbeat is 2 minutes ago. [13:22] jcsackett: hmm. i wonder if charmworld hit the LP API failure and it threw an exception causing it to restart [13:23] actually, no, it should only affect the ingest process [13:23] rick_h_: thanks for creating the google doc and the card. juju-dev ml is internal, right? [13:24] frankban: no, i think it's public. I don't think there's anything we need to worry about in this plan/doc though or am I missing it? [13:24] as long as it doesn't sing the blues [13:27] jcsackett: actually i see m.j.c and qa.m.j.c servers restarting a lot. webops says LP is undergoing scheduled maint which has pushed the DB load very high. [13:27] interesting. [13:27] i'll poke around QA and see if the logs say anything useful [13:27] redir, rick_h_: we should also investigate a way to migrate a partial bzr history to git [13:28] jcsackett: do you have access to QA on canonistack? [13:28] frankban: yea, they've pulled out some other tools. Make that part of the email? [13:28] bac: i do not. [13:28] frankban: I guess loggo/errgo or whatever was out then in and then out again. Maybe they don't have experience with it [13:28] jcsackett: oh. you should grab the staging tools and get set up, if you want. i can walk you through it. [13:29] bac: perhaps this afternoon? i don't believe we're blocking anyone now on investigating the charmworld goof. [13:37] frankban: for each of these migrate cards. I'm going to set them at a size of 3, but we'll have to get a feel for how long it takes. So don't worry about it too much. [13:37] frankban: I figure after a couple we'll have a better idea [13:37] rick_h_: sounds good [13:38] jcsackett: sure, whenever [13:41] bac: cool. [13:41] redir: so let's get you in the loop on the deps and maybe you can help with them this week. [13:42] heh, kanban cards update, lots more than the budgeted 10, wheeee [13:42] jujugui: can i get a review of https://github.com/juju/juju-gui/pull/330 [13:42] oops, old link. [13:42] Looking… [13:42] heh, yea merged :P [13:42] kadams54: https://github.com/juju/juju-gui/pull/332 [13:42] there's the new link. [13:43] sorry. :p [13:43] np [13:47] jcsackett: looks good. [13:47] kadams54: thanks. :) [13:55] kadams54: how goes, how's your test card? I went to look at it yesterday but didn't see the new revisions/changes. [13:55] kadams54: is that wip or need a review? [13:57] WIP - hatch and I need to chat about how to handle data-ids before it'll be ready for a review/merge. Currently tracking down/cleaning up some leaky tests in that whole test suite. [13:58] kadams54: gotcha ok [13:58] Makyo: can I ask you the favor of setting up the JS code question for the interview this afternoon please? [13:59] jcsackett: lint issues in test run fyi [13:59] rick_h_: ah, thanks. [14:00] kadams54 wasssssuuupppp? [14:01] hatch: chat now or after standup? [14:01] a few mins, just going to move into the office [14:05] kadams54 https://plus.google.com/hangouts/_/g7is4cqz7uzgb4wawoc6n75yaia?hl=en === hatch__ is now known as hatch [14:20] bac, wanna look at http://bazaar.launchpad.net/~reedobrien/charmworld/es-migration/revision/517 [14:20] need anything else before marking approved? [14:20] redir: see PM please [14:27] * rick_h_ takes a quick walk the dog break [14:32] redir: looks good [14:32] bac: tx marking as approved. [14:32] cool [14:33] redir: let's see if it gets deployed to qa by itself. i suspect not. [14:35] bac because? [14:35] redir: b/c jenkins lander is a lazy bum [14:39] mmm [14:44] holy so many emails batman [14:50] jujugui call in 10. [14:51] rick_h_: talk post standup about pyramid-y stuff? [14:51] redir: you see my msgs in that other channel? [14:52] jcsackett: souds good [14:54] curious, why didn't we choose twisted or tornado? [14:54] frankban: running into a problem where test code is causing problems here: https://github.com/juju/juju-gui/blob/develop/app/models/models.js#L712 [14:54] The test code passes in a machine ID of 0 [14:55] And the function returns the list of unplaced units, rather than the list of units associated with machine 0. [14:55] in my research the hype machine likes tornado it seems [14:56] of course popularity !== best :) [14:56] hatch: tornado is newer, smaller, more web centric [14:56] The hype machine also liked Jersey Shore [14:56] hatch: and it came from facebook, so it gets social media bonus points [14:56] bac which channel [14:56] redir: private channel [14:57] kadams54: I'll take a look, FWIW a machine id should always be a string [14:57] kadams54: so filterByMachine('0') [14:57] Ah yes, I'll fix that in the test code… [14:58] And tests pass! Yay. [14:59] yay! [14:59] :-) [15:00] jujugui call in 1, kanban and all that [15:01] jcsackett: ^ [15:01] jumping in now. [15:13] lol [15:13] when all else fails, kill the tab [15:13] lol [15:15] rick_h_, http://collabedit.com/h4nym [15:15] ty Makyo [15:16] binary yo! [15:22] rick_h_: looks like forking a private remains private. [15:22] so we can delete testing private and have ourselves a lovely time. [15:22] jcsackett: wrong channel bro [15:22] :) [15:23] bloody hell. [15:23] lol [15:23] at least it wasn't a key or anything :) [15:23] please take my password kthx! [15:24] * jcsackett briefly contemplates the old "irc makes your password *****" joke [15:24] frankban so I'm looking at calling env.add_unit() instead of placeUnit() but the add_unit call creates a new unit which requires us deleting the one already created for the unplaced-unit [15:25] you've got to wrap it in tags for that to work [15:25] lol [15:25] and only works for html enabled irc readers [15:26] frankban and manually deleting the old unit doesn't follow the same pattern for every other unplaced unit interaction....which I was trying to avoid [15:26] hatch: ok, let me look at the placeUnit code [15:28] frankban https://gist.github.com/d1b7a1250b148a6f7051 [15:28] this is the diff of what I've got...but it doesn't feel right either because it's the only place which manually modifies the records' command [15:30] hatch: that seems fine to me: if the machine does not exist, we register the corresponding addMachines parent so that the record can be updated later. Otherwise we update the record right now [15:31] hatch: there are already many places where records are mutated [15:32] frankban well those other places are internal to the 'lazy' commands [15:32] I'm concerned that this ecs code is doing a lot of 'magic' and will be very hard to follow for the 'uninitiated' [15:33] this will be like our own shibboleth lol [15:33] hatch: lol [15:34] * frankban adds an irc beep for "shibboleth" [15:34] ok I'll continue down this path and once we are close to being done with the ecs we can figure out how to make it easier to follow for others [15:34] lol [15:34] such a good word [15:34] jcsackett, rick_h_ : is there any more work being done on improving tests for machines? [15:35] kadams54: the only thing left is my functional test branch, which is only in the containers column describe block. [15:35] kadams54: at least insofar as active work. [15:35] OK [15:35] Some of the new tests around machines deal with functions that don't exist any more in my branch :-\ [15:36] kadams54: kill 'em then. [15:36] hatch: I see two possible improvement paths: short term, we should document the ecs framework, so that those things sound less magic. long term, we can investigate other solutions, like automatically generating the changeset from the ghost models in our database, e.g. we have these ghost entities, make them real [15:36] as long as what's replaced things is covered. [15:37] frankban yeah we are going to need to do the ghost stuff fairly soon when we have to update the inspector to represent the changes made in the ECS [15:37] cool [15:38] jcsackett: I think we may have coverage out the whazoo :-) [15:40] kadams54: that was my hope. [15:40] let this be a shining beacon from which we begin improving coverage elsewhere. [15:47] bac the non-promoted juju-gui charm is in the 'popular' list now [15:47] we probably shouldn't promote that one over the promoted version [15:50] hatch, bac: uhm, it seems the recommended juju-gui cannot be found: https://jujucharms.com/sidebar/search/?text=juju-gui [15:50] hatch: i'm looking to see what you're talking about [15:51] hatch: the link frankban posted has none that are under Recommended [15:52] hmm, now I can't reproduce it. bac under 'popular' the first result was the juju-gui but one that wasn't promulgated [15:52] it was the juju-gui user one [15:53] so where is the real one? :) [15:53] it seems the results are being updated while we talk [15:53] oh.... that's odd [15:53] heh [15:54] hatch: featured=promulgated. popular is something else [15:54] bac oh ok....so we are still left with the lack of promulgated juju-gui charm [15:54] unless it's doing some re-indexing or something [15:55] hatch: i think so. i'd work with the charmers to get that done as they manage all of the charm promulgationing [15:56] oh there it is now [15:56] it must have been being updated or something [15:59] sorry to bother you :) [16:02] * rick_h_ runs to get lunch between meetings. 30min gap wooo! [16:05] frankban I added a container using app.db.machines.add({id: '0/lxc/0'}) which ends up in the db, when I then place a unit on it, and deploy. Accessing the machine db from the fakebackend.js the machine I added is no longer in the db [16:05] any idea why this would be? [16:06] oh.... [16:06] app.db.machines and this.db.machines are different db's [16:08] hatch: yes, the fakebackend should simulate incoming data from the watcher [16:08] something about changes IIRC [16:09] yeah ok - I was just testing the new placeunit and ran into this. So for testing I'm just going to make fakebackend accessible on the app instance [16:09] so I can add to both db's [16:10] nice, that worked [16:11] bug....fixed [16:12] \o/ [16:13] oh yeah there are no placeUnit tests [16:13] lol damn [16:26] frankban could you do me a favour and write a more detailed "why" comment for https://github.com/juju/juju-gui/blob/develop/app/utils/environment-change-set.js#L582-L584 so I can include it in this branch - I'm not 100% clear on it myself [16:30] hatch: when placeUnit is called, the unit could have been already placed to a ghost machine. In that case the corresponding addMachines parent has been set to the addUnit record. When we place the unit again, that parent is no longer relevant and must be removed. At this point either the unit is re-placed to a ghost, in which case a new parent is added, or an existing machine will host the unit, in which case we do [16:30] n't need a parent at all [16:31] thanks that REALLY clears it up :) I'll add that comment to the code [16:31] hatch: kadams54 Makyo call time, https://docs.google.com/a/canonical.com/document/d/1mXy66SH05B1U-E_vxGDP_Ig4Ez50VvsmZ7sMW2jiLSI/edit is the doc [17:20] hatch: come back [17:20] you can't leave us yet :P [17:28] rick_h_ for my exploratory day I filed a bug....not sure how the heck this one happened heh [17:28] hatch cool, saw it in my email but not looked at it yet [17:29] will triage and add a card shortly and thanks for running with QA day [17:29] I forgot to bring it up on the standup again [17:29] :) [17:29] * rick_h_ ponders adding a non-moving card for qa on the board [18:00] hi jcsackett, got a sec? [18:02] rick_h_: heads up, manage.j.c is restarting the server every few minutes and i've been trying to find out why. also the charmworld lander cannot update qa.mjc and i cannot use juju to communicate with it, so it may need to be torn down and redeployed. [18:03] bac: ugh and double ugh. Thanks for the heads up. Let me know if there's something we can do to help debug or anything. [18:14] oh i'm going to miss canonicaladmin... [18:14] lol [18:14] I feel like we should be throwing a party [18:15] though I guess we've not seen the new system yet [18:15] and I guess expenses will still live on canonicaladmin [18:15] which is probably the worst part of it [18:15] oh, i misread that part. [18:16] yeah I'm curious as to the advantage for us now having to use two systems [18:16] hatch: twice the quirks and work-arounds to remember [18:16] well hopefully I can more easily see the filed time off of the team [18:16] that's a pita to do every 2wk cycle [18:16] lol [18:16] woot [18:16] rick_h_ yeah that would be nice [18:18] I don't think i've ever used hr software that wasn't a total pile [18:18] here is to hoping this one bucks the trend [18:26] hatch: https://github.com/juju/juju-gui/pull/325 is ready for QA [18:26] hatch: If you're interested, of course :-) [18:28] oh you're so thoughtful :) [18:28] I can do it but I'd like to get these placeUnit tests done first [18:28] so i'll leave it up for grabs until then [18:29] for some reason I can't capture something which throws :/ [18:30] ugh, tornado warning. you should get hurricanes or tornadoes but not both. [18:30] hah, "nasty wind-weather your way" [18:30] problem solved, it's all in the marketing [18:31] guihelp: Looking for QA on https://github.com/juju/juju-gui/pull/325 [18:35] hey rick_h_ it looks like you added the beaker support to charmworld. you have any recollection? [18:35] bac: yep, kindof [18:35] rick_h_: specifically, app-exception.log fills up with warnings that mongo already exists when beaker init is called. it looks like the stanza in setup.py for beaker may be causing us to try to load the namespace twice. [18:35] bac: it does the session management stuff [18:36] bac: hmm, do we store the beaker session in mongo or just use filed backed? [18:36] mongo [18:36] I'd switch to just file based and drop mongo? [18:36] traffic isn't nearly high enough to deal with it tbh [18:37] http://beaker.readthedocs.org/en/latest/configuration.html [18:37] rick_h_: well, i think i'd like to just remove the part from setup.py as it seems to cause trouble. verifying that it isn't needed. was curious if you knew that it was needed. [18:37] bac: no, without going through it don't recall the setup.py part or what that would be setup that way. [18:37] * rick_h_ goes to look at charmworld setup.py [18:38] bac: hmm yea. I guess it's one way of configuring beaker. I bet the same config can go into the app __init__.py or something [18:39] rick_h_: i think it isn't needed. with it in, pkg_resources.iter_entry_points('beaker.backends') lists mongo twice. when i remove it from setup.py it is listed just once. [18:40] bac: cool [18:41] bac: yea, if it's config'd correctly without the setup.py INI block info then peachy. I bet it was tweaked at some point from some copy/paste instructions [18:45] jcsackett: kadams54 can you guys swap reviews? [18:48] Stepping out for a short appointment [18:52] rick_h_: yup. [18:53] kadams54: https://github.com/juju/juju-gui/pull/333 [18:54] bac: as soon as i do this review, i can chat with you if you like. sorry i missed your ping earlier. i was at the mechanics. thought i had pinged in the channel to say so, but evidently not. [18:54] jcsackett: taking a look [18:55] jcsackett: I see merge conflicts in my future :-) [18:55] jcsackett: fortunately minor… [18:56] kadams54: just have to race him [18:56] the carrot to get up for review fast and furious ) [18:57] Au contraire, it's incentive to review his very, very slowly ;-) [18:59] yeah, we're in a standoff here. [18:59] :p [19:00] if we're both being honest though, there's much less to my PR. :p [19:00] jcsackett: Yup. That's why it's already marked as "LGTM" :-) [19:00] I did ask that one formatting change be dropped, which will be one less merge conflict for me. [19:01] ok, gotta pick up kiddos from school. Will be back in 30. [19:02] kadams54: i will happily help you have less merge conflicts. [19:17] jujugui looking for a review/QA plz and thx https://github.com/juju/juju-gui/pull/334 [19:19] jcsackett I always thought functional tests were unit tests and integration tests were the ones which actually spanned multiple methods [19:19] this is just in reference to your branch, it could just be my terminology fail :) [19:22] hatch: really? i've always been told: unit = small unit of functionality (method, function), functional = larger test of a components functionality, integration = test of several components working together. [19:22] I suppose that makes more sense :) [19:22] carryon [19:22] hah [19:22] the lines blur a bit, to be sure. [19:23] jcsackett I'm looking at kadams branch and I'm wondering about all of these removed tests [19:23] they seem like they should still be there...no? [19:24] specifically the 'can create machine tokens' ones [19:24] maybe I'm missing that it's covered in the new ones he added [19:25] hatch: there's no longer an individual renderToken method for machine tokens, b/c the tokens generally always exist. [19:25] it's reasonably tested, still, i think. [19:25] ahh ok cool [19:25] though i am weeping at the deletion of all the stuff i wrote only days ago. [19:25] :p [19:25] rick_h_ any preference on my next card? [19:25] jcsackett haha yeah....that happens :) [19:27] hatch: I htink that il card needs to get done [19:27] hatch: if you're up for taking it, I have a call in 4min [19:27] hatch: but I'd like to go over it before you get started onit [19:27] *caugh caugh* I'm sick, bb after that card is done [19:27] :P [19:27] OH FINE! [19:27] haha, eta on call? [19:27] I can probably start on the bug card first [19:27] hatch: interview, 3:30 to 4:30 [19:28] ok I'll track down the bug first [19:28] hatch: sounds good to me [19:28] get something to work on for today and let's start the big whammy tomorrow? [19:28] sure [19:39] hatch: can you weigh in on https://github.com/juju/juju-gui/pull/325#discussion_r12862040 ? [19:39] I'm guessing the answer is "yes" but just want to make sure. [19:40] looking [19:41] commented - yes :) [19:44] rick_h_, jcsackett: could one of you sanity check this small change: https://codereview.appspot.com/95480044/ [19:44] jujugui was there a change in charmworld recently which would have passed 'juju-gui-90' in the charm url vs 'juju-gui' ? [19:45] I've found the bug/fix in the GUI, I just want to know if this is a bug in charmworld or just a change that we didn't protect against in the gui [19:45] hatch: not that i'm aware [19:45] hatch: I've never seen those letters before... [19:45] hmm odd...ok I'll fix it in the GUI to be more defensive, wonder how this just started happening heh [19:45] redir :) [19:46] the code that's failing hasn't been touched in 6 months so that's why I'm curious about the chamworld stuff....no biggy though [19:48] bac I'd review that for you but don't know if I can or how in googleapps [19:49] redir: that's ok. it's more archeology than code review [19:50] hatch: 508 is running on prod. there was some restart biz this morning. were there more details about that bac? [19:51] from webops [19:51] redir this was in relation to the data being returned from charmworld api calls [19:51] so a restart woudln't make any difference [19:52] redir: not yet. i got log files but saw nothing conclusive [19:56] hatch: k. the ngrams stuff I did is in 509 and 512 which isn't in production yet. [19:57] hatch: it must be something as it didn't do it without the flags [19:57] hatch: so seems like it must be something we changed recently in the GUI [19:58] bac: did you get your review you need? [19:58] hatch: my thing got done early, got time to chat? [19:58] rick_h_: no. it is just about the beaker stuff. [19:58] yeah sure - one min, going to get the dogs in [19:58] bac: LGTM [19:58] hatch: k, shoot me a link please [19:59] bac looking now. [19:59] https://plus.google.com/hangouts/_/g3qq5xtbryxutddyon5gstiqsea?hl=en [19:59] ^ rick_h_ [19:59] oh, nm, i see rick_h_ looked. [20:03] bac: do you still need to chat re:charmworld? [20:04] jcsackett: no, thanks. [20:04] jcsackett: i may need to bug you later, though, if i can't figure out why the server is restarting every three minutes [20:06] bac: i will try to be more available, should that happen. can you pass me the link for staging tools? i can set that back up on my own (did it once before). [20:09] jcsackett: bzr+ssh://bazaar.launchpad.net/~ce-orange-squad/charmworld/staging-tools [20:09] bac: thanks. [20:09] jcsackett: note that juju now requires the sharing of the jenv file used to bootstrap a system in order to collaborate on it [20:10] jcsackett: so those jenv files are a part of this branch. [20:10] jcsackett: whenever someone uses these tools to bootstrap a new system, and the jenv changes, it needs to be checked in and pushed back to LP [20:10] ok. [20:12] jcsackett: i think a lot of this could be replaced with a bundle. it is on my todo list. [20:34] * rick_h_ runs away to watch the boy do tumbling bits and such, c-ya'll later [20:34] anyone able to do review/qa? [20:34] jujugui [20:34] cya rick_h_ [20:37] later rick_h_ [20:50] jujugui looking for another review/qa https://github.com/juju/juju-gui/pull/335 this one is an easy one [20:51] jcsackett: do you know where the command jnova is packaged? [20:54] bac: jnova is script in staging-tools [20:55] ohhh [20:55] jcsackett: i did it through the dashboard. [20:55] jcsackett: i changed the doc to have it as ./jnova as a hint [20:56] bac: good plan. [21:22] got charmworld lander running again. man, that process needs some devops love. [21:22] * bac walks dog. bbiab to check [21:37] man I wish I could come up with a better name than Editorial [21:37] ^ jcsackett :P [21:40] hatch: agreed. [21:40] I'm thinking Curated [21:41] so we have a Search list and a Curated list [21:42] eh, i dunno--it's not really curated given it shows any crap that's new. [21:43] well it has promoted, popular, new [21:43] those are curated lists....even if they are curated by a machine :) [22:22] hatch: fair. [22:23] curated >> editorial [22:24] select? [22:24] prime [22:24] I'm rewriting the charmbrowser code so it's been changed to curated [22:24] :) [22:24] mohohahaha [22:24] if only there were a big book that listed like words [22:26] I tried that hah, curated and editorial are no where near eachother in the graph :) [22:39] huh San Juan is having its first barcamp. is that still a thing? [22:42] OOooo this is a new error: Uncaught TypeError: Cannot read property '_yuiaop' of undefined [22:42] bac heh, we have two of those a year [22:43] I always speak at them, great events [22:43] hatch: yeah, i went to the original ones in raleigh and i've been to one in saigon. but not for years. [22:44] ugh I hate widget child/parent stuff so much magic, impossible to debug [22:44] the first one they made me combine my session with the guy from redhat talking about git. :( [22:45] s/session/bzr session/ [22:45] hatch: but it was at RedHat HQ and i distributed Ubuntu CDs. that was fun. [22:46] lol nice [22:52] hatch: it's why I just went with charmresults [22:52]