[00:47] hatch: fyi https://codereview.appspot.com/13242046 [00:47] rick_h: u rock [00:48] odly enough I didn't get an email about this [00:48] you probably did but at first it had no title since it was WIP [00:48] the update just came through hopefully [00:50] ok, I'm toast for the night. See you tomorrow. [00:51] thanks! have a good night === schwuk_away is now known as schwuk [08:41] frankban: ping [08:50] TheMue: pong [08:51] frankban: morning [08:51] hi TheMue [08:51] frankban: i'm currently working on https://bugs.launchpad.net/juju-core/+bug/1194945 [08:51] <_mup_> Bug #1194945: juju set is overloaded [08:52] frankban: so on command line service options can be unset with "juju unset mysvc foo" [08:52] frankban: and an empty string is allowed for sting options "juju set mysvc foo=" [08:53] frankban: this leads to internal changes [08:53] frankban: and in https://codereview.appspot.com/12867044/diff/5001/state/statecmd/config_test.go fwereade_ reminds me that it may influence the GUI [08:53] TheMue: taking a look [08:55] TheMue: so, with unset you restore a default value (as it is found in config.yaml). with set option="" you explicitly set an empty string, right? [08:55] frankban: with that change unsetting an option with passing an empty string won't work anymore [08:55] frankban: exactly [08:56] frankban: and if the option is int, float or bool this would return an error [08:57] TheMue: how does this influence the behavior of the ServiceSet and ServiceUpdate API calls? [08:58] frankban: have to take a look [09:04] frankban: as far as i can see right now the setting mechanism works as before. only the implicit unsetting by setting the value of a setting (wow, how many settings ;) ) to "" won't work [09:04] frankban: so the API may need a ServiceUnset too [09:05] frankban: does the GUI uses the setting to "" for unsetting to default? or does it "only" provides settings to new values? [09:06] TheMue: maybe I am wrong, but doens't it already work passing nil values to ServiceSet? [09:07] frankban: sadly not. it's a map[string]string, so "settings["foo"] = nil" is not possible, the value has to be a string. [09:17] TheMue: OIC, and I double checked we effectively pass values as strings in the API call. From a quick look at the code, I believe we already have a problem with the config (I suspect we pass "null" or something like that when the stripped value is empty, I need to test this). anyway, I guess we will need a UX for differentiating between the "set to empty" and "restore the default" behaviors. and yes, we might want [09:17] to implement a ServiceUnset API eventually, (or, just a crazy idea, perhaps change the ServiceSet to be map[string]*string?). [09:18] frankban: here I would prefer the ServiceUnset, it would be more explicit [09:19] TheMue: I'll discuss this with gary_poster when he becomes available, thank you for the heads up [09:20] frankban: how is setting to default handled from the users perspective? a button or flag? or the implicit knowledge that setting to an empty value unsets it? [09:20] frankban: you have to thank william ;) [09:23] TheMue: AFAICT we just have input and check boxes. So I guess we don't have a story to restore boolean values to default, and for inputs/textareas we implicitly do that by setting an empty string (but I need to check). That's why I mentioned we need a UX story for the "restore defaults" semantic. [09:24] frankban: would be cool, indeed [09:25] TheMue: ah! what happens when you unset the value? is it just deleted from the db or is the db field set to the charm default value? [09:26] TheMue: I am asking because I am curious about what we can expect to receive back from the megawatcher when you unset a config field [09:26] frankban: how simple/complex is it to install the GUI on a local provider environment? i give a talk about juju end of October and would like to present the GUI there too. [09:27] frankban: unsetting deletes the setting [09:27] TheMue: ok [09:27] frankban: will add this to the release notes today [09:28] TheMue: the deploy story for the GUI is a juju charm, so it should be straightforward to deploy the GUI in a juju-core lxc env. However, never tried the local provider. If for some reason it does not work, then that's something we have to fix. [09:29] frankban: ok, will check it [11:21] jujugui anyone around for a second JS review of https://codereview.appspot.com/13242046/ to unblock the IE release from hatch [11:23] frankban: that branch up for review is the card in coding in bundles? [11:24] rick_h: I can take a quick look before lunch if QA is not required [11:24] rick_h: yes, I'll move it [11:24] frankban: hatch qa'd it already so I think it'd be safe to review sans-qa [11:25] frankban: cool, will review the charm one and qa it when I get back from dropping the boy off at day care. I should be able to qa it with -core in lxc right? [11:25] cool rick_h, never tried the local provider (I am usually on ec2), but it's worth trying [11:26] frankban: cool yea. <3'ing lxc provider for speed/testing [11:27] frankban: gary wants me to get up to speed on the charm so will be fun to poke at. [11:27] rick_h: cool [11:32] rick_h: feel free to ping me if you need any help with the charm/guiserver code [11:32] frankban: will do, thanks. [11:54] rick_h: review done [11:54] * frankban lunches [12:37] hey bac. thank you for adding branch to bundle info. were you planning on adding that to API as well? I think it would be nice to have there. [12:38] good point gary_poster [12:46] gary_poster: do we have a story for restoring default config values from the GUI? TheMue reported that the juju-core internals are changing: an empty string no longer means "restore default" [12:47] Thanks bac. Another question for you. In https://code.launchpad.net/~abentley/charms/bundles/wiki-bundle/bundle, we have three instances of the "bundle" string. Am I right that we can at least get rid of the "-bundle" or is that important for preventing some kind of name conflict? [12:47] gary_poster: that portion "wiki-bundle" is user-chosen [12:48] s/wiki-bundle/gary/ [12:48] man, that N4 at $199 is screaming "take me to london" [12:49] frankban, interesting. Potentially connected to https://bugs.launchpad.net/juju-gui/+bug/1214087 (see comment 3) in terms of further complicating the picture. Investigating. [12:49] <_mup_> Bug #1214087: GUI fails to deploy keystone [12:49] bac, right, but "wiki-bundle" is a project name, right? [12:49] bac, so mediawiki would be a project [12:50] which could have bundles and charms? [12:50] that seems fine to me, but I wanted to make sure that you all had not identified some concern for overlap [12:51] gary_poster: that name is the "basket" name, the name of the deployer file. it can contain multiple bundles [12:51] bac, yes, but doesn't that correspond to a project? I mean...let me get a concrete example [12:53] bac, so given https://code.launchpad.net/~charmers/charms/precise/mediawiki/trunk [12:54] huh...it is not listed in https://launchpad.net/charms/precise/+source/mediawiki [12:54] or https://launchpad.net/charms/+source/mediawiki [12:54] but still it could be [12:54] "mediawiki" is a package [12:54] so [12:55] https://code.launchpad.net/~charmers/charms/bundles/mediawiki/bundle [12:55] is a bundle for the same "mediawiki" package [12:55] right? [12:56] gary_poster: maybe, even if that seems more related to a check in the Keystone charm. I don't think the new behavior is already merged. But with that landed, unsetting a config passing an empty string will no longer work, and if you do that with int/float or bool config options, an error will be returned [12:56] benji: rocks! http://askubuntu.com/questions/108689/how-can-i-get-tab-completion-with-juju-in-zsh [12:56] :) [12:57] benji: I was thinking of writing one that parsed the results of `juju help commands` [12:57] benji: is there a good reason to not do that before I get started? [12:58] ooh, I'll have to MP that with switch added in [12:58] frankban we have no GUI approach for resetting values to my knowledge. That only happens in the ghost config controls. I think we are ok for now, though we may well want an explicit way of controlling this in the future [12:58] rick_h: that's what that script does [12:58] see http://bazaar.launchpad.net/~benji/+junk/zsh-juju-completion/view/head:/make-completion.py [12:58] gary_poster: yes, if that is how the devop chooses to name the basket. i hope this doesn't sound obtuse. if so, maybe we should talk. [12:58] it was written against PyJuju though, so it could probably use some updating [12:58] benji: ah, I was thinking of the _arguments doing that in shell [12:58] benji: gotcha, cool. Thanks. Most helpful. [12:59] gary_poster: sounds good [12:59] cool thanks frankban [12:59] bac, but the two identical package names (for charm and bundle) should be considered to be related, yeah? [13:00] bac or do you consider the underlying LP concepts to be irrelevant? [13:00] if we are going to be doing much with the CLI, I'd like to generate shell completion in a way inspired by how bzr does it; i.e., there is a subcommand that returns an easily parsable description of all the commands, options, etc. [13:00] benji: ah, that's cool [13:00] benji: yea, need some zsh subcommand completion awesomeness in there [13:01] benji: completing service names to destroy and such would be awesome [13:01] gary_poster: perhaps we can infer relevancy for the official/promulgated ones, if the ~charmers enforce a naming structure. otherwise, it is user-provided naming and therefore i don't think relevant. [13:01] yep; I'm not sure how best to do it though, caching would seem manditory [13:01] benji: yea :/ though for a first pass can wait [13:02] rick_h, benji: did you see allenap's shell script for autocompleting project names within a go environment? it's pretty nice. [13:03] gary_poster, bac, I see i walked into the middle of a conversation. charm/bundle/package names are arbitrary. Convention says they are related. They have changed and that caused confusion in the past [13:03] bac: nope, I havn't. Sounds cool. [13:03] bac: I use a forked/butchered workon script from doug hellman https://github.com/mitechie/workit to project hop [13:04] bac: sounds similar? [13:04] bac ok fair enough, thanks. I'm writing a summary email of some of the stuff we discussed earlier this week, so I'm trying to get the details right. In that vein, I'm almost done with the email, but I have one other related question. In https://code.launchpad.net/~abentley/charms/bundles/wiki-bundle/bundle, the last "bundle" could be migrated to "trunk" trivially, once we are ready to go live with a new version o [13:04] f charmworld, right? [13:04] sinzui, ack, thank you. [13:04] Novice users often try to version a package in the name and the same happened with charms. Charms don't accept charms with versions unless there are co-installable version. [13:04] that is a rare case [13:04] gary_poster: We're changing the name of bundle branches from "bundle" to "trunk"? [13:05] abentley, I'm asking whether that would be trivial [13:05] gary_poster: i think not. (sinzui may correct.) but if it were 'trunk' it would look just like a charm. using 'bundle' vs 'trunk' as the branch name is all that distinguishes them. [13:05] not the "bundles" series? [13:06] gary_poster: I think it would be trivial, for us, but it would require changes to the charm store, because it would then treat it as a charm. [13:06] abentley, ah! got it. thanks. [13:06] btw, abentley, I much appreciate your participation, but I apologize for unintentionally pinging you with those urls [13:07] gary_poster: np. [13:10] gary_poster: re always using the go sandbox in the charm when sandbox==true, I suspect we would have backward compatibility problem, i.e. someone upgrading to the new charm without updating the GUI to the (not yet released) last version. I guess this would be a problem even if we just allow sanbox mode in a juju-core env. do we care about those cases? [13:10] bac, abentley gary_poster , trunk does conflict the store. The store will report an error, but wont break [13:11] sinzui, right. That adds detail to what abentley said but does not contradict it, right? [13:11] right [13:11] cool thank you [13:12] frankban: I'm getting errors about no shelltoolbox when running the make command? It looks like it's using a venv, does it not install it in the venv? [13:13] hmm, looks like it's not in pypi, so no python package available. [13:14] rick_h: it should, it is listed in the tests/requirements.pip file [13:14] gotcha, frankban , your core situation is that I (a user) have an old GUI release with a new charm release, whatever the case. Mm. Thinking. [13:14] frankban: hmm, did a make clean and trying again. Will see what's up. [13:14] rick_h: try "make clean" and male [13:14] make even [13:14] :-) [13:15] frankban: oh, I'm missing something before it, a headers lib. http://paste.mitechie.com/show/1010/ [13:15] (and this will be the case for anyone using the ~juju-gui charm until we make the upcoming release) [13:17] gary_poster: yes [13:17] that's the only part that really bothers me :-P and we should have a release today [13:17] rick_h: interesting, so something is missing that is required to build apt_pkg [13:17] frankban: yea, looking [13:18] hatch btw when you get in that's the third headline for this release: the sandbox mode now has a complete Juju-Core-based implementation, which is now the default [13:18] gary_poster: ok, I'll just make sure that the "go sandbox activated and used everywhere" lands after the new release [13:19] cool thanks frankban. you agree with that? I don't see a safe way to do anything else, really [13:19] frankban: looks like might be libapt-pgk-dev. Retrying now [13:19] frankban, gary_poster: the new behavior is already merged by accident. i'll now do a rollback. but we have then to find a good way for (a) setting string settings to empty strings and (b) unset settings [13:21] gary_poster: I agree with that, the only other solution would be activating features based on the gui version, but that's not simple to accomplish and probably not worth the effort [13:21] frankban, exactly [13:21] TheMue, for (a) I'm guessing the commandline is the issue? The API is straightforward AIUI: you send an empty string, and an empty string is stored. [13:21] In the new change I mean [13:23] gary_poster: so based on your email my plan was to start looking at adding api3, the feature flag, etc today? or should I grab another inspector bug before heading over there? [13:23] gary_poster: yes, sending a string leads to this string stored in case of a string setting [13:23] gary_poster: but there's no unsetting [13:23] TheMue, gary_poster: AFAICT (b) requires a new API call or a change in the current config params (map[string]string -> map[string]*string) [13:23] TheMue, ok. That seems good for us for now. unsetting will be a new feature. Agree with frankban. [13:23] gary_poster: and this has been done by sending an empty string before [13:24] ack [13:24] frankban: yep, so libapt-pkg-dev is a dep to run the make command. [13:24] rick_h, let me see how we are doing on inspector. IE put us behind, and getting the inspector done is what we should do first to get things out the door. [13:24] gary_poster: rgr [13:25] rick_h, please grab an inspector one. I'd rather have a chance to get one thing done and the other thing unstarted than be inprogress on two :-) [13:25] gary_poster: k [13:25] rick_h: yeah, it is already listed in the HACKING.md file indeed: sudo apt-get install build-essential bzr libapt-pkg-dev python-pip \ [13:25] python-virtualenv xvfb [13:25] rick_h, you good with that under circumstances? [13:25] frankban: ah, gotcha. Sorry, just went off the notes. I should have looked at it [13:26] gary_poster: sure thing [13:26] cool thanks [13:26] rick_h: sorry, didn't remember that [13:26] frankban: all good, I'm used ot charmworld having the make sysdeps so it's still in make [13:27] rick_h: so you run "sudo make sysdeps"? that seems a good idea [13:27] frankban: yea, it's still a manual listed step so we dont' have to wait for it to run apt on every make command though. So I'd still be 'duh, didn't see that' [13:28] frankban: but make shows the extra step [13:29] frankban, TheMue, from the GUI perspective, setting config values to their defaults is a non-issue. We can do it now because we can get the charm, discover the default value, and explicitly tell Juju to use that. We already have everything available to do that now, pretty trivially. The CLI would need its own solution though, and that's where that API might come in handy. [13:29] rick_h: yeah [13:30] gary_poster: agreed. from the GUI there is no difference in having the default value or no value stored in the juju-core db [13:30] right [13:31] filed #1217884 fwiw [13:31] <_mup_> Bug #1217884: Include a way to reset config values to their defaults [13:31] triaged low [13:31] if we get people asking for it we can reprioritize, or if someone wants to champion it on the team, cool [13:33] bac, looking at #1217884. Out of curiosity, how much work does it look like it would take to get Safari working? Do things mostly work, or...? [13:33] <_mup_> Bug #1217884: Include a way to reset config values to their defaults [13:33] bac sorry I meant to paste #1202408 [13:33] <_mup_> Bug #1202408: Left panel charm browser inter-cell scrolling in Safari [13:34] This morning is apparently Gary's time to bother Brad as many times as possible [13:35] gary_poster: i do not see the visual effects reported in that bug with jujucharms.com [13:36] frankban: so when I enable the build in server I get that chrome websocket issue that came up in the call/fixed on the rapi side. [13:36] bac, oh! so, fix released? [13:36] frankban: so this is our tornado server now? It needs the same fix? [13:36] gary_poster: so perhaps whatever styling issues were causing it have been resolved. [13:36] cool. bac, more generally, though, how far away are we from Safari working well? [13:36] gary_poster: or invalid. i don't think it was addressed directly just fixed as a by-product [13:36] ok [13:36] rick_h: yes, the browser now connects to tornado [13:37] rick_h: what version of chrome? [13:37] frankban: ok, so I'm going to file that as a seperate bug, or find the old bug and tie to the charm [13:37] frankban: 30 [13:37] frankban: 31 actually as of today [13:37] frankban: Error during WebSocket handshake: Sec-WebSocket-Protocol mismatch, the headers need to match from request to reply [13:37] frankban: I think rapi was dropping the header? I'll have to look at the diff hazmat did [13:38] rick_h, yeah, chrome was setting the value to undefined, i just had it return [13:38] gary_poster: thanks for the info, will discuss it with fwereade_ , currently rolling back my last CL [13:38] not unconditionally though [13:38] gary_poster: i just poked around at it briefly and saw no issues [13:38] frankban: ^ [13:38] gary_poster: so perhaps we're really close. [13:38] frankban: so doesn't effect this branch, but heads up [13:39] hazmat: so the server must include a Sec-WebSocket-Protocol=undefined header? [13:39] frankban: yes [13:39] frankban: so that they version headers match [13:40] frankban, the server must send back what the client sent, and optionally use it to inform codec selection if it matches something known. [13:40] frankban, fwiw upstream has change signficantly.. therve has done a lot refactoring on the twisted ws branch, though it appears to have stalled again [13:40] frankban, oh this is with tornado? [13:41] * hazmat pokes at tornado src [13:41] gary_poster: turning on the inspector i do see some icon placement issues [13:42] frankban, see select_subprotocol in tornado websocket.py [13:42] probably needs to be overridden [13:42] abentley, benji, do either of you have time to review https://code.launchpad.net/~sinzui/charmworld/search-featured/+merge/182509 [13:42] this may just be a chrome beta/alpha bug as well [13:42] considering it doesn't really care about the reply [13:42] sinzui: sure. [13:42] hazmat: yeah [13:43] gary_poster: and big problems with ff on os x [13:45] hazmat: IIUC a "return subprotocols[0] if subprotocols else None" in select_subprotocol should do the trick [13:48] rick_h: thanks for QAing, and great to know that the GUI charm works well on a juju-core lxc provider [13:48] TheMue: ^^^^ [13:48] frankban: yep, <3 the faster lxc local stuff. [13:48] TheMue: ^^^^ ;-) [13:48] frankban: though I need to look into juju ssh, it uses the current username for the ssh so I had to ssh manually ubuntu@xxxx [13:49] frankban: ah, great news [13:49] frankban: will be a nice show to demonstrate it [13:50] rick_h: I'll start using it in place of ec2, do you have a ready-made local env conf? [13:50] frankban: heh yea, local: type: local admin-secret: 2c6e61e81c9ad07af67f833129bef2d0 [13:50] cool, thank you [13:51] frankban: nice short, and using juju switch to avoid the -e go and such [13:51] bac, ok thank you for details. [13:52] rick_h: I look forward to migrating the CI (and I guess jujucharms.com) and forget about multiple implementations ;-) [13:54] sinzui: At 310-311, do you really care about the order? If not, you can use assertItemsEqual instead. [13:54] guihelp, anyone available for a second review (no QA) of https://codereview.appspot.com/13340043 ? thanks! [13:55] abentley, indeed I don't care. [13:56] sinzui: r=me [13:57] benji, bac, sinzui: Can we have a chat about migrations? [13:57] sure [13:57] * benji finds his camera. [13:57] abentley, +1 [13:58] hey benji, could you verify that https://bugs.launchpad.net/juju-gui/+bug/1202413 is still an issue? I just tried it on saucy's FF 23 and it is OK [13:58] <_mup_> Bug #1202413: Relation drag line persists after clicking [13:58] benji I tried it on jujucharms and comingsoon [13:59] sinzui: guichat [13:59] abentley: could you have a look at the changes in this branch lp:~adeuring/charmworld/spurious-failures ? I know that some stuff can be refactored -- my main point is so far to just reduce test failures... Comments should show why I made each change [13:59] adeuring: On a call, but I can look after. [14:02] * hatch crosses finger final qa doesn't break ;) [14:03] * gary_poster crosses toes [14:03] easier to type but harder to walk [14:03] haha [14:07] hatch: stop poking the bear! :P [14:07] hatch: actually, it's good stuff. We've found some legit bugs along the way so appreciate the firm stabbing at things [14:08] dran straight! [14:08] darn, even [14:08] if we're down to the bugs darn near impossible to figure out, then must mean we're getting down to the last ones [14:08] * rick_h can hope at least! [14:17] Last bugs. I like it. [14:21] haha === schwuk is now known as schwuk_away [14:29] *sigh* [14:30] hatch: guichat? [14:31] sure [14:31] busy [14:38] gary_poster: I think we need to roll the core-backend debug change back [14:39] rick_h: is just testing core right now [14:39] but the sandbox is definitely broken when creating relations [14:39] hatch ok. you mean the default, right? [14:39] yep [14:39] hatch, ack. +1 on doing that and getting this release out the door [14:44] ugh lots of test failures when I switch back === schwuk_away is now known as schwuk [14:44] trying now with only the config changed but the default still at go [14:45] hatch, look for that commit and try reverting the commit [14:47] gary_poster: if I set the default to 'go' and then set the configs to 'python' the tests all pass [14:48] hatch, heh ok [14:48] hatch, and it still works? :-P [14:48] it appears so :) [14:49] so looks like we won't be setting go as the default for now [14:50] now when someone calls me on hangouts, my computer, tablet, cell phone all ring....it's a little much haha [14:50] hah [14:51] This is a public apology to bac for incorrectly correcting his use of "monotonically increasing"; he was right, I was wrong. Let this be a less on to the rest of you... or something [14:52] * rick_h bows before the great and powerful bac [14:52] hatch, ok. that slows down other tasks--frankban has a branch that was going to switch that so we could move forward with bundles. but...frankban, eh, let's talk. I think maybe we can still move forward. It will just be a bit riskier. :-/ [14:52] he has a charm branch that was going to switch the sandbox there to use go, I mean [14:53] gary_poster: yea, there's a real bug/issue with the ondelta model event that's either not calling or getting the right thing. [14:53] gary_poster: I could take a 30min timebox to try and fix [14:53] luca__, do we have an agenda for webteam call? we will see them in a week, and we potentially have some important things to talk about then, but if there's no topics immediately let's skip it (again) [14:54] gary_poster: I thought I cancelled it yesterday =O [14:54] luca__, oh cool. Sorry, just my phone not catching up to the times then :-P sorry, thanks [14:54] gary_poster: no worries, how is everything? [14:55] hatch, yes, if you are up for it. Maybe pair with Makyo, or me? [14:55] ok gimme a few minutes to track it down [14:55] thanks hatch [14:56] gary_poster, hatch: so there is a problem in building relations in the go sandbox? [14:56] luca__, hectic. post-vacation catch up combined with more re-prioritization discussions and discussions about Dustin's feedback :-) [14:57] luca__, looking forward to seeing juju.ubuntu.com 1.0 or whatever it is called Monday :-) [14:57] frankban: yes models.js:975 calls the handler with the wrong information [14:57] currently looking into the fix [14:57] gary_poster: haha, hopefully it looks good :) I'll give you a sneak peak when it's in a better shape. [14:57] hatch: thank you [14:57] frankban: and it's all your fault :P [14:57] cool luca__ :-) [14:57] gary_poster: would you like to catchup on anything tomorrow? [14:58] hatch: that's surprising! [14:58] lol [15:03] so it appears that the delta that's coming back isn't including the endpoint data [15:03] hatch: isn't that code used also for real environments? if so, why relations work in real envs? [15:04] frankban: rick_h tried it in a real env and it didn't error [15:04] so possibly sandbox issue? [15:04] hatch: but the service also didn't come up since it was in error [15:04] hatch: more probably, yes [15:05] frankban: here is the delta data https://gist.github.com/hatched/6367087 [15:05] which I'm certain is incorrect [15:07] hatch: perhaps you can take examples of how the delta should be from the test_model_handlers.js file, starting from line 351 [15:08] frankban: thanks I'll check it out [15:10] jujugui units binding of service-overview viewlet does not work in FF against a real core env. Any ideas? [15:10] (in trunk) [15:11] Makyo, works in chrome? [15:11] gary_poster, I haven't checked yet, currently in the process of deploying the charm. [15:12] ack, Makyo. Worth a check; I'd assume it is browser agnostic, and a diff between our sandbox and the real world, but would be good to verify [15:12] gary_poster, yep. Just can't test from local devel server in chrome. [15:12] oh :-/ [15:12] k [15:13] Makyo, old pre-inspector interface works [15:13] ? [15:13] bac: I'd like to re-enable test_ensures_category_questions. Can you describe the spurious test failures that caused you to disable it in https://code.launchpad.net/~bac/charmworld/spurious-test-failures/+merge/181418 ? [15:13] Checking. [15:14] gary_poster, yes. [15:14] k [15:14] frankban: somewhere between sandbox.js:handleClientAddRelation and sandbox.js:receiveNow it loses the Endpoints object :/ [15:15] abentley: the test failed. spuriously. these failures are what prompted abel's investigation, that is on-going. he did seem to correlate some failures with slower machines. [15:15] bac: Were there any particular error messages you got? [15:15] abentley: i do not recall the exact error conditions for each failure [15:17] abentley: i'd suggest talking with adeuring [15:17] Makyo, hatch is deep in release. I have no further ideas without digging. Would you like a pair with me or rick_h, or an initial timebox investigation? The obvious thing to me is to suspect the deltas; dunno [15:18] gary_poster, I know where it's going wrong, if it is, will update once I can test from the charm [15:18] Makyo, cool ty [15:18] bac: Please report spurious failures more thoroughly. Even a paste from the console would be very useful to me now. [15:19] bac, abentley: as far as I understand the problem currently, it seems that ES is sometimes just a bit slow, especially when an index is created, deleted or when its mapping is changed. I have a branch that catches many errors. [15:19] adeuring: I'm looking at that branch you asked me to earlier. I assume it's the same? [15:20] abentley: yes, that's the one I meant [15:21] abentley: there were a set of about 6 tests that were failing in various combinations or not at all. they were reproducible by sinzui and adeuring in trunk. the failures were blocking the landing of other work. the work to fix them was handed off to adeuring. [15:21] well, I got meanwhile errors/failures in many more tests.... [15:22] when base.js fires a 'msg' event, can someone tell me what listens for that event? [15:22] grep is failing me.... [15:22] abentley: in the future attaching a failure report to a bug referenced in an XXX when disabling the tests is a fine idea [15:22] bac: Thanks. [15:23] some problems still remain, but they look to me like real ES errors, like this one: sending faile [15:23] d shard for [temp_index][3], node[gWVu1uWMSvegY4D7uW2c7g], [R], s[INITIALIZING], reason [F [15:23] ailed to start shard, message [RecoveryFailedException[[temp_index][3]: Recovery failed fr [15:23] om [Ezekiel][6hy-kGs5Qwe5OSuRjxJrXw][inet[/192.168.2.30:9300]] into [Gregory Gideon][gWVu1 [15:23] uWMSvegY4D7uW2c7g][inet[/192.168.2.209:9300]]]; nested: RemoteTransportException[[Ezekiel] [15:23] [inet[/192.168.2.30:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineExc [15:23] eption[[temp_index][3] Phase[2] Execution failed]; nested: RemoteTransportException[[Grego [15:23] ry Gideon][inet[/192.168.2.209:9300]][index/shard/recovery/prepareTranslog]]; nested: Engi [15:23] neCreationFailureException[[temp_index][3] failed to create engine]; nested: LockObtainFai [15:23] ledException[Lock obtain timed out: NativeFSLock@/var/lib/elasticsearch/elasticsearch/node [15:23] s/0/indices/temp_index/3/index/write.lock]; ]] [15:24] ignore me, I missed the defaultfn [15:25] restarting the ES server seems to help [15:27] adeuring: Have you tried ElasticSearchClient.wait_for_startup rather than calling health directly? [15:27] abentley: yes, the "yellow" status is quite often not enough. [15:27] adeuring: Ah. [15:27] * gary_poster laughs at still having "yellow" ping him [15:28] abentley: and even the "green" status sometimes not suffient. but a refresh() call seems to help in this case [15:29] Ugh, I've made this change before, I know I have >:/ [15:30] adeuring: This makes me worry that the calls may not be fixing the problem, merely delaying execution, and coincidentally fixing the problem. [15:30] Most of the time. [15:30] Makyo: maybe made it in the python side and never got to the go side? Now with go default it's hitting again? [15:32] abentley: that's possible. OTOH, it seems that ES responds for several requests before "the work is really finished", so waiting for a certain status may help indeed. [15:32] rick_h, it's go only; we're just not passing an argument (and if that argument's undefined, it exits). I could've sworn I fixed this in the branch where I first added the unit list to the inspector. [15:32] Makyo: ok, yea just tossing idea out there on 'what's changed recently' to trigger it [15:32] abentley: and I don't see any better way to query the status of the ES server... [15:32] adeuring: Yes, the health is reasonable, it's the refresh that I worry about. [15:33] rick_h, Yeah, now I'm confused too, because the code's there in trunk in the source, but not in the compressed prod file (at least, according to source maps) [15:33] Makyo: orly, now that's interesting [15:34] Makyo: cache? or build step issue for new js files ? [15:34] abentley: well, the documenntation of refresh() says "making all operations performed since the last refresh available for search", so calling it should have some effects [15:34] adeuring: Right, but if the only operation was creating the index, I would expect that health was the only thing I had to wait for. [15:35] rick_h, yeah, I'm not sure! Poking around further. [15:36] abentley: maybe I need to check where exactly I added refresh() calls, but right now I see just one after a run_migration() call. And that call could do more that just create/delete indexes. I would beed to check more closely though to be sure... [15:38] adeuring: we'll be deleting those migrations anyway, so spurious test failures there don't matter. The only migration code we're keeping is the bit where we ensure category questions. [15:40] rick_h, gary_poster - alright, I found it. It's disappointing. rick_h no clue why the sourcemaps are showing out-of-date code, maybe they weren't generated properly. The issue at the heart of it, though, is that we're now getting units before services. When we get unit-adds, we try to update the list of units stored with the service, which doesn't exist yet. That list is what's bound in the inspector. [15:41] Makyo, ah. :-/ [15:41] abentley: right, but I saw also errors in test_search.py, for example. And they were not related to migrations either... [15:41] adeuring: Right, I [15:42] so I think at least the changes I made to testing.__init__ are useful [15:42] * gary_poster wants time for more reliable, non-canonistack testing against juju core. [15:42] adeuring: Right, I'm talking about the tests of migrations, since that's what you mentioned. I know that there are other tests that you updated. [15:42] timebox is up and I have no idea what's going on [15:43] Makyo, I think we talked about this possibility way back when and had an idea of how to approach it. [15:43] somewhere between receiving the delta and the onDelta call the endpoints get lost - but even the d3 stuff gets the proper data [15:44] gary_poster, Yeah, sort by delta type to maintain the hierarchy. Investigating that now. [15:44] Makyo, cool thanks [15:44]