[00:01] thumper: for context we've been building up a backlog of code in the juju-actions repo - your feature flags made it possible for us to begin landing that code without it being available by default while it's incomplete [00:04] * thumper nods [00:08] ericsnow: is it tomorrow for you already? [00:08] I sent the two first patches for restore :p [00:08] perrito666: yeah, I saw [00:08] perrito666: sweetness [00:09] * perrito666 wonders how many more times is the power going to go off [00:09] * perrito666 also wonders why he forgot to get an UPS for the internet [00:13] thumper: was my answer to block.ProcessBlockedError doesn't use errors.Cause(err) digestable? [00:14] thumper: "was my answer to your question - why block.ProcessBlockedError doesn't use errors.Cause(err) - digestable?" even [00:14] anastasiamac_: hey [00:14] just busy right now [00:14] with you shortly [00:14] thumper: np [00:30] anastasiamac_: well yesterday with my machine branch, I changed it to use errors.Cause [00:31] thumper: did ur machine branch land? [00:31] anastasiamac_: yep [00:32] anastasiamac_: all tests pass [00:32] anastasiamac_: don't stress, I know what I'm doing [00:44] menn0: I'll fix that, looks like an unrelated ci-tools change brought in a dep it probably shouldn't [00:47] thumper: m not stressed about tests but the actual output on command line :D [00:47] anastasiamac_: doesn't change [00:49] jw4: review done... [00:49] thumper: much appreciated [00:50] thumper: just getting ready to update the TestHelpCommands for with and without feature [00:52] * thumper nods [00:52] thumper: good feedback - I'll work on some of these, but bodie_ knows more about most of this. [00:52] jw4: I recomment quite a few changes [00:53] thumper: thnx ') [00:54] thumper: s/recomment/recommend/? : if so, yeah - good stuff [00:55] was going for recommended [00:55] hehe [00:55] * thumper shrugs [00:55] * thumper is very tired [00:59] perrito666: are you looking at the joyent bug? [01:03] menn0: requeued [01:15] menn0: it seems that the immutable values aren't tested in the state tests, but probably are in the config [01:15] * thumper double checks [01:16] mgz_: thank you! [01:17] menn0: yep tested there [01:17] thumper: ok. so is that good enough for them not be retested at the API level? [01:18] menn0: I think I'll leave one there to show the message [01:18] but won't do all [01:18] thumper: that sounds good to me [01:21] menn0: you seem to be invoking the boyscout rule of code [01:21] menn0: I'll clean the code even though it isn't my mess :-) [01:27] thumper: boyscout rule of code... [01:28] anastasiamac_: leave the code cleaner than you found it [01:28] anastasiamac_: like "leave the camp ground cleaner than you arrived" [01:29] thumper: oh.. and i thought it was "pray at the end of the session'... [01:29] haha [01:29] maybe that is apt too [01:29] pray before submitting [01:29] thumper: at least, that's what boyscout groups do here [01:29] thumper: :D [01:30] menn0: thnx for cleaning messes! [01:30] menn0: and campsites [01:42] anastasiamac_, thumper: :) [01:42] * thumper frowns [01:42] ah... [01:42] fark [01:43] found it [01:44] I just saw all the Ship Its for one of my branches. Thanks all for reviewing my fix that moves one line of code :) [01:48] thumper: should someone be looking at bug 1401130? [01:48] Bug #1401130: Joyent deployments cannot download agents from state-server

[01:48] that's the other CI blocker [01:49] menn0: I just had to get in on that action [01:49] menn0: I thought perrito666 was looking at it, but he enver marked himself as in progress [01:49] ericsnow: do you know? [01:49] thumper: nope [01:50] thumper: I have been having power issues the whole day and I tried to look at it without much success [01:50] thumper: nate said he was going to get someone on it earlier in the day [01:50] menn0: if you could look that would be very helpful [01:50] * perrito666 has not had a full hour with power today [01:50] do we have creds we can use? [01:50] thumper: ok === kadams54-away is now known as kadams54 [01:50] thumper: I think I know where to look [01:51] menn0: oh, ok [02:00] menn0: just one question on your review on environments command that made no sense to me [02:00] menn0: could I get you to clarify? [02:00] thumper: sure [02:00] currently running the entire test suite [02:01] thumper: I dont, I was running things through sinzui [02:02] perrito666: that's ok, menn0 is on it :) [02:02] \o/ [02:03] thumper: yeah, that comment makes no sense. if there was something to unset it wouldn't be unknown :) [02:03] thumper: drop that [02:03] ok [02:11] thumper: RB shows that block tests were lost again (on environment super-command branch) [02:11] thumper: is it just RB's perspective? [02:11] anastasiamac_: yeah [02:11] also... wat? [02:11] no definitely there [02:12] but moved [02:12] and changed [02:12] \o/. [02:12] set_test.go, unset_test.go [02:12] thumper: tyvm for making them cleaner and more readibale [02:12] at the bottom [02:13] thumper: a bit of boyscouting seem to run in nz [02:13] that's fine [02:13] all good [02:15] by faking out the api fully we get much more control over tests [02:15] and we stop testing everything at every layer [02:15] which is a waste [02:15] as long as we still have enough tests to show that everything is wired up correctly [02:17] thumper: makes sense! thnx again :-) [02:17] that's fine [02:23] axw, ping [02:24] thumper, afternoon [02:26] http://reviews.vapour.ws/r/611/ can any of you have a look at this MP for bug 1397376 please? [02:26] Bug #1397376: maas provider: 1.21b3 removes ip from api-endpoints

[02:27] o/ dimitern [02:29] dimitern: pong [02:29] ok [02:31] axw, ta! [03:08] jam, leads call [03:47] thumper: ping? [03:47] menn0: otp [03:48] thumper: i've reproed and characterised bug 1401130. I've added the details to the ticket. [03:48] Bug #1401130: Joyent deployments cannot download agents from state-server

[03:49] thumper: not sure where to look now. we might need to contact Joyent. [03:49] ok [04:10] thumper: I need to head out for a bit but will be back later [04:10] kk [04:12] jam, dimitern: passing on this critical bug to you two: https://bugs.launchpad.net/juju-core/+bug/1401130 [04:12] Bug #1401130: Joyent deployments cannot download agents from state-server

[04:13] thumper, ok [04:13] dimitern: menno added details, including some diagnostics [04:13] btw my chrome crashed, but I still hear audio - no video from the call [04:13] thumper: so all of the discussion I read on that seems to be pointing fingers at Joyent. [04:13] dimitern: very networky [04:14] dimitern: we can still see your video feed [04:14] I've seen that when you run into JS in another window locking up [04:14] wow - it did crash without kicking me out of the hangout and then fixed itself [04:26] fwereade: I keep meaning to schedule time to chat with you again, and now we're at the end of my week. Ping me when you're around [04:30] menn0, ping [04:31] thumper, menn0, jam, do we have any joyent testing account I could use? === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away [05:10] dimitern: hi i'm back [05:11] menn0, hey, it's fine jam got me sorted out; but can I still bug you if I have questions re bug 1401130? [05:11] Bug #1401130: Joyent deployments cannot download agents from state-server

[05:12] dimitern: sure can [05:12] dimitern: i've got to help get kids to bed but just ask here and i'll get back to you [05:12] dimitern: did you see the details i added to the ticket [05:12] menn0, cheers, no worries [05:13] menn0, yeah, it looks weird.. I'll dig into it some more [05:13] dimitern: my theories are that either something's wrong at joyent [05:13] dimitern: or Juju is using the wrong addresses or wrong routing [05:14] or wrong netmasks [05:15] dimitern: i'll check back in a bit [05:15] menn0, right [06:13] dimitern: hey hey, kids asleep. i've got a little bit of time. [06:13] dimitern: figure anything out? [06:13] menn0, nope, I was preparing a response to axw's review [06:13] menn0, but since you're here, I'll try it now [06:14] dimitern: kk [06:14] dimitern: i'm just creating a card on LK [06:14] axw, thanks btw - you've raised some very good points [06:14] dimitern: nps [06:15] dimitern: did you go to sleep last night? you were on very early :) [06:15] or are you sprinting [06:15] axw, I did - shortly after 8pm [06:15] to wake up in time for the 5am call :) [06:15] ah, fun [08:06] jam, heyhey [08:27] hey fwereade, how's it going? [08:27] just finishing up lunch, care to meet in about 30min? [08:27] jam, ehh, tolerable, a bit low-level fluey, but yes let's [08:43] I'd appreciate a review on http://reviews.vapour.ws/r/623/ - it unblocks ci for the critical bug 1401130 [08:43] Bug #1401130: Joyent deployments cannot download agents from state-server

[08:45] jam, axw, menn0, ^^ [08:48] dimitern: that seems like our Networker needs some lovin', but otherwise LGTM [08:48] dimitern: you did test that it both *failed* before this, and *worked* after, right? [08:50] jam, indeed it does :) [08:51] jam, I couldn't get it to fail as described, but I've seen the overwritten /etc/network/interfaces file, which wasn't including eth1.cfg for the internal network the api server is on [08:51] jam, after the fix it doesn't happen [08:53] jam, so even though the networker didn't stop eth1 it won't be brought up after reboot [08:58] fwereade: whenever you would like to chat [08:58] jam, joining sapphire standup [08:58] dimitern: right, so they have a custom /etc/network/interfaces which we are overwriting and not quite matching what was there [08:59] and dimitern, we aren't really going to be having a smart networker on Joyent soon anyway, so we're fine [08:59] jam, yeah, and by sheer chance eth0 wasn't brought down (the one with the public ip) as it was considered "primary" :) [09:00] jam, so I could ssh into it, and eth1 wasn't present in state (no network info during provisioning), so it wasn't stopped immediately [09:00] jam, definitely not a priority now [09:10] mgz_, ping [10:01] jam, omw to standup [10:01] dimitern: core-team meeting today [10:01] oops team meeting actually [10:01] jam, backport of that joyent bug fix for 1.21 if you're so kind :) http://reviews.vapour.ws/r/624/ [10:02] dimitern: did you have time to test it explicitly as well, or is it just a strict backport w/o testing ? [10:02] axw: team meeting? (if you're available) [10:03] mgz_: do you come to the juju-core-team meetings to see how we're doing ? [10:03] jam, testing now [10:04] dimitern: reviewed [10:05] wwitzel3: if you are around as well [10:05] jam, cheers [10:19] dimitern: i'm back [10:20] dimitern: glad that you got it figured out [10:20] menn0_, yeah, I got lucky I guess :) === menn0_ is now known as menn0 [10:41] fwereade: selfie got added to the Oxford dictionary, but I don't think Moar has reached that level yet :) [10:42] fwereade: I need moar coffee, but then I'm up for moar chatting if you would like [12:34] wwitzel3: ping me when you are back please [12:52] axw: tx for the reviews you are the first person ever to review that part of restore :p it was too far at the end of the large patch [14:32] ping abentley [14:32] mbruzek: pong [14:33] Good morning. I kicked off 5 jenkins builds yesterday to retest some of the charms hoping to update the report by charm: http://reports.vapour.ws/charm-tests-by-charm [14:34] abentley: #10658, #10659, #10660, #10663, #10664. [14:34] Bug #10658: evms: new changes from Debian require merging [14:34] Bug #10659: mozilla-firefox-locale-it: new changes from Debian require merging [14:34] Bug #10660: Modem lights not working with "networks" [14:34] Bug #10663: Boot menu help references Morphix [14:34] Bug #10664: Framebuffer bootsplatch logo jpeg-like [14:34] Those are not bugs sorry about that. [14:34] Those are jenkins build numbers. [14:35] abentley: None of the results were reflected on the report on vapour. Is there another step that has to happen to get the report to update? [14:36] mbruzek: No, it should update automatically within a minute. I'll try to see what's going on. [14:36] abentley: The consol output shows the s3cmd run at the end of the last 4 of those 5. [14:36] abentley: Thanks for looking into it [14:42] mbruzek: I see results for everything except 10658. [14:44] abentley: I see results for all of them but they are not updated with the Dec 10 results. http://reports.vapour.ws/charm-tests-by-charm [14:45] abentley: for example build 10659 is cs:precise/shelrtv, that entry has a date of August 26 2014, not December 10 [14:46] abentley: build 10660 is cs:precise/solr, the entry on reports has it at September 09 [14:47] abentley: I see other December 10 entries, just not for the ones I request on December 10. [14:47] wwitzel3: should we just get started after standup? [14:48] mbruzek: Everything you built is shown in the "All charm and bundle results" table. [14:48] wwitzel3: I'm available before too [14:48] mbruzek: I suspect this is a sorting bug. [14:49] abentley: OK I was looking at the charm-tests-by-charm what process builds that report? [14:51] abentley: I see a difference. the ones I requested were cs:precise/charm-name the ones in the charm-tests-by-charm have a revision number in the name. [14:51] mbruzek: That report is generated on the fly. The results are ingested from s3 by a shell script. [14:51] mbruzek: Well, a python script. [14:51] abentley: So the ones I requested are missing a revision number. [14:52] mbruzek: That's right. We test specific charm versions when we do automatic testing. [14:52] abentley: But how would I know the version numbers? I just wanted to test the latest charm so I sent "cs:precise/solr" to charmguardian [14:53] mbruzek: That's not something we support, but the charm store and manage.jujucharms.com know the charm versions. [14:54] mbruzek: Since you're retesting the same version, you can also just look at the previous test results. [14:57] dimitern: any objection to me moving decimalToIP and ipToDecimal into network and making them public? [14:57] dimitern: I need them in ec2 [14:57] voidspace, +1 [14:57] dimitern: or at least I *may* do, I'll only move them if I actually need them [14:57] dimitern: cool, thanks [14:57] abentley: How is it supposed to work? When the charms get a new version number are new tests kicked off? I thought I saw a few instances where the charms are of a newer revision. [14:58] And the tests did not get kicked off. [14:58] voidspace, sorry, I got deep into bug fixing again and haven't actually asked how's it going today? :) [14:58] Yes, when the charms get a new version number we see that we have no results for that number in the database, so we kick off a new test. [14:58] voidspace, any issues? [15:00] voidspace, I hope you are feeling better today [15:00] fwereade: axw suggested I ask you about http://reviews.vapour.ws/r/608/ [15:02] mbruzek: Please file a bug, but be aware that certain charms are blacklisted because they break charmguardian: cs:precise/cassandra cs:trusty/hdp-hadoop cs:trusty/swift-proxy [15:03] abentley: I am not trying to create more work, I just did not know an easy way to find the version number. [15:04] abentley: bug against charmguardian? [15:04] mbruzek: No, if you see instances where we failed to test a new version, file a bug against https://launchpad.net/juju-reports [15:09] dimitern: a bit better [15:09] dimitern: well enough to not be in bed anyway :-) [15:09] dimitern: yeah, going fine - I've just picked back up the Subnets work [15:09] dimitern: no issues - you were right about aws [15:09] dimitern: first four and last IP in a CIDR are reserved [15:10] voidspace, \o/ [15:11] voidspace, great, ping me if there's anything then :) [15:11] dimitern: I will do, thanks [15:28] ericsnow, I'm a bit unconvinced by putting az into *every* relation setting [15:28] ericsnow, what's the thinking behind that? [15:29] ericsnow, we definitely agreed to make AZ available to units [15:29] fwereade: to be honest I just followed the example of the IP address [15:29] ericsnow, but I don't see how that translates to making them part of everyrelation protocol [15:30] fwereade: as far as I understand it is part of the spec: https://docs.google.com/a/canonical.com/document/d/1yVlzgKqfhKccUbm3WcZBq7WnzglBygknscr_I_u4bUg/edit [15:30] mbruzek: So here's what's going on: cs:trusty/hdp-storm-3 breaks charmguardian, so we never get any results for it. Since we don't have any results for it, we test it, and we never get around to anything else. This has been going on with hdp-zookeeper, hdp-tez, hdp-storm, hdp-pig, hdp-hive, hdp-hive since Nov 22. [15:30] when deploying a local charm using local provider, do that charm get cached anywhere? [15:31] fwereade: it's entirely possible I took the wrong approach [15:31] or will juju already just read from the path I provide it [15:31] abentley: thus the black list that I read about? [15:31] fwereade: I've been nervous at how small that patch is :) [15:32] mbruzek: Yes. [15:32] ericsnow, hmm, so, I do see that in there, but (1) it only mentions peers and (2) I think it's a really bad idea [15:32] fwereade: See the "exposing to charms" section and the part about relation-get in the "mechanisms" section [15:32] ericsnow, so on one side we shouldn't be inserting it in non-peer relations [15:32] fwereade: I'm fine with dropping it but it's not my spec :) [15:33] fwereade: good point [15:33] ericsnow, and on the other, every charm which uses a peer relation now suddenly has two protocols, and might be using one or the other depending on what the juju version is [15:33] ericsnow, which is probably not *that* harmful because I'm pretty certain we don't distinguish between "" and `missing` in relation-get [15:35] abentley: What about the storage charm? Is that on your blacklist? I tried running that one several times and it never gets to the s3cmd to upload the results. [15:35] ericsnow, but still :) [15:35] hazmat, ping [15:35] abentley: build 10647, 10658 [15:35] mbruzek: No, it's not on the blacklist. [15:36] mbruzek: I have not figured out why that one doesn't upload to s3, but it isn't breaking charmguardian like the others did. [15:36] fwereade: so is "private-address" set for all relations? [15:37] abentley: I will file a bug [15:37] ericsnow, yes it is [15:37] fwereade: k [15:38] fwereade: so at the least "availability-zone" should only be set if the units in the relation are peers? [15:38] fwereade, pong [15:38] fwereade: or are you opposed to the adding zones to relation-get at all? [15:38] * hazmat reads backlog [15:39] fwereade, ericsnow, hazmat: my understanding is that we needed AZ for all relations [15:39] hazmat, I'm surprised about putting AZ into any relation by default [15:39] natefinch, so we should change the protocol for every relation, when we have enough trouble with protocols already? colour me unconvinced [15:39] fwereade, at minimal to achieve the use case it needs to be in peer relations. [15:40] fwereade: I was just following the spec... it says "The set of zone names in which peer **and per-relationship counterpart units are running**" [15:40] fwereade: I might be misunderstanding the spec [15:40] fwereade, its not a change in protocol its analogous to the work we already do with address [15:40] hazmat, auto-adding it doesn't seem worth the effort -- can't the charms that need it make it part of theirprotocol explicitly? [15:40] fwereade, they could.. however. its implicit to the iaas infrastructure provisioning responsibility juju is already taking [15:41] having juju provide that for relations to me automatically seems a logical extension just as we do with ip addrs [15:41] hazmat, fwiw it*is* a change in protocol, you may have a case that it's not an *important* change in the protocol [15:41] fwereade, hence the notion of prefixing the key in the rel [15:42] fwereade, the protocol communication itself is unchanged as its a seed value [15:42] hazmat, and we make those keys unwritableby normal means? [15:42] fwereade, no.. they should be writable [15:42] hazmat, so the juju- prefix is just a sort of hint? [15:42] fwereade, natefinch, ericsnow .. fwiw. i'm fine if we don't pre-seed the rels. the advantage requires explicit charm awareness and therefore they can seed [15:43] fwereade, proxy charms need to override such values and it may not have meaning to them [15:43] hazmat, indeed [15:43] hazmat, if we don't *need* them in every relation I'd much rather not put them in any relations by default [15:44] i think thats okay [15:44] hazmat, cool, thanks [15:46] hazmat, btw, about the suggested unit-get changes [15:47] hazmat, given that the two existing uses of unit-get are basically broken in any world with interesting networks [15:47] hazmat, is there any strong reason to add to unit-get vs putting them in env vars? [16:04] hazmat, also I fear that spec needs rather more detail re listing zones and/or configuring zone spread [16:05] hazmat, eg it makes no sense to make all provider zones available to a charm that's meant to spread over only 3 of them [16:05] hazmat, and I'm not 100% sure it makes sense to tell a service with 2 units about the 3 zones over which it might spread [16:06] hazmat, possibly specifying a zone spread should also imply a minimum unit count? [16:07] hazmat, (also, what happens if zone spread changes? is it immutable?) [16:07] fwereade: we had decided not to list available zones [16:07] fwereade: only list zones in which units exist [16:08] fwereade: or rather, only list zones for units. No general list of zones [16:08] natefinch, yeah, but *that's* tricky when you go from one unit to 3 [16:08] natefinch, in general, if we're changing info on which a charm depends, we should be firing some hook to let it know about it [16:09] natefinch, which hooks should we fire? [16:09] natefinch: standup? [16:09] fwereade: it's just exposing information about the units [16:09] fwereade: so, peer-relation-changed I guess? [16:09] natefinch, right, but then every single unit comes up seeing itself alone in a single zone [16:10] natefinch, so it's more about goal-state/active-state [16:10] hazmat, btw, I would really appreciate your thoughts on that [16:10] hazmat, there was a thread a couple of weeks back on juju-dev [16:10] hazmat, something like "feature request: expose relations in juju status" [16:11] hazmat, AFAICS we need to expose both those measures to be useful [16:11] natefinch, (did you see that thread btw?) [16:11] fwereade: btw I'm taking this over from eric, since he and wayne are heads down on GCE [16:12] natefinch, ah ok :) [16:12] fwereade: yeah, I remember the thread, though not too many specifics. [16:13] natefinch, tbh I was hoping it'd get some more thought and responses [16:14] natefinch, it may be that I need to write up a half-assed proposal before people will really start saying what won't work for them [16:14] fwereade: the only way to get the right answer is to propose the wrong answer :) [16:15] natefinch, yeah, indeed [16:15] natefinch, http://thecodelesscode.com/case/170 [16:16] fwereade, sorry stepped out, catching up [16:17] fwereade, unit-get still makes sense, and even more so with multiple networks, the issue atm that bites with unit-get is the address is both ip and dns addr which is inappropriate depending on the use case and needs disambiguation [16:18] fwereade, zone is immutable for an instance lifetime [16:20] fwereade, the zone spread behavior is independent of surfacing zone to a unit. i agree it would be nice to have some configurable policy on the zone spread but thats not really the issue at hand, and even a default behavior of max 3 is good enough any use case that comes to mind.. the exception is max zone 1, but we've decided to punt on that one till we have actual need. [16:21] fwereade, the information doesn't change, and the charm already hooks for observing and conveying this information via relations if it so choses [16:22] hazmat, I know that a single unit's zone is immutable, but the list of zones -- be it those in the provider, or those across which units are spread -- is not necessrily so === kadams54 is now known as kadams54-away [16:23] hazmat, if we're exposing the list, we need to notify on changes to the list [16:23] fwereade, the mutation awareness in the zone set is accompanied by new units joining the relation and there by conveying that information [16:23] fwereade, we're not exposing the list [16:23] hazmat, the spec has unit-get list-zones [16:23] hazmat, if we can drop that I'm totally fine there [16:24] fwereade, i think that was speculative, and even then it would only be those of the units in the service [16:24] hazmat, but then I'm not sure that there's much benefit to unit-get zone vs $JUJU_AVAILABILITY_ZONE [16:24] fwereade, the full zone list isn't really meaningiful if the service isn't present [16:24] in all of them [16:24] hazmat, agreed [16:24] hazmat, so then we still have the active/goal state issue, I think [16:25] hazmat, that everything comes up in an empty universe and needs to wait around until it has the full picture [16:25] fwereade, unit-get is an information channel.. we have other uses for it, including arrays of data (net lists, addrs. etc). i'd rather not baby and bath water everything into env vars [16:25]