[00:00] waigani, oh, I understand. Yes i think so. I can confirm though that the "upgrade-charm --force" hack is a permanent fix, so envs/users can resolve the problem in minutes if they need too [00:01] waigani, We cannot an an upgrade step to 1.22, since we cannot distribute that version any more. [00:02] waigani, We need 1.23 to do the fix an 1.24 to do the fix. [00:02] sinzui: right, got it. perrito666 ^ [00:03] waigani, one complicate is that our early adopters still believe odd numbered jujus are dangerous, so they only upgrade to even numbers [00:26] wallyworld: I bootstrapped on aws without issue: http://pastebin.ubuntu.com/10946635/. As I understand it: PrepareForCreateEnvironment is checking UnknownAttrs for control-bucket. configFields, which already had control-bucket, is used to determine what attrs are unknown, thus control-bucket will not be in the unknown attrs, it will be nil. The configDefault is only used to check Coercion of the value types of the attributes: [00:26] environs/config/config.go:1384 [00:29] wallyworld: this is the same for openstack [00:30] wallyworld: though if that is correct, and control-bucket will never be an unknownAttr, it begs the question - why check if it's set at all? [00:52] waigani: you are misunderstanding what UnknownAttrs are [00:52] they are attrs specific to different environment types [00:53] control bucket will always be there [00:53] there are other areas in the codebase that will fail if control bucket defaults to "" [00:53] are you sure you didn't have a value set in your env yaml for that bootstrap? [00:54] wallyworld: ah, sorry that makes sense. Let me check... [00:55] wallyworld: nothing set in environments.yaml let me read the code again and have another crack at understanding what's going on [00:58] waigani: also, if bucket name is "", bootstrap will not fail but errors will be logged (i haven't seen the code before but just checked then) [00:58] so the tl;dr; is we must not allow bucket name to be "" [01:03] wallyworld: no errors all-machines.log on machine 0 [01:13] waigani: does the jenv file have a control bucket attr? [01:24] wallyworld: not in .jenv I just bootstrapped again and dumped out the attrs - only access-key and secret-key are set, contol-bucket is not passed through. It is later set by the prepare func. It looks as though empty attrs are being removed. [01:28] Bug #1450265 was opened: juju depends on google.golang.org/cloud/compute/metadata but is not included in dependencies.tsv [01:32] wallyworld: actually no. I still think configFields and configDefaults are used to validate the unknownAttributes - to make sure they are expected attrs for the provider and not typos and the values are of the right type. They are not being passed through as default values to prepare. The config values are being read from environments.yaml [01:34] waigani: when control bucket is generated, it should end up in jenv, if it's not there, that means there's an issue [01:36] wallyworld: after bootstrap it's there, before it's not. [01:37] correct, it will not be there until after [01:37] wallyworld: yep, that's whats happening [01:38] I bootstrapped again to double check that [01:38] hmm, i can't see how it's being generated then if it's in the map (but "") [01:38] not without the extra != "" check [02:28] Bug #1450265 changed: juju depends on google.golang.org/cloud/compute/metadata but is not included in dependencies.tsv [02:44] wallyworld: can you please review https://github.com/juju/charm/pull/125 ? [02:44] syre [02:47] ta [02:47] axw: you may have seen, 1.24 and master now have no storage ff and also have upgrade step for block devices [02:47] wallyworld: I have, thank you [02:47] had minor merge conflicts [02:48] just minor, that's good [02:50] davecheney: constabulary is a terrible name for a github org [02:52] anyone used the canonical VPN? I can't understand their directions... seems like they're missing a step [02:53] natefinch: yes, what bit are you stuck on? [02:55] axw: thanks. So, I installed network-manager-openvpn-gnome, restarted network manager, then the instructions say: [02:55] Select network manager -> VPN Connections -> Configure VPN... [02:55] ....select network manager from where? [02:55] ....I think I just fdigured it out [02:55] they mean the network icon in the top right there [02:55] yep [02:55] .. I had no idea that was called the network manager :/ [02:58] axw: ok, another dumb question - how do I get the select file dialog to show dot files? it says to pick the file from the ~/.sesame directory, but the file picker isn't showing hidden directories. [02:58] ericsnow: can you confirm that the fix for bug 1447446 made it into the 1.24 branch and master? (and update the ticket accordingly) [02:58] Bug #1447446: 1.23.1: bootstrap failure, vivid, local provider

[02:59] natefinch: you can use Ctrl+L to enter the location [02:59] not sure if there's a better way, that's what I do [02:59] ericsnow: it was one the tickets that was incorrectly targetted [02:59] ah, right click has "show hidden files" [02:59] OMG... why doesn't that show all the time :/ [03:00] ahh yeah, I didn't think to right click there... much better [03:02] menn0: done [03:02] ericsnow: cheers [03:04] Bug #1447446 changed: 1.23.1: bootstrap failure, vivid, local provider

[03:11] axw: Thanks for the help, got it working. . [03:11] sinzui: you around? [03:11] natefinch: cool [03:38] axw, menn0: do you guys know how to make the JFDI thing on github work? I swear I got the syntax right, but it's not going through (still says "doesn't match fixes-blah") [03:38] natefinch: $$__JFDI__$$ [03:38] :/ did that [03:39] natefinch: that's what I would have said [03:39] wonder if the functionality got broken at somepoint (or maybe it just doesn't like me) [03:39] hahaha ... or maybe I mispelled jfdi... :/ [03:39] natefinch: hum, not sure what's up with that. I've been doing that today, so it's not broken in general [03:40] JDFI is not gonna work [03:40] heh [03:40] that'll do it :) [03:40] natefinch: maybe check one of axw's uses of it today? [04:08] menn0: I'm pretty sure that change went in before 1.24 was cut... I'll verify [04:09] jw4: thanks! i'm not saying it make it I just wanted to be sure. the ticket wasn't targetted correctly so it could have been missed. just update the ticket once you've checked. [04:09] menn0: will do [04:10] Bug #1450299 was opened: api/client: test fails on ppc64le

[04:14] wallyworld: the bucket val is being set because it's not in the map. the validation (which would set the default) happens later. [04:14] wallyworld: I've sent an email with the details [04:15] ok, ty, will look [04:21] waigani: thanks for explanation. i just wanted to be 100% sure that setting the default to "" wouldn't accidentally fail to generate a control bucket if one were not specified, and that config stuff is a bit of a mess which we've managed to break before :-( [04:22] wallyworld: no, I appreciate it. I now understand what's going on much more, so thanks :) [04:23] waigani: i did too 2 years back but have since only vaugue recollections (nightmares?) about config [04:23] haha [04:23] the schema default vs omit stuff is really hard to get right [04:23] and it really is a convoluted web [04:24] and we have relased juju which broke set ups in the wild [04:25] yeah, it would be good to see if there is a better design pattern that could be used to clear things up a bit [04:25] until then, comments are really important! [04:28] wallyworld: so is that a shipit? (not that we can land anything) [04:28] waigani: yeah, but if it's a bug fix for 1.24, feel free to jfdi at this point [04:29] wallyworld: thanks [07:26] axw: this fixes the critical blocker http://reviews.vapour.ws/r/1532/ [07:26] hopefully [07:49] morning o/ [07:50] hmm, got a fix for 1.24 but CI doesn't accept it *sigh* [08:00] wallyworld: reviewed [08:00] ty [08:06] TheMue: if the bug you're fixing is actually urgent, you have discretion - but it seems that issue doesn't actually break our bundle test on maas (it probably should) [08:07] mgz: it's a port that of a critical bug in 1.23 to 1.24, where it is only high. don't know why. [08:08] mgz: the bug is #1445063 [08:08] Bug #1445063: addressable containers cannot resolve non-FQDN in maas

[08:08]

[08:08] TheMue: that actually makes some sense - we couldn't release a 1.23 with it, but we have time before we're putting out 1.24 so a [08:08] and it's not actually stopping any of *us* doing work [08:09] ic [08:09] whereas the deployer stuff being borked prevents us improving our tests to catch issues like lxc networking problems :) [08:09] mgz: just saw that dimiter commented is with JFDI [08:09] yeah, it's fine [08:10] we don't gain anything from doing it but it's not really harmful [08:11] that's great, thx [08:13] axw: updated, i also added an index, i think it may help with the sorts [08:15] wallyworld: you didn't explain the change from updated to created though. is it for performance? [08:15] axw: oh, i see what you're asking, sorry missed the point [08:16] updated is only to the nearest second [08:16] we need much finer grained to get the order right [08:16] most timestamps we store in mongo for juju are to the nearest second [08:18] wallyworld: so why not store a number somewhere and just increment it each time you need to create an entry? using time is fragile, whatever the resolution [08:19] could do, but creating a sequence seems overhead for what is being done here, it's just charm status entries, hardly expected to change even a few times per second [08:20] and the sort is done after filtering for the unit [08:22] wallyworld: if you're confident with that, okay. I would just like to point out that it's trivial to do, and you could drop the additional index. you could just use State.sequence(), and use the result as the _id field [08:22] anyway [08:22] * axw takes a final look [08:22] ok, i'll do that [08:26] wallyworld: if you get a mo, can I have a stamp on [08:26] (thanks for the status info) [08:26] sure give me a sec [08:28] it is sadly much larger now, but the commit history should make sense to you [08:44] axw: sequence number used [08:45] wallyworld: thanks [08:45] shipit [08:45] np, thanks for pointing it out [08:46] mgz: i need dinner, i'll look at branch after [08:55] wallyworld: no probs, won't be landing till this evening I guess [09:37] Wallyworld: nice catch on the legacy status [09:37] perrito666: yeah. also found issues with history, incl tests not wired up :-) [09:42] The test for legacy was long due (which test was not wired I just briefly went trough the diff and didn't notice them) [09:42] perrito666: status_test.go - the status history tests [09:43] so they were never run nd hence missed picking up bugs [09:48] Duh, just saw it dang [09:48] Btw [09:48] + c.Assert(len(history), gc.Equals, 100) c.Assert(history, gc.HasLen, 100) [09:48] You are checking twice the same [09:48] oh i added that to make the failures easier to debug [09:48] i'll remove thanks [09:49] Sorry for that not being a comment I am in the phone [09:49] np :-) [09:50] And getoldesttimetokeep no longer gets time so it might need a name change and comment correction :) [09:51] perrito666: yeah true, same sentiment though :-) [09:51] mgz: you around? [09:53] natefinch: I am here [09:57] mgz: I;m trying to connect to stilson-7, but even connected to the VPN, I can't even ping the server, and ssh never connects... any ideas? [09:57] mgz: also, does it matter if I use the US or UK VPN? [10:03] natefinch: shouldn't [10:06] natefinch: I can get in via our ssh bouncer [10:07] if the vpn isn't playing ball for you [10:08] natefinch: also ssh -vv is useful for general debugging [10:08] mgz: that might be a good option. I don't know what's up with the VPN... but if all I'm supposed to need to do is connect to it, and then ssh... yeah, it's not working. [10:08] mgz: yeah, it just gets stuck at connecting to the IP..... verifying, stilson-07 is 10.245.67.135 ? [10:19] natefinch: yup - you should see that ip via either the vpn or the bouncer [10:32] lunching now [10:56] lunch, afk === liam_ is now known as Guest24325 [11:38] Bug #1450437 was opened: Juju bootstrap fails with sub error code 1 [13:00] sinzui: ping me when you are around please [13:00] perrito666, I have just arrived, though a little low on caffeine [13:01] I sadly have been awake for too much already [13:02] so, I have a fix for https://bugs.launchpad.net/juju-core/+bug/1447853 that should go into 1.22, 1.23 and 1.24 and master :p can I do that? I mean push a fix to 1.22 [13:02] Bug #1447853: Local charms are not added to storage on upgrade to 1.22.x

[13:13] mgz, r=me for joyent, but you conflict with my changes in check_blockers [13:21] katco, can you ask someone to read the new comment on bug 1437266. Do we need this fixed in 1.24 or 1.25? [13:21] Bug #1437266: Bootstrap node occasionally panicing with "not a valid unit name"

[13:30] natefinch, Are bug 1340749 and bug 1450437 the same as bug 1412621. Can I make them dupes? [13:30] Bug #1340749: Replicaset initiation failure reports wrong error

[13:30] Bug #1450437: Juju bootstrap fails with sub error code 1 [13:30] Bug #1412621: replica set EMPTYCONFIG MAAS bootstrap

[13:32] dooferlad: TheMue: alexisb: I'm off to the dentist [13:32] bbiab [13:32] voidspace: enjoy! [13:33] voidspace: just a check or more? have to do that the next days too ;) [13:33] voidspace: yes, enjoy [13:33] a check, but for all the family... [13:34] sinzui: dont forget my question now that you are caffeinated :p [13:35] perrito666, 1.22 is superseded in our PPAs by 1.23. and the packaging rules know it is an error to try to release a lower version [13:36] soo, 1.23, 1.24, master? [13:38] perrito666, alexisb, mramm, and xwwt to agree to purge our ppa and rush a 1.22 fix out...which means 1.23 will be released next week, not today [13:43] perrito666, yes. I will add the 1.23 task to the bug [13:57] Bug #1450437 changed: Juju bootstrap fails with sub error code 1 [14:00] sinzui: i don't know whether or not bug 1437266 need be fixed in 1.24 or 1.25. it looks like it only affects a few people, so probably 1.25? we should discuss in the release call [14:00] Bug #1437266: Bootstrap node occasionally panicing with "not a valid unit name"

[14:02] ericsnow: stand up [14:03] ericsnow: stup? [14:04] sinzui: I'll take a look at those bugs in a few minutes === kadams54 is now known as kadams54-away [14:48] question about upgrades, when upgrading to a minor, are the steps for that whole mayor rerun? [14:53] perrito666: not sure. I would hope not. [14:53] natefinch: they are supposed to be idempotent [14:54] perrito666: then why do you care? :) [14:56] natefinch: I need them to [14:58] perrito666, we *should* run any steps that haven't already been run, but I'm pretty we shouldn't rerun them (ok, if we do by accident they should be idempotent... but still) [14:58] perrito666, why do you need them? === kadams54-away is now known as kadams54 [15:42] can anyone stamp this oneliner? http://reviews.vapour.ws/r/1536/ [15:43] natefinch: [15:44] katco: ericsnow ? [15:44] * ericsnow takes a look [15:44] perrito666: looking... are there no tests for this stuff? [15:45] perrito666: nice 6 line oneliner :) [15:45] right? [15:45] natefinch: there are but the migration step was already there in the wrong version [15:45] ericsnow: I meant another work but since I dont know it in english my brain replaced with oneliner [15:45] sorry [15:45] word* [15:45] perrito666: no worries :) [15:45] natefinch: or you mean tests for the contents of the step? [15:46] perrito666: I mean... this code was wrong before, and obviously no test was failing because of it. It would be nice if there were a test that would fail if your addition was not there. [15:46] natefinch: I expect the testing is already there in state/upgrades_test.go [15:46] natefinch, perrito666: ah, there should be a test that checks that the step was run [15:47] ericsnow: yes, I should have committed it :p [15:48] sorry I had all these changes in patches that a testing script applied after checkout [15:49] perrito666: looks like the test is missing in steps122_test.go [15:49] (or the step, rather) [15:49] perrito666: so TestStateStepsFor122 should be failing [15:50] * perrito666 looks why his unstaged changes checker did not fail [15:52] fixed [15:52] ericsnow: can I consider you a holy reviewer or do I need a signature on that stamp? [15:52] perrito666: I wasn't going to bring it up [15:53] perrito666: i trust ericsnow's judgement [15:53] perrito666: go with it [15:54] perrito666: me too [15:54] so do I it is the burocratic weight of the stamp what I was looking for, but this is heavy enough for me to let it pass [15:55] perrito666: I rubber stamped it [15:55] natefinch: I already hit merge anyway :p [15:55] I am hurried enough [15:57] wow, those are some... ummm.... high level tests [15:57] natefinch: ?? [15:58] there is a separate test for the function being run :) if that is what worries you [15:58] sorry, high level isn't the right word..... what's the opposite of comprehensive? [15:58] perrito666: ahh, ok [15:58] perrito666: just with the changes, it looks like all we're testing is that the upgrade step has the right description [15:59] natefinch: but again, htat was already there, only in the wrong version, the merge of the patch got delayed until 1.22 so the steps where added in a place where they would never run [15:59] natefinch: well there is no way to test that the upgrade function does what it says [16:00] we check that the steps are the right ones based on their description, we know that at least that is what we want to run [16:18] perrito666: I'm sure there's a way to test that the upgrade function at least calls AddEnvUUIDToCharms.... [16:25] mgz, sinzui: I'm trying to connect to stilson, but can't seem to make the connection no matter how I do it. pure VPN doesn't work for me, I just never can get a response from the IP address. Martin told me about the CI bouncer, and I can connect to it, but I think I must be messing up the proxy command, because it never hops to stilson from the bouncer. [16:30] natefinch, I will get you a ssh config stanza [16:31] sinzui: thank you. I'll be afk for an hour-ish, so no rush === natefinch is now known as natefinch-afk [16:34] ericsnow: nttac meeting [16:34] nttca [17:32] ericsnow: hey can you join the nttca meeting again? === natefinch-afk is now known as natefinch [17:45] Bug #1450573 was opened: HA and backup recovery tests failed

[17:49] anyone willing for a review of a forward port of a fix from 1.23 and 1.24 to now master? see http://reviews.vapour.ws/r/1538/. thx [17:51] TheMue: you should reference the original PR so that it's easy to check that the port is the same as the original [17:55] natefinch: running a little behind, may be slightly late [17:55] katco: no worries. I know how it is. [17:56] natefinch: will add it [18:00] natefinch: done === cmars` is now known as cmars [18:02] TheMue: ship it! [18:02] natefinch: thx [18:07] TheMue: seems like we should be able to have an automated check that a forward port is identical to an existing PR on another branch and skip the second review. obviously, if changes needed to be made to the PR for the forward port, then it needs human eyes, but for ones like this where it's identical... would be nice to skip that. [18:08] natefinch: sounds like a good idea. but now also the fix doesn't match the listed fixes, so I've got to use JNDI [19:05] hmm, my CI output shows "Extant directories unknown: gopkg.in/juju/charm.v6-unstable". any idea what this means? [19:11] natefinch: any idea here ^^ [19:11] TheMue: no idea. sinzui? ^ [19:13] natefinch, are you using the make-release-tarball script? [19:13] natefinch, That error looks like the checks that ensure every package in the tarball is documented. [19:14] sinzui: it's me, and I'm only trying to merge a PR into master. [19:14] sinzui: only 5 files changes, dependencies are unchanged [19:16] TheMue, I think one of things you are trying to merge is injecting charm.v6-unstable. either we document that it is needed, or fix the code to not get it [19:19] sinzui: strange, I only ported a fix, a few code lines [19:20] TheMue: if you brought charm v6 definitely its a bigger change, we where using v5 yesterday [19:21] perrito666: no, I haven't. no change of imports. and I just scanned the code here, no where found [19:24] abentley, jog, something in CI is very wrong. it is trying to upgrade from 1.23.2 to 1.22.2. I think the upgrade-jenkins script was run a few hours too early [19:24] * sinzui looks for downgrade options [19:25] sinzui: on the bright side, upgrade-jenkins-branches ran to completion without errors :-/ [19:25] yep Ci is on the new 1.23.2, but we weren't ready for it [19:26] abentley, I will take this opportunity to downgrade to 1.21.1 to watch the upgrade [19:28] I just got a tax bill for $326,000 for the sale of my home 2 years. ago. I think I am going to pieces right now [19:29] sinzui: Wow. Ugh. [19:29] sinzui: Yeah, you might want to take some time out. [19:29] ok, I'll retry then tomorrow [19:30] abentley, I think I need an account to explain to the IRS that I did pay tax, on the sale of my house, not the sale of a business under different tax lass [19:30] law === kadams54 is now known as kadams54-away [19:51] perrito666, We had a misfire with the release of the new juju. all the CI machines got upgraded during the test. So it tried an impossible upgrade of 1.23.2 to 1.22.3 :( [19:52] perrito666, But CI has a cache of all the old jujus. I installed 1.21.3, then reset all the upgrade tests. All is well! The logs show after upgrade we could set new charms configs with and juju did the right thing \o/ [19:53] I will update CI to the real stable when we are finished testing 1.22.3 [19:53] wow, and we just got a bless [20:00] ericsnow: running just a little late brt [20:00] katco: k [20:02] Sinzui cool can i fwport now? [20:03] perrito666, yep. [20:04] Sweet [20:09] perrito666, should I make CI wait for your 1.23 merge? if it will be several hours, I would test 1.24 [20:12] I am at the dentist currently so it should be 1.5h from now :p [20:16] There is a limit to the things you can do with a phone [20:23] natefinch, thank you! [20:24] natefinch, maybe I should also grab the old broken package and run in there too [20:24] sinzui: certainly seems like a good idea. I wish I knew more of how the packaging process worked, I might be of more help. But I'm glad I was able to at least do this much. [20:28] natefinch, the container doesn't have our build deps ppa attached, so it is pure trusty, but I don't see anything in the ppa that could uncouple: https://launchpad.net/~juju/+archive/ubuntu/golang/+packages [20:28] regardless, natefinch you have given me something I can work with [20:31] sinzui: I am really glad it's useful. [20:36] TheMue, CI s broken as you described in your own branch. I think an upstream change to a repo has broken all builds [20:37] TheMue, mgz discovered some months ago that Go will always get all the deps in the git master branch even when they will not be used. It pollutes the tree. mgz's fix was to fix the upstream repo. === \b is now known as benonsoftware [21:06] davecheney: morning [21:09] katco, wallyworld: we need to form an angry mob and address bug 1450631. [21:09] Bug #1450631: Something is injecting gopkg.in/juju/charm.v6-unstable into the tree

[21:09] I will send an email to the list since it break all merges and CI testing [21:19] sinzui: sorry i was in a meeting... TAL now [21:22] Bug #1450631 was opened: Something is injecting gopkg.in/juju/charm.v6-unstable into the tree

[21:24] sinzui: it looks like it may be the github.com/juju/jujusvg repo that has a reference to gopkg.in/juju/charm.v6-unstable [21:25] jw4: ty [21:25] sinzui, katco: I wonder if referring to jujusvg through gopkg.in would resolve this? [21:25] jw4, uhg, we didn't change our dep of github.com/juju/jujusvg to get it though. [21:25] jw4, maybe [21:26] ps jw4, did you get credentials to see test results? [21:26] sinzui: not yet [21:26] sinzui: alexisb is still working on that :) [21:26] okay, I will see If I can get that to you tommorow jw4 [21:28] sinzui: very likely the trigger: https://github.com/juju/jujusvg/pull/28 [21:29]