[01:12] wallyworld_: can I have a chat to you about one of the current CI blockers? [01:13] sure [01:13] wallyworld_: hangout? [01:14] menn0: https://plus.google.com/hangouts/_/canonical.com/tanzanite-stand [01:14] wallyworld_: i'm there now [01:45] menn0: ping me when you're back [01:48] menn0: i found where we have to update the cert stuff - i suspect that before we were using "*" as the cert host and that matched any string; now we also have IP addresses of the state server, we need to explicitly enumerate the fake hostnames we use [01:49] we enumerate localhost and juju-apiserver, but not juju-mongodb [01:50] this is in certupdater.go and the change to add juju-apiserver there was made back in Dec 14, so it looks like it's been a sleeper cell waiting all this time to explode [01:51] the fix would be simply to add juju-mongodb in there so our fake mongo server name matches [01:52] i don't think it was a good idea to use fake server names like we did when this cert stuff was first implemented [01:52] wallyworld_: yeah [01:52] wallyworld_: adding juju-mongodb is probably the best fix right now [01:53] wallyworld_: either that, or using separate certs for the API server and mongodb [01:53] wallyworld_: but that's a bigger change [01:55] menn0: so i think this is a one line fix then, plus test change. you were looking to make the fix? [01:56] wallyworld_: yeah, i'll do it [01:56] ok, i'll review [01:56] wallyworld_: I have a feeling there might also be a problem in 1.22 itself [01:56] wallyworld_: which I'm trying to prove/disprove now [01:56] could be yeah [01:56] wallyworld_: i.e. if you restart juju or mongo you could end up with mongo using the wrong cert [01:57] 1.22.1 is open for fixes, so there's scope to do a fix for 1.22 [01:57] i suspect you're right it will be an issue [01:58] wallyworld_: I just haven't been able to make it happen yet with 1.22 [01:58] surprising [02:25] wallyworld_: http://reviews.vapour.ws/r/1225/ [02:25] sure, will look in 5 [02:25] wallyworld_: there was a natural split between the changes anyway - worker/storageprovisioner will be done spearately [02:25] thanks [02:25] ok [02:46] wallyworld_: ok I can make this mongo cert problem happen with 1.22 alone but only if I change or remove the upstart script for juju's mongo [02:46] wallyworld_: and then restart jujud [02:46] wallyworld_: this convinces jujud to write out a new server.pem [02:46] joy, ok [02:47] wallyworld_: which has since been updated by the certupdater worker with specific addresses [02:47] wallyworld_: although it's hard to trigger I think it's worth fixing for 1.22 [02:47] we we need to patch 1.22, 1.23 and master [02:47] yep [02:47] win [02:48] wallyworld_: the fix should be the same for all if we're just adding juju-mongodb the list of addresses [02:48] yep [02:48] at least the fix is trivial [02:48] wallyworld_: i'll try it out now and see if that definitely fixes it [02:49] sounds good [02:56] wallyworld_: yep that works [02:56] updating bug 1434680 with findings and will then get patches ready [02:56] Bug #1434680: 1.22.0 cannot upgrade to 1.23-beta1 or 1.24-alpha1

[02:57] whoot [02:58] horrible bug [03:08] wallyworld_: yeah - really non-obvious [03:09] wallyworld_: from a security and least-surprise perspective it might be better to not use the same cert for mongodb and the apiserver [03:26] wallyworld: here's the fix for 1.22: http://reviews.vapour.ws/r/1226/ [03:27] axw: reviewed, i have asked for the fs cmd abstractions to be reworked to line up with what we currently do - we abstract at the logical api level, not the fs commands [03:27] menn0: looking [03:28] menn0: lgtm [03:28] wallyworld: thanks [03:28] wallyworld: thanks [03:28] axw: let me knoe if you disagree [03:29] wallyworld: some of those commands are very linux-specific, so can't really be abstracted I think. will see what makes sense and reply [03:29] wallyworld: re hidden file, I disagree since it's in the agent storage dir, not in the target one [03:30] I did have it hidden initially, then figured it wasn't useful since it's exposed directly to charms anyway [03:30] fair enough [03:30] axw: sure, np. i think though that the concept of "mount" or "tell me the target of this mount" could be abstracted eg on Windows it would be junctions or something [03:36] wallyworld: it's never easy... I think we're going to need some kind of special hack in 1.23 b/c of this bug [03:36] wallyworld: when upgrading from 1.22.0 the cert in stateservinginfo doesn't have "juju-mongodb" [03:37] wallyworld: when upgrading to 1.23 the updated server.pem gets written out using that cert, before certupdater can start and fix the problem [03:38] wallyworld: it's even to early for upgrade steps to fix them [03:38] menn0: hmmm, we could stop 1.22 graduating from proposed [03:38] wallyworld: that might actually be the best course of action [03:38] but there would still be some 1.22 deployments out there [03:39] i wonder if there's a manual work around we could document [03:39] wallyworld: the workaround would be to upgrade to 1.22.1 /before/ upgrading to 1.23 or later [03:41] wallyworld: or I can try and get some kind of hack in that fixes up the cert at jujud start time [03:41] i'd have to think it through - they started on 1.21, upgraded to 1.22.0 [03:41] would an upgrade to 1.22.1 fix it [03:42] wallyworld: it would as long as 1.22.1 doesn't touch the juju-db upstart script [03:42] wallyworld: b/c if that changes then the new (broken) server.pem gets written out [03:42] wallyworld: alternatively we could remove eric's change from 1.23 that touches the upstart script [03:43] wallyworld: but that leaves thing working by coincidence. it's pretty fragile. [03:43] yeah, that sounds like a recipe for future issues [03:43] * menn0 agrees [03:44] maybe jujud could force a cert regen when it starts up [03:44] but only if supported [03:44] ie upgrades from older systems don't regenreate the certs [03:44] wallyworld: well, you don't want to regen the whole cert, because it needs to be the same across state servers [03:45] wallyworld: I think we can do something targetted that just fixes up the server hostnames field [03:45] if req'd [03:45] right, sorry, that's what i meant [03:45] wallyworld: let me have a look and see what that would look like [03:45] ok [03:58] wallyworld: PTAL [03:58] sure [04:02] axw: ty, lgtm. i think it's nice to have the filesystem calls out of the main code [04:03] wallyworld: it's a bit clearer if nothing else [04:03] yeah [04:04] wallyworld: I'll have another branch up later that does the storageprovisioner bits, and then later today hopefully will have a branch up that drops the ec2 StartInstance volume code [04:04] wallyworld: for the latter, I need to extend storageprovisioner API to allow it to watch environ config [04:04] sounds good [04:39] wallyworld: here's a standalone POC that updates the DNSNames in the cert [04:39] http://paste.ubuntu.com/10657968/ [04:39] looking [04:40] wallyworld: i'm thinking we do something like this in jujud soon after it loads it's config [04:40] wallyworld: perhaps updating the config immediate after the first load so that it has the correct DNSNames set [04:41] for future jujud starts [04:41] wallyworld: most of this could live in the certs package [04:42] wallyworld: in fact I see there's already some helpers in the certs package which could be used with this cert update code [04:42] menn0: looks ok i think, and we'd abort of the cacert private key were missing, since in earlier juju we don't have it [04:43] wallyworld: ok [04:44] wallyworld: that would mean that people who had an old juju and then upgraded to 1.22.0 could still hit this problem [04:44] menn0: if there's no ca cert private key, the cert updater doesn't do anything, as you need that private key to regenerate the certs, and we used to just throw the private key waway once the ca cert was generated [04:44] wallyworld: cool cool [04:45] so no private key = no problem [04:45] wallyworld: i'm running out of day to get this done [04:45] i can look at it [04:45] wallyworld: cool [04:45] i'll update the ticket again [04:46] wallyworld: not sure if you saw that the fix for 1.22.1 has merged [04:46] menn0: is there a PR for master so we can unblock? [04:46] awesome, did notice yet [04:46] n't [04:47] i can forward port to master [04:47] wallyworld: no b/c we need to this cert updating hack for master and 1.23 [04:47] wallyworld: without it upgrade from 1.22.0 are still broken [04:47] yeah, we also need to unblock master [04:47] i'll port to 1.23 amd master [04:48] wallyworld: with the cert updating hack? [04:48] menn0: yeah, i'll write for 1.23 and then port [04:48] wallyworld: sounds good [04:48] and also the other fix that just landed [04:49] cool [04:49] thanks for looking into all this [04:49] np [04:49] wallyworld: it was kinda fun... and educational. digging into hard bugs always tends to teach you a few new things. [04:50] indeed [04:50] you have a strange idea of fun though :-) [04:50] i prefer a nice gass of red [04:56] wallyworld: well I like a good drink too :) [04:57] helps dealing with this stuff [04:57] wallyworld: just thinking... with this cert fixup hack it might be easier to just empty out the DNSNames and IPAddresses fields rather than try to include the right values [04:58] wallyworld: the initial cert created at bootstrap is like this [04:58] and that seems to allow anything [04:58] the certupdater will put the right values back in when it runs [04:58] just a thought anyway [04:59] menn0: was thinking similar thoughts, but i'm not sure the cert updater would trigger, as there won't necessarily be new address change events [04:59] need to check [05:01] wallyworld: from what i've seen in the logs it always makes an update soon after it starts [05:01] wallyworld: but that might just be coincidence [05:02] yeah, i need to check [05:02] wallyworld: created a LK card here: https://canonical.leankit.com/Boards/View/103148069/113981127 [05:02] rightio [05:02] ty [05:03] ah shit [05:03] wallyworld: just saw another problem [05:03] wallyworld: the certs created by certupdater only last for a week [05:03] \o/ [05:04] wallyworld: so if they're also being used by mongodb [05:04] i thought they lasted longer than that, like years [05:04] wallyworld: but we only update the mongodb cert very occasionally [05:04] maybe they did and something changed [05:05] wallyworld: after bootstrap with 1.22: [05:05] $ sudo openssl x509 -in ~/.juju/local/server.pem -text -noout [05:05] [sudo] password for menno: [05:05] Certificate: [05:05] Data: [05:05] Version: 3 (0x2) [05:05] Serial Number: 0 (0x0) [05:05] Signature Algorithm: sha1WithRSAEncryption [05:05] Issuer: O=juju, CN=juju-generated CA for environment "local" [05:05] Validity [05:05] Not Before: Mar 16 03:32:17 2015 GMT [05:05] Not After : Mar 23 03:32:17 2025 GMT [05:05] ... [05:06] that's 10 years [05:06] and a week in the past [05:06] that's how i remembered it [05:06] wallyworld: sorry [05:06] :-) np [05:07] wallyworld: I've managed to read that 2025 as 2015 multiple times today then! [05:07] all good [05:07] i do that too [05:07] we have just one problem them :) [05:07] yeah, kust the one fairly big one [05:07] and with that I'm EOD today [05:07] ty again [06:19] wallyworld: the EBS provider won't work as-is :( CreateVolumes needs to know the AZ of the instance that will be attached to. So I guess VolumeParams.Attachment needs to stay, though I think it should only be used for advice on how to create the volume, rather than for attaching [06:20] axw: create volumes does need to know, yes :-( [06:20] axw: so i added that as a named attribute [06:21] wallyworld: it doesn't make sense to configure it, volumes are only ever created to be attached to an instance; an instance is in a specific AZ [06:22] true, i added the param const so something else could poke a value in the Attrubutes when needed, not at config time [06:22] to alleviate the need for Attachment params stuct [06:22] but whatever works best.... [06:23] school pickup, bbiab [06:42] Bug #1435152 was opened: Can't deploy local charm in non-server environment [07:45] jam, morning [07:45] dimitern: morning [07:45] jam, should we do our 1:1? [07:45] dimitern: sure [07:45] brt [07:59] wallyworld: more changes are required to goamz. volumes created with the dynamic provider are always DeleteOnTermination=false, regardless of whether the persistent flag is set [07:59] wallyworld: we need to call ModifyInstanceAttribute to add a block-device mapping field it seems [08:01] oh [08:13] axw, is ModifyInstanceAttribute even implemented yet? [08:14] dimitern: nope [08:14] axw, ah, I was planning to do it to allow setting SourceDestCheck [08:14] axw, so if you'll need it for storage, I'd appreciate if you add that as well - it's just a flag [08:15] dimitern: sure, just trying to figure out my next steps. I'll make sure SourceDestCheck is in there when I do it [08:15] axw, cheers [08:16] wallyworld: this puts the nail in the coffin of the idea of removing VolumeParams.Attachment. you can't create a non-persistent volume without also creating an attachment at the same time (at least, not without more significant changes) [08:22] oh dear :-( [09:01] Bug #1435215 was opened: juju metadata validate-tools error [09:55] dimitern: ping [09:56] dimitern: I'm looking at using an attempt strategy for a test and was browsing provider/maas/environ.go [09:56] dimitern: can you confirm my suspicion that the use of shortAttempt in maasEnviron.getCapabilities() is bogus [09:56] dimitern: as it immediately returns any error [09:56] dimitern: jam: here's a fix for bug 1434680 http://reviews.vapour.ws/r/1229/ [09:56] Bug #1434680: Certificate generated by certupdater worker cannot be used by MongoDB

[09:56] dimitern: in fact it's worse! [09:57] dimitern: if the code succeeds (no error) then it will retry [09:57] dimitern: if it errors then it will return immediately! [09:57] voidspace, hey [09:57] dimitern: so the use of attempt means that on success it will try several times needlessly [09:57] voidspace, let's discuss it on the standup [09:57] dimitern: but stop on the first error - afaics [09:57] dimitern: ok :-) [09:57] wallyworld, looking [10:13] Bug #1435244 was opened: juju cycle dead lock [10:30] dimitern: hangout died! [10:30] brb [10:39] wallyworld: lgtm [10:40] jam: ty, will land to 1.23 and port to master [11:14] wallyworld: ping [11:14] menn0: hi [11:14] wallyworld: still working eh? [11:15] wallyworld: I just saw your fix for mongo cert issue [11:15] menn0: just about to merge to master, already fixed 1.23 [11:15] wallyworld: i'm a little concerned [11:16] wallyworld: when there's multiple state servers won't they all end up with different certs and keys? [11:16] about? [11:16] wallyworld: I was suprised that the fix generates completely new certs on startup instead of extending the existing one [11:17] so long as they all are generated off the master ca cert i think it will be ok [11:17] that's what happens now [11:17] ok cool [11:17] the cert updater now generates a new one [11:18] i think it's ok [11:18] yep, it's becoming clearer now [11:18] sorry [11:18] menn0: i do feel though the full 1.23 fix will need backporting to 1.22.1 [11:18] so each state server uses it's own cert signed by a central CA [11:18] yes [11:18] seems to work [11:18] for some reason I thought that all the state servers used the same cert [11:19] they used to [11:19] generated once only at bootstrap [11:19] but using separate certs is definitely more sensible [11:19] yeah [11:19] ignore me then [11:20] agreed that porting the full fix to 1.22.1 is probably good to do [11:20] removes some risk [11:27] menn0: good questions to ask [11:31] dooferlad, I'll be 5m late for our 1:1 btw [11:32] dimitern: no problem [11:37] Bug #1435283 was opened: juju occasionally switches a units public-address if an additional interface is added post-deployment [11:39] dooferlad, I'm in the hangout [12:13] is someone around to give me a quick review? http://reviews.vapour.ws/r/1232/ [12:45] mattyw, you have a review on 1005 [12:45] mattyw, why's deadbeef-0bad-f00d-0000-4b1d0d06f00d not a valid uuid btw? [12:48] dimitern, I assume because it's not a valid Version 4 UUID [12:49] dimitern - in a version 4 uuid the 13th hex digit is always 4 [12:49] tasdomas, oh, I see [12:49] tasdomas, thanks [13:07] Bug #1432331 was opened: better config interface [13:18] dimitern, tasdomas is right, the 14th digit needs to be hex, also the 18th needs to be 8,9,a,b or something like that, and it was 0 [14:18] ericsnow, I am in #server asking about bug 1434737. hallyn is our lxc expert. [14:18] Bug #1434737: lxc-start failure on vivid [14:25] ericsnow, natefinch: the internet here is very poor, I keep dropping from hangout [14:25] wwitzel3: no worries, we're almost done [14:25] wwitzel3: no worries [14:25] jinx! [14:43] sinzui: awesome, thanks [14:57] alexisb, I still hope for release note for "Service Leader Elections". [14:58] sinzui: FYI, I put as much info as I could find on that ticket, but couldn't remember exactly what vivid we're running and if we are upgrading from daily every day [14:58] sinzui, ack [14:59] ericsnow, #server reports that rbasak will be pushing a fix for us [14:59] sinzui: cool, thanks! [15:00] ericsnow, oh, auto updates on vivid-slave-b were stalled because I didn't let it restart services. I have fixed that [15:00] too [15:01] sinzui: ah, cool [15:01] sinzui: that runs daily or in response to some remote update? [15:02] Bug #1432331 changed: better config interface

[15:02] Bug #1434544 changed: backups broke non-linux builds

[15:02] Bug #1435215 changed: juju metadata validate-tools error [15:02] ericsnow, yes, Plus test several tests we run on the machine also do upgrades [15:02] ericsnow, it allows us to catch issues in new series packages...which just makes all our jobs harder [15:12] dimitern, natefinch: can you find someone to read the log for bug 1431286 and decide if we need more information? I cannot triage it [15:12] Bug #1431286: juju bootstrap fails when http_proxy is set in environments.yaml

[15:15] sinzui: the root cause in lp:1432683 seems to apply to the bug I opened (lp:1434737) [15:15] sinzui: so I'm hopeful :) [15:15] \o/ [15:15] sinzui: I should have thought to look at those logs earlier [15:19] sinzui: looking at the http_proxy bug.... not sure what's going on [15:20] Bug #1435244 changed: juju cycle dead lock [15:23] ericsnow, Do you know how to force apparmor reload on that machine? [15:23] sinzui: no [15:23] sinzui: I'm not familliar with apparmor [15:24] sinzui: reboot? [15:24] * natefinch has been a windows dev for a long time ;) [15:24] natefinch, I think we have done that several times now [15:24] sinzui: you must mean something different than I was thinking for reloading apparmor [15:25] natefinch, I think new profiles were added to the machine, but were not registered, so maybe I don't mean reload [15:26] sinzui: ahh, ok. I am also not familiar with apparmor, unfortunately [15:46] hi all, is SpecializeCharmRepo still used in juju-core? in essence, do we still support "test-mode" and "charm-store-auth" in the configuration? [15:46] AFAICT at least the latter does not have any effects [15:47] dimitern: ping [15:48] voidspace, in a call [15:48] dimitern: ok [15:48] natefinch: hey, you familiar much with the dummy provider? [15:49] natefinch: the dummy provider records the operations it does, presumably so you can make asserts that it was used in a particular way [15:49] natefinch: do you know of any examples of this in the code base I can look at? [15:50] voidspace: no, not really. I try to avoid it as much as possible. It is used fairly extensively, however [15:50] natefinch: ok, I'll grep around [15:50] natefinch: I do have an interface (that only requires one method to be implemented), maybe I should use that instead... [15:51] wwitzel3, I see you're OCR today - would you mind reviewing http://reviews.vapour.ws/r/1070/ and http://reviews.vapour.ws/r/1097/ ? [15:54] frankban: I haven't looked too deeply into it, but I do see code using that config option [15:56] if anyone else could also take a look at http://reviews.vapour.ws/r/1097/ and http://reviews.vapour.ws/r/1070/, I would very much appreciate that [16:01] natefinch: I also see it used, and it adds an Authorization header. I don't see how this affects the server side though (the legacy charm store) [16:01] voidspace, back [16:04] natefinch: the legacy charm store does not check auth headers AFAICT, so I wonder if the SetAuthAttrs can be removed [16:04] dimitern: do you know off the top of your head any test files likely to use the dummy provider ops for asserts about operations [16:04] dimitern: I'm currently trying with the interface, but I might still want to look in the operations log for other tests [16:05] voidspace, yeah, just a sec [16:05] dimitern: if you don't know off the top of your head don't worry - I can search myself [16:05] ah, ok [16:05] my ubuntu phone just arrived [16:07] voidspace, so you mean outside of the dummy provider tests, right? [16:07] dimitern: as a whitebox test using the dummy provider [16:07] dimitern: I haven't even looked at the dummy provider tests though... :-) [16:07] frankban: yeah, I don't know the internals of the new charm store stuff [16:07] dimitern: that would be a logical place to start - but I kind of assume they're whitebox [16:08] I mean as a *blackbox* test [16:08] voidspace, well, it's like this: dummy provider tests can use the OpXYZ internally to check calls [16:08] dimitern: the dummy provider tests are good [16:09] dimitern: they use dummy.Listen and provide a channel [16:09] voidspace, outside the dummy package, you could mostly just use the "broken" setting to selectively return errors from environ methods [16:09] I should have looked there [16:09] voidspace, yeah, in fact several of the environs.Networking methods are tested using those dummy features [16:10] dimitern: I have another question if you don't mind [16:10] voidspace, e.g. "break" AllocateAddress(), check what happens [16:10] voidspace, sure [16:10] dimitern: see this code http://pastebin.ubuntu.com/10661843/ [16:10] dimitern: it starts with a "releaser" interface [16:10] dimitern: followed by my attempt at a mock implementation [16:10] the mockReleaser [16:11] voidspace, yeah [16:11] but it doesn't compile, Go doesn't like ReleaseAddress being a pointer receiver [16:11] " mockReleaser does not implement addresser.releaser (ReleaseAddress method has pointer receiver) [16:11] " [16:11] I *need* it to have a pointer receiver [16:11] ah [16:11] NewMockReleaser [16:11]