| menn0 | wallyworld_: can I have a chat to you about one of the current CI blockers? | 01:12 |
|---|---|---|
| wallyworld_ | sure | 01:13 |
| menn0 | wallyworld_: hangout? | 01:13 |
| wallyworld_ | menn0: https://plus.google.com/hangouts/_/canonical.com/tanzanite-stand | 01:14 |
| menn0 | wallyworld_: i'm there now | 01:14 |
| wallyworld_ | menn0: ping me when you're back | 01:45 |
| wallyworld_ | menn0: i found where we have to update the cert stuff - i suspect that before we were using "*" as the cert host and that matched any string; now we also have IP addresses of the state server, we need to explicitly enumerate the fake hostnames we use | 01:48 |
| wallyworld_ | we enumerate localhost and juju-apiserver, but not juju-mongodb | 01:49 |
| wallyworld_ | this is in certupdater.go and the change to add juju-apiserver there was made back in Dec 14, so it looks like it's been a sleeper cell waiting all this time to explode | 01:50 |
| wallyworld_ | the fix would be simply to add juju-mongodb in there so our fake mongo server name matches | 01:51 |
| wallyworld_ | i don't think it was a good idea to use fake server names like we did when this cert stuff was first implemented | 01:52 |
| menn0 | wallyworld_: yeah | 01:52 |
| menn0 | wallyworld_: adding juju-mongodb is probably the best fix right now | 01:52 |
| menn0 | wallyworld_: either that, or using separate certs for the API server and mongodb | 01:53 |
| menn0 | wallyworld_: but that's a bigger change | 01:53 |
| wallyworld_ | menn0: so i think this is a one line fix then, plus test change. you were looking to make the fix? | 01:55 |
| menn0 | wallyworld_: yeah, i'll do it | 01:56 |
| wallyworld_ | ok, i'll review | 01:56 |
| menn0 | wallyworld_: I have a feeling there might also be a problem in 1.22 itself | 01:56 |
| menn0 | wallyworld_: which I'm trying to prove/disprove now | 01:56 |
| wallyworld_ | could be yeah | 01:56 |
| menn0 | wallyworld_: i.e. if you restart juju or mongo you could end up with mongo using the wrong cert | 01:56 |
| wallyworld_ | 1.22.1 is open for fixes, so there's scope to do a fix for 1.22 | 01:57 |
| wallyworld_ | i suspect you're right it will be an issue | 01:57 |
| menn0 | wallyworld_: I just haven't been able to make it happen yet with 1.22 | 01:58 |
| wallyworld_ | surprising | 01:58 |
| axw | wallyworld_: http://reviews.vapour.ws/r/1225/ | 02:25 |
| wallyworld_ | sure, will look in 5 | 02:25 |
| axw | wallyworld_: there was a natural split between the changes anyway - worker/storageprovisioner will be done spearately | 02:25 |
| axw | thanks | 02:25 |
| wallyworld_ | ok | 02:25 |
| menn0 | wallyworld_: ok I can make this mongo cert problem happen with 1.22 alone but only if I change or remove the upstart script for juju's mongo | 02:46 |
| menn0 | wallyworld_: and then restart jujud | 02:46 |
| menn0 | wallyworld_: this convinces jujud to write out a new server.pem | 02:46 |
| wallyworld_ | joy, ok | 02:46 |
| menn0 | wallyworld_: which has since been updated by the certupdater worker with specific addresses | 02:47 |
| menn0 | wallyworld_: although it's hard to trigger I think it's worth fixing for 1.22 | 02:47 |
| wallyworld_ | we we need to patch 1.22, 1.23 and master | 02:47 |
| menn0 | yep | 02:47 |
| wallyworld_ | win | 02:47 |
| menn0 | wallyworld_: the fix should be the same for all if we're just adding juju-mongodb the list of addresses | 02:48 |
| wallyworld_ | yep | 02:48 |
| wallyworld_ | at least the fix is trivial | 02:48 |
| menn0 | wallyworld_: i'll try it out now and see if that definitely fixes it | 02:48 |
| wallyworld_ | sounds good | 02:49 |
| menn0 | wallyworld_: yep that works | 02:56 |
| menn0 | updating bug 1434680 with findings and will then get patches ready | 02:56 |
| mup | Bug #1434680: 1.22.0 cannot upgrade to 1.23-beta1 or 1.24-alpha1 <ci> <regression> <upgrade-juju> <juju-core:Triaged by menno.smits> <juju-core 1.22:In Progress by menno.smits> <juju-core 1.23:In Progress by menno.smits> <https://launchpad.net/bugs/1434680> | 02:56 |
| wallyworld_ | whoot | 02:57 |
| wallyworld_ | horrible bug | 02:58 |
| menn0 | wallyworld_: yeah - really non-obvious | 03:08 |
| menn0 | wallyworld_: from a security and least-surprise perspective it might be better to not use the same cert for mongodb and the apiserver | 03:09 |
| menn0 | wallyworld: here's the fix for 1.22: http://reviews.vapour.ws/r/1226/ | 03:26 |
| wallyworld | axw: reviewed, i have asked for the fs cmd abstractions to be reworked to line up with what we currently do - we abstract at the logical api level, not the fs commands | 03:27 |
| wallyworld | menn0: looking | 03:27 |
| wallyworld | menn0: lgtm | 03:28 |
| menn0 | wallyworld: thanks | 03:28 |
| axw | wallyworld: thanks | 03:28 |
| wallyworld | axw: let me knoe if you disagree | 03:28 |
| axw | wallyworld: some of those commands are very linux-specific, so can't really be abstracted I think. will see what makes sense and reply | 03:29 |
| axw | wallyworld: re hidden file, I disagree since it's in the agent storage dir, not in the target one | 03:29 |
| axw | I did have it hidden initially, then figured it wasn't useful since it's exposed directly to charms anyway | 03:30 |
| wallyworld | fair enough | 03:30 |
| wallyworld | axw: sure, np. i think though that the concept of "mount" or "tell me the target of this mount" could be abstracted eg on Windows it would be junctions or something | 03:30 |
| menn0 | wallyworld: it's never easy... I think we're going to need some kind of special hack in 1.23 b/c of this bug | 03:36 |
| menn0 | wallyworld: when upgrading from 1.22.0 the cert in stateservinginfo doesn't have "juju-mongodb" | 03:36 |
| menn0 | wallyworld: when upgrading to 1.23 the updated server.pem gets written out using that cert, before certupdater can start and fix the problem | 03:37 |
| menn0 | wallyworld: it's even to early for upgrade steps to fix them | 03:38 |
| wallyworld | menn0: hmmm, we could stop 1.22 graduating from proposed | 03:38 |
| menn0 | wallyworld: that might actually be the best course of action | 03:38 |
| wallyworld | but there would still be some 1.22 deployments out there | 03:38 |
| wallyworld | i wonder if there's a manual work around we could document | 03:39 |
| menn0 | wallyworld: the workaround would be to upgrade to 1.22.1 /before/ upgrading to 1.23 or later | 03:39 |
| menn0 | wallyworld: or I can try and get some kind of hack in that fixes up the cert at jujud start time | 03:41 |
| wallyworld | i'd have to think it through - they started on 1.21, upgraded to 1.22.0 | 03:41 |
| wallyworld | would an upgrade to 1.22.1 fix it | 03:41 |
| menn0 | wallyworld: it would as long as 1.22.1 doesn't touch the juju-db upstart script | 03:42 |
| menn0 | wallyworld: b/c if that changes then the new (broken) server.pem gets written out | 03:42 |
| menn0 | wallyworld: alternatively we could remove eric's change from 1.23 that touches the upstart script | 03:42 |
| menn0 | wallyworld: but that leaves thing working by coincidence. it's pretty fragile. | 03:43 |
| wallyworld | yeah, that sounds like a recipe for future issues | 03:43 |
| * menn0 agrees | 03:43 | |
| wallyworld | maybe jujud could force a cert regen when it starts up | 03:44 |
| wallyworld | but only if supported | 03:44 |
| wallyworld | ie upgrades from older systems don't regenreate the certs | 03:44 |
| menn0 | wallyworld: well, you don't want to regen the whole cert, because it needs to be the same across state servers | 03:44 |
| menn0 | wallyworld: I think we can do something targetted that just fixes up the server hostnames field | 03:45 |
| menn0 | if req'd | 03:45 |
| wallyworld | right, sorry, that's what i meant | 03:45 |
| menn0 | wallyworld: let me have a look and see what that would look like | 03:45 |
| wallyworld | ok | 03:45 |
| axw | wallyworld: PTAL | 03:58 |
| wallyworld | sure | 03:58 |
| wallyworld | axw: ty, lgtm. i think it's nice to have the filesystem calls out of the main code | 04:02 |
| axw | wallyworld: it's a bit clearer if nothing else | 04:03 |
| wallyworld | yeah | 04:03 |
| axw | wallyworld: I'll have another branch up later that does the storageprovisioner bits, and then later today hopefully will have a branch up that drops the ec2 StartInstance volume code | 04:04 |
| axw | wallyworld: for the latter, I need to extend storageprovisioner API to allow it to watch environ config | 04:04 |
| wallyworld | sounds good | 04:04 |
| menn0 | wallyworld: here's a standalone POC that updates the DNSNames in the cert | 04:39 |
| menn0 | http://paste.ubuntu.com/10657968/ | 04:39 |
| wallyworld | looking | 04:39 |
| menn0 | wallyworld: i'm thinking we do something like this in jujud soon after it loads it's config | 04:40 |
| menn0 | wallyworld: perhaps updating the config immediate after the first load so that it has the correct DNSNames set | 04:40 |
| menn0 | for future jujud starts | 04:41 |
| menn0 | wallyworld: most of this could live in the certs package | 04:41 |
| menn0 | wallyworld: in fact I see there's already some helpers in the certs package which could be used with this cert update code | 04:42 |
| wallyworld | menn0: looks ok i think, and we'd abort of the cacert private key were missing, since in earlier juju we don't have it | 04:42 |
| menn0 | wallyworld: ok | 04:43 |
| menn0 | wallyworld: that would mean that people who had an old juju and then upgraded to 1.22.0 could still hit this problem | 04:44 |
| wallyworld | menn0: if there's no ca cert private key, the cert updater doesn't do anything, as you need that private key to regenerate the certs, and we used to just throw the private key waway once the ca cert was generated | 04:44 |
| menn0 | wallyworld: cool cool | 04:44 |
| wallyworld | so no private key = no problem | 04:45 |
| menn0 | wallyworld: i'm running out of day to get this done | 04:45 |
| wallyworld | i can look at it | 04:45 |
| menn0 | wallyworld: cool | 04:45 |
| menn0 | i'll update the ticket again | 04:45 |
| menn0 | wallyworld: not sure if you saw that the fix for 1.22.1 has merged | 04:46 |
| wallyworld | menn0: is there a PR for master so we can unblock? | 04:46 |
| wallyworld | awesome, did notice yet | 04:46 |
| wallyworld | n't | 04:46 |
| wallyworld | i can forward port to master | 04:47 |
| menn0 | wallyworld: no b/c we need to this cert updating hack for master and 1.23 | 04:47 |
| menn0 | wallyworld: without it upgrade from 1.22.0 are still broken | 04:47 |
| wallyworld | yeah, we also need to unblock master | 04:47 |
| wallyworld | i'll port to 1.23 amd master | 04:47 |
| menn0 | wallyworld: with the cert updating hack? | 04:48 |
| wallyworld | menn0: yeah, i'll write for 1.23 and then port | 04:48 |
| menn0 | wallyworld: sounds good | 04:48 |
| wallyworld | and also the other fix that just landed | 04:48 |
| menn0 | cool | 04:49 |
| wallyworld | thanks for looking into all this | 04:49 |
| menn0 | np | 04:49 |
| menn0 | wallyworld: it was kinda fun... and educational. digging into hard bugs always tends to teach you a few new things. | 04:49 |
| wallyworld | indeed | 04:50 |
| wallyworld | you have a strange idea of fun though :-) | 04:50 |
| wallyworld | i prefer a nice gass of red | 04:50 |
| menn0 | wallyworld: well I like a good drink too :) | 04:56 |
| wallyworld | helps dealing with this stuff | 04:57 |
| menn0 | wallyworld: just thinking... with this cert fixup hack it might be easier to just empty out the DNSNames and IPAddresses fields rather than try to include the right values | 04:57 |
| menn0 | wallyworld: the initial cert created at bootstrap is like this | 04:58 |
| menn0 | and that seems to allow anything | 04:58 |
| menn0 | the certupdater will put the right values back in when it runs | 04:58 |
| menn0 | just a thought anyway | 04:58 |
| wallyworld | menn0: was thinking similar thoughts, but i'm not sure the cert updater would trigger, as there won't necessarily be new address change events | 04:59 |
| wallyworld | need to check | 04:59 |
| menn0 | wallyworld: from what i've seen in the logs it always makes an update soon after it starts | 05:01 |
| menn0 | wallyworld: but that might just be coincidence | 05:01 |
| wallyworld | yeah, i need to check | 05:02 |
| menn0 | wallyworld: created a LK card here: https://canonical.leankit.com/Boards/View/103148069/113981127 | 05:02 |
| wallyworld | rightio | 05:02 |
| wallyworld | ty | 05:02 |
| menn0 | ah shit | 05:03 |
| menn0 | wallyworld: just saw another problem | 05:03 |
| menn0 | wallyworld: the certs created by certupdater only last for a week | 05:03 |
| wallyworld | \o/ | 05:03 |
| menn0 | wallyworld: so if they're also being used by mongodb | 05:04 |
| wallyworld | i thought they lasted longer than that, like years | 05:04 |
| menn0 | wallyworld: but we only update the mongodb cert very occasionally | 05:04 |
| wallyworld | maybe they did and something changed | 05:04 |
| menn0 | wallyworld: after bootstrap with 1.22: | 05:05 |
| menn0 | $ sudo openssl x509 -in ~/.juju/local/server.pem -text -noout | 05:05 |
| menn0 | [sudo] password for menno: | 05:05 |
| menn0 | Certificate: | 05:05 |
| menn0 | Data: | 05:05 |
| menn0 | Version: 3 (0x2) | 05:05 |
| menn0 | Serial Number: 0 (0x0) | 05:05 |
| menn0 | Signature Algorithm: sha1WithRSAEncryption | 05:05 |
| menn0 | Issuer: O=juju, CN=juju-generated CA for environment "local" | 05:05 |
| menn0 | Validity | 05:05 |
| menn0 | Not Before: Mar 16 03:32:17 2015 GMT | 05:05 |
| menn0 | Not After : Mar 23 03:32:17 2025 GMT | 05:05 |
| menn0 | ... | 05:05 |
| wallyworld | that's 10 years | 05:06 |
| wallyworld | and a week in the past | 05:06 |
| wallyworld | that's how i remembered it | 05:06 |
| menn0 | wallyworld: sorry | 05:06 |
| wallyworld | :-) np | 05:06 |
| menn0 | wallyworld: I've managed to read that 2025 as 2015 multiple times today then! | 05:07 |
| menn0 | all good | 05:07 |
| wallyworld | i do that too | 05:07 |
| menn0 | we have just one problem them :) | 05:07 |
| wallyworld | yeah, kust the one fairly big one | 05:07 |
| menn0 | and with that I'm EOD today | 05:07 |
| wallyworld | ty again | 05:07 |
| axw | wallyworld: the EBS provider won't work as-is :( CreateVolumes needs to know the AZ of the instance that will be attached to. So I guess VolumeParams.Attachment needs to stay, though I think it should only be used for advice on how to create the volume, rather than for attaching | 06:19 |
| wallyworld | axw: create volumes does need to know, yes :-( | 06:20 |
| wallyworld | axw: so i added that as a named attribute | 06:20 |
| axw | wallyworld: it doesn't make sense to configure it, volumes are only ever created to be attached to an instance; an instance is in a specific AZ | 06:21 |
| wallyworld | true, i added the param const so something else could poke a value in the Attrubutes when needed, not at config time | 06:22 |
| wallyworld | to alleviate the need for Attachment params stuct | 06:22 |
| wallyworld | but whatever works best.... | 06:22 |
| wallyworld | school pickup, bbiab | 06:23 |
| mup | Bug #1435152 was opened: Can't deploy local charm in non-server environment <juju-core:Triaged by waigani> <https://launchpad.net/bugs/1435152> | 06:42 |
| dimitern | jam, morning | 07:45 |
| jam | dimitern: morning | 07:45 |
| dimitern | jam, should we do our 1:1? | 07:45 |
| jam | dimitern: sure | 07:45 |
| jam | brt | 07:45 |
| axw | wallyworld: more changes are required to goamz. volumes created with the dynamic provider are always DeleteOnTermination=false, regardless of whether the persistent flag is set | 07:59 |
| axw | wallyworld: we need to call ModifyInstanceAttribute to add a block-device mapping field it seems | 07:59 |
| wallyworld | oh | 08:01 |
| dimitern | axw, is ModifyInstanceAttribute even implemented yet? | 08:13 |
| axw | dimitern: nope | 08:14 |
| dimitern | axw, ah, I was planning to do it to allow setting SourceDestCheck | 08:14 |
| dimitern | axw, so if you'll need it for storage, I'd appreciate if you add that as well - it's just a flag | 08:14 |
| axw | dimitern: sure, just trying to figure out my next steps. I'll make sure SourceDestCheck is in there when I do it | 08:15 |
| dimitern | axw, cheers | 08:15 |
| axw | wallyworld: this puts the nail in the coffin of the idea of removing VolumeParams.Attachment. you can't create a non-persistent volume without also creating an attachment at the same time (at least, not without more significant changes) | 08:16 |
| wallyworld | oh dear :-( | 08:22 |
| mup | Bug #1435215 was opened: juju metadata validate-tools error <juju-core:New> <https://launchpad.net/bugs/1435215> | 09:01 |
| voidspace | dimitern: ping | 09:55 |
| voidspace | dimitern: I'm looking at using an attempt strategy for a test and was browsing provider/maas/environ.go | 09:56 |
| voidspace | dimitern: can you confirm my suspicion that the use of shortAttempt in maasEnviron.getCapabilities() is bogus | 09:56 |
| voidspace | dimitern: as it immediately returns any error | 09:56 |
| wallyworld | dimitern: jam: here's a fix for bug 1434680 http://reviews.vapour.ws/r/1229/ | 09:56 |
| mup | Bug #1434680: Certificate generated by certupdater worker cannot be used by MongoDB <ci> <regression> <upgrade-juju> <juju-core:Triaged by menno.smits> <juju-core 1.22:Fix Committed by menno.smits> <juju-core 1.23:In Progress by wallyworld> <https://launchpad.net/bugs/1434680> | 09:56 |
| voidspace | dimitern: in fact it's worse! | 09:56 |
| voidspace | dimitern: if the code succeeds (no error) then it will retry | 09:57 |
| voidspace | dimitern: if it errors then it will return immediately! | 09:57 |
| dimitern | voidspace, hey | 09:57 |
| voidspace | dimitern: so the use of attempt means that on success it will try several times needlessly | 09:57 |
| dimitern | voidspace, let's discuss it on the standup | 09:57 |
| voidspace | dimitern: but stop on the first error - afaics | 09:57 |
| voidspace | dimitern: ok :-) | 09:57 |
| dimitern | wallyworld, looking | 09:57 |
| mup | Bug #1435244 was opened: juju cycle dead lock <juju-core:New> <https://launchpad.net/bugs/1435244> | 10:13 |
| voidspace | dimitern: hangout died! | 10:30 |
| voidspace | brb | 10:30 |
| jam | wallyworld: lgtm | 10:39 |
| wallyworld | jam: ty, will land to 1.23 and port to master | 10:40 |
| menn0 | wallyworld: ping | 11:14 |
| wallyworld | menn0: hi | 11:14 |
| menn0 | wallyworld: still working eh? | 11:14 |
| menn0 | wallyworld: I just saw your fix for mongo cert issue | 11:15 |
| wallyworld | menn0: just about to merge to master, already fixed 1.23 | 11:15 |
| menn0 | wallyworld: i'm a little concerned | 11:15 |
| menn0 | wallyworld: when there's multiple state servers won't they all end up with different certs and keys? | 11:16 |
| wallyworld | about? | 11:16 |
| menn0 | wallyworld: I was suprised that the fix generates completely new certs on startup instead of extending the existing one | 11:16 |
| wallyworld | so long as they all are generated off the master ca cert i think it will be ok | 11:17 |
| wallyworld | that's what happens now | 11:17 |
| menn0 | ok cool | 11:17 |
| wallyworld | the cert updater now generates a new one | 11:17 |
| wallyworld | i think it's ok | 11:18 |
| menn0 | yep, it's becoming clearer now | 11:18 |
| menn0 | sorry | 11:18 |
| wallyworld | menn0: i do feel though the full 1.23 fix will need backporting to 1.22.1 | 11:18 |
| menn0 | so each state server uses it's own cert signed by a central CA | 11:18 |
| wallyworld | yes | 11:18 |
| wallyworld | seems to work | 11:18 |
| menn0 | for some reason I thought that all the state servers used the same cert | 11:18 |
| wallyworld | they used to | 11:19 |
| wallyworld | generated once only at bootstrap | 11:19 |
| menn0 | but using separate certs is definitely more sensible | 11:19 |
| wallyworld | yeah | 11:19 |
| menn0 | ignore me then | 11:19 |
| menn0 | agreed that porting the full fix to 1.22.1 is probably good to do | 11:20 |
| menn0 | removes some risk | 11:20 |
| wallyworld | menn0: good questions to ask | 11:27 |
| dimitern | dooferlad, I'll be 5m late for our 1:1 btw | 11:31 |
| dooferlad | dimitern: no problem | 11:32 |
| mup | Bug #1435283 was opened: juju occasionally switches a units public-address if an additional interface is added post-deployment <juju-core:New> <https://launchpad.net/bugs/1435283> | 11:37 |
| dimitern | dooferlad, I'm in the hangout | 11:39 |
| mattyw | is someone around to give me a quick review? http://reviews.vapour.ws/r/1232/ | 12:13 |
| dimitern | mattyw, you have a review on 1005 | 12:45 |
| dimitern | mattyw, why's deadbeef-0bad-f00d-0000-4b1d0d06f00d not a valid uuid btw? | 12:45 |
| tasdomas | dimitern, I assume because it's not a valid Version 4 UUID | 12:48 |
| tasdomas | dimitern - in a version 4 uuid the 13th hex digit is always 4 | 12:49 |
| dimitern | tasdomas, oh, I see | 12:49 |
| dimitern | tasdomas, thanks | 12:49 |
| mup | Bug #1432331 was opened: better config interface <juju-core:New> <https://launchpad.net/bugs/1432331> | 13:07 |
| mattyw | dimitern, tasdomas is right, the 14th digit needs to be hex, also the 18th needs to be 8,9,a,b or something like that, and it was 0 | 13:18 |
| sinzui | ericsnow, I am in #server asking about bug 1434737. hallyn is our lxc expert. | 14:18 |
| mup | Bug #1434737: lxc-start failure on vivid <lxc:New> <https://launchpad.net/bugs/1434737> | 14:18 |
| wwitzel3 | ericsnow, natefinch: the internet here is very poor, I keep dropping from hangout | 14:25 |
| ericsnow | wwitzel3: no worries, we're almost done | 14:25 |
| natefinch | wwitzel3: no worries | 14:25 |
| natefinch | jinx! | 14:25 |
| ericsnow | sinzui: awesome, thanks | 14:43 |
| sinzui | alexisb, I still hope for release note for "Service Leader Elections". | 14:57 |
| ericsnow | sinzui: FYI, I put as much info as I could find on that ticket, but couldn't remember exactly what vivid we're running and if we are upgrading from daily every day | 14:58 |
| alexisb | sinzui, ack | 14:58 |
| sinzui | ericsnow, #server reports that rbasak will be pushing a fix for us | 14:59 |
| ericsnow | sinzui: cool, thanks! | 14:59 |
| sinzui | ericsnow, oh, auto updates on vivid-slave-b were stalled because I didn't let it restart services. I have fixed that | 15:00 |
| sinzui | too | 15:00 |
| ericsnow | sinzui: ah, cool | 15:01 |
| ericsnow | sinzui: that runs daily or in response to some remote update? | 15:01 |
| mup | Bug #1432331 changed: better config interface <config> <juju-core:New> <https://launchpad.net/bugs/1432331> | 15:02 |
| mup | Bug #1434544 changed: backups broke non-linux builds <ci> <osx> <regression> <windows> <juju-core:Fix Released by hduran-8> <https://launchpad.net/bugs/1434544> | 15:02 |
| mup | Bug #1435215 changed: juju metadata validate-tools error <juju-core:Invalid> <https://launchpad.net/bugs/1435215> | 15:02 |
| sinzui | ericsnow, yes, Plus test several tests we run on the machine also do upgrades | 15:02 |
| sinzui | ericsnow, it allows us to catch issues in new series packages...which just makes all our jobs harder | 15:02 |
| sinzui | dimitern, natefinch: can you find someone to read the log for bug 1431286 and decide if we need more information? I cannot triage it | 15:12 |
| mup | Bug #1431286: juju bootstrap fails when http_proxy is set in environments.yaml <ubuntu-openstack> <juju-core:New> <https://launchpad.net/bugs/1431286> | 15:12 |
| ericsnow | sinzui: the root cause in lp:1432683 seems to apply to the bug I opened (lp:1434737) | 15:15 |
| ericsnow | sinzui: so I'm hopeful :) | 15:15 |
| sinzui | \o/ | 15:15 |
| ericsnow | sinzui: I should have thought to look at those logs earlier | 15:15 |
| natefinch | sinzui: looking at the http_proxy bug.... not sure what's going on | 15:19 |
| mup | Bug #1435244 changed: juju cycle dead lock <juju-core:Invalid> <https://launchpad.net/bugs/1435244> | 15:20 |
| sinzui | ericsnow, Do you know how to force apparmor reload on that machine? | 15:23 |
| ericsnow | sinzui: no | 15:23 |
| ericsnow | sinzui: I'm not familliar with apparmor | 15:23 |
| natefinch | sinzui: reboot? | 15:24 |
| * natefinch has been a windows dev for a long time ;) | 15:24 | |
| sinzui | natefinch, I think we have done that several times now | 15:24 |
| natefinch | sinzui: you must mean something different than I was thinking for reloading apparmor | 15:24 |
| sinzui | natefinch, I think new profiles were added to the machine, but were not registered, so maybe I don't mean reload | 15:25 |
| natefinch | sinzui: ahh, ok. I am also not familiar with apparmor, unfortunately | 15:26 |
| frankban | hi all, is SpecializeCharmRepo still used in juju-core? in essence, do we still support "test-mode" and "charm-store-auth" in the configuration? | 15:46 |
| frankban | AFAICT at least the latter does not have any effects | 15:46 |
| voidspace | dimitern: ping | 15:47 |
| dimitern | voidspace, in a call | 15:48 |
| voidspace | dimitern: ok | 15:48 |
| voidspace | natefinch: hey, you familiar much with the dummy provider? | 15:48 |
| voidspace | natefinch: the dummy provider records the operations it does, presumably so you can make asserts that it was used in a particular way | 15:49 |
| voidspace | natefinch: do you know of any examples of this in the code base I can look at? | 15:49 |
| natefinch | voidspace: no, not really. I try to avoid it as much as possible. It is used fairly extensively, however | 15:50 |
| voidspace | natefinch: ok, I'll grep around | 15:50 |
| voidspace | natefinch: I do have an interface (that only requires one method to be implemented), maybe I should use that instead... | 15:50 |
| tasdomas | wwitzel3, I see you're OCR today - would you mind reviewing http://reviews.vapour.ws/r/1070/ and http://reviews.vapour.ws/r/1097/ ? | 15:51 |
| natefinch | frankban: I haven't looked too deeply into it, but I do see code using that config option | 15:54 |
| tasdomas | if anyone else could also take a look at http://reviews.vapour.ws/r/1097/ and http://reviews.vapour.ws/r/1070/, I would very much appreciate that | 15:56 |
| frankban | natefinch: I also see it used, and it adds an Authorization header. I don't see how this affects the server side though (the legacy charm store) | 16:01 |
| dimitern | voidspace, back | 16:01 |
| frankban | natefinch: the legacy charm store does not check auth headers AFAICT, so I wonder if the SetAuthAttrs can be removed | 16:04 |
| voidspace | dimitern: do you know off the top of your head any test files likely to use the dummy provider ops for asserts about operations | 16:04 |
| voidspace | dimitern: I'm currently trying with the interface, but I might still want to look in the operations log for other tests | 16:04 |
| dimitern | voidspace, yeah, just a sec | 16:05 |
| voidspace | dimitern: if you don't know off the top of your head don't worry - I can search myself | 16:05 |
| voidspace | ah, ok | 16:05 |
| voidspace | my ubuntu phone just arrived | 16:05 |
| dimitern | voidspace, so you mean outside of the dummy provider tests, right? | 16:07 |
| voidspace | dimitern: as a whitebox test using the dummy provider | 16:07 |
| voidspace | dimitern: I haven't even looked at the dummy provider tests though... :-) | 16:07 |
| natefinch | frankban: yeah, I don't know the internals of the new charm store stuff | 16:07 |
| voidspace | dimitern: that would be a logical place to start - but I kind of assume they're whitebox | 16:07 |
| voidspace | I mean as a *blackbox* test | 16:08 |
| dimitern | voidspace, well, it's like this: dummy provider tests can use the OpXYZ internally to check calls | 16:08 |
| voidspace | dimitern: the dummy provider tests are good | 16:08 |
| voidspace | dimitern: they use dummy.Listen and provide a channel | 16:09 |
| dimitern | voidspace, outside the dummy package, you could mostly just use the "broken" setting to selectively return errors from environ methods | 16:09 |
| voidspace | I should have looked there | 16:09 |
| dimitern | voidspace, yeah, in fact several of the environs.Networking methods are tested using those dummy features | 16:09 |
| voidspace | dimitern: I have another question if you don't mind | 16:10 |
| dimitern | voidspace, e.g. "break" AllocateAddress(), check what happens | 16:10 |
| dimitern | voidspace, sure | 16:10 |
| voidspace | dimitern: see this code http://pastebin.ubuntu.com/10661843/ | 16:10 |
| voidspace | dimitern: it starts with a "releaser" interface | 16:10 |
| voidspace | dimitern: followed by my attempt at a mock implementation | 16:10 |
| voidspace | the mockReleaser | 16:10 |
| dimitern | voidspace, yeah | 16:11 |
| voidspace | but it doesn't compile, Go doesn't like ReleaseAddress being a pointer receiver | 16:11 |
| voidspace | "mockReleaser does not implement addresser.releaser (ReleaseAddress method has pointer receiver) | 16:11 |
| voidspace | " | 16:11 |
| voidspace | I *need* it to have a pointer receiver | 16:11 |
| voidspace | ah | 16:11 |
| voidspace | NewMockReleaser | 16:11 |
| voidspace | NewMockReleaser should return a pointer | 16:11 |
| voidspace | I wonder if that's it | 16:11 |
| voidspace | dimitern: yep, that was the problem :-) | 16:12 |
| dimitern | voidspace, :) | 16:12 |
| voidspace | sorry for the noise, thanks for being my rubber ducky... | 16:12 |
| dimitern | voidspace, no worries, glad I could help :) | 16:13 |
| frankban | natefinch: well, the new charm store will use a different authentication machinery, but I was pointing out that I don't see the auth header to be used in old one either | 16:15 |
| frankban | natefinch: so basically SetAuthAttrs seems to be ineffective to me | 16:15 |
| natefinch | frankban: ouch, yeah, I just went one step further looking, and nothing implements SetAuthAttrs, so that code will never be used. | 16:19 |
| * dimitern is eod | 16:19 | |
| natefinch | frankban: actually, hang on... nothing *on my disk* implements it.... may well be code in github that I don't have | 16:20 |
| frankban | natefinch: SetAuthAttrs is implemented by the charm store in charm.v4 | 16:20 |
| natefinch | frankban: figured | 16:21 |
| frankban | natefinch: it sets an header to be used when retrieving a charm | 16:21 |
| frankban | natefinch: but I don't see that header to be used in the old charm store | 16:22 |
| frankban | natefinch: so basically nothing in https://github.com/juju/charmstore/blob/master/server.go checks an Auth header | 16:24 |
| frankban | natefinch: I'd plan to remove that SetAuthAttr of the specializer when migrating juju-core to the new charm store, if that's ok | 16:24 |
| sinzui | rogpeppe, are you blocked from merging https://code.launchpad.net/~rogpeppe/godeps/002-multi-dependency-update/+merge/247684 | 16:25 |
| natefinch | frankban: you should bring it up on juju-dev. Sounds fine to me, but I've never touched the charm store code. | 16:25 |
| frankban | natefinch: sure I will, ty | 16:26 |
| natefinch | rogpeppe, cmars: do you guys know if we still need SetAuthAttr for the charm store? Doesn't look to be used currently. See ^^ w/ frankban | 16:27 |
| natefinch | evilnickveitch, rick_h_: why is this page missing in the current docs? http://webcache.googleusercontent.com/search?q=cache:sKeqAxbxmdIJ:https://jujucharms.com/docs/config-general+&cd=5&hl=en&ct=clnk&gl=us | 16:29 |
| rick_h_ | natefinch: checking, we hit an issue where docs using {{ were interpreted by the template engine. I'm guessing it's another case. | 16:29 |
| * rogpeppe is ill | 16:30 | |
| rogpeppe | sinzui: it's currently in an intermediate state - i got a certain way with it and realised i needed to think harder about desired semantics and haven't got back to it yet, i'm afraid | 16:30 |
| rogpeppe | sinzui: thanks for the reminder. i'll try to get around to doing it very soon | 16:31 |
| natefinch | rick_h_: sounds like we need a test that ensures len(input_files) == len(output_files) | 16:32 |
| mup | Bug #1435413 was opened: rename "juju backups restore" to "juju backups recover" <backup-restore> <juju-core:New> <https://launchpad.net/bugs/1435413> | 16:32 |
| rick_h_ | natefinch: yep, we've got a fix in trunk atm, double checking what's up with this file to see if we can tweak it for next build or needs a deployment to fix | 16:32 |
| evilnickveitch | natefinch, rick_h_ I have just noticed that the current version of published docs has reverted to an old navigation bar too, so that page isn't listed | 16:34 |
| rick_h_ | evilnickveitch: natefinch ok, that's the docs defaulting to stabled 1.20, the file isn't in that branch so it's not shown but it's in the latest https://jujucharms.com/docs/latest/config-general | 16:36 |
| rick_h_ | evilnickveitch: natefinch so the next release will have full multi-release browsing support which would allow us to get to that page | 16:37 |
| sinzui | rogpeppe, thank you | 16:37 |
| rick_h_ | evilnickveitch: and we don't have 1.21 and 1.22 in the versions file | 16:38 |
| natefinch | rick_h_: that page doesn't seem (terribly) version specific, maybe we could backport it | 16:38 |
| evilnickveitch | natefinch, rick_h_ Ah, ok. yes, i was planning to backport it - it does have some 1.21 specific stuff in it though. Actually, quite a bit | 16:40 |
| evilnickveitch | we don't have 1.21 in the versions file yet, because I was waiting until we had fixed the issues with the versions we had before adding more | 16:40 |
| rick_h_ | evilnickveitch: k, we're targeting to have the UX for browing versions/etc in the next release in April, but let me know what we need to get in/done to help move things forward until then. | 16:41 |
| evilnickveitch | rick_h_, I'll make a list | 16:42 |
| evilnickveitch | thx | 16:42 |
| rick_h_ | evilnickveitch: ty, I've told the guys today I want to have the docs work wrapped up this week and I've asked the UX folks to put docs UI on their near term radar in our UX sync this AM | 16:42 |
| frankban | natefinch: do you know how to update juju-core deps without removing the platform specific ones? if I use just "godeps -t ..." I get some of them removed (e.g. kardianos/osext, winsvc...) | 16:47 |
| frankban | natefinch: found your solution at https://github.com/juju/juju/pull/980 | 16:54 |
| frankban | perhaps that should be documented in the contributing file | 16:55 |
| wwitzel3 | tasdomas: yep, I can do that | 17:01 |
| cmars | frankban, natefinch rick_h_ pretty sure we don't need the charm-store-auth environ config option anymore | 17:09 |
| frankban | cmars: great | 17:10 |
| frankban | cmars: I'll work on removing the option | 17:11 |
| cmars | frankban, awesome, thanks! | 17:11 |
| frankban | ocr: is anyone available for reviewing https://github.com/juju/juju/pull/1913 ? thank you! | 17:28 |
| jw4 | wallyworld: are we anywhere close to lp 1434680 being marked as fix-released (and unblocking CI? ) | 17:28 |
| frankban | urulama: and here is where we release the hounds: https://github.com/juju/juju/pull/1913 | 17:32 |
| === kadams54 is now known as kadams54-away | ||
| wwitzel3 | tasdomas: you have reviews | 17:36 |
| === kadams54-away is now known as kadams54 | ||
| jcastro | natefinch, around? | 19:15 |
| === kadams54 is now known as kadams54-away | ||
| cmars | hey, if anyone's up for it.. http://reviews.vapour.ws/r/1238/ | 19:19 |
| cmars | fixes a map-ordering bug in state/logs.go | 19:20 |
| natefinch | jcastro: sorta, in a meeting | 19:34 |
| jcastro | natefinch, no worries, wayne helped me | 19:34 |
| davecheney | cmars: not lgtm | 19:37 |
| davecheney | if you're fixing a bug | 19:37 |
| davecheney | fix only the bug | 19:37 |
| davecheney | don't include a bnuch of cleanup which obscures the fix | 19:38 |
| cmars | davecheney, PTAL | 19:46 |
| thumper | morning folks | 19:47 |
| cmars | hey thumper | 19:47 |
| thumper | cmars: you are up at a weird time | 19:47 |
| cmars | lol | 19:47 |
| thumper | no | 19:47 |
| mattyw | thumper, morning | 19:47 |
| thumper | I need to get my TZ brain in | 19:47 |
| * thumper reboots his head | 19:48 | |
| cmars | thumper, how did you know it was afternoon nap time? | 19:48 |
| thumper | heh | 19:48 |
| mattyw | thumper, I was about to land this: http://reviews.vapour.ws/r/1232/ wanted to let you know - as you folks have been doing a lot of env stuff lately | 19:48 |
| * thumper looks | 19:48 | |
| * thumper sighs | 19:49 | |
| thumper | mattyw: the testing uuid is valid, jut not a valid type 4 | 19:49 |
| thumper | which is fine | 19:49 |
| thumper | why do you feel the need to make it a valid type 4 uuid? | 19:49 |
| thumper | wwitzel3: I know why you move around all the time when on calls now | 19:50 |
| thumper | wwitzel3: I'm trialing a standing desk | 19:50 |
| thumper | did the upgrade issue get fixed? | 19:52 |
| alexisb | thumper, arent you on vacation? | 19:58 |
| alexisb | on that is your monday :) | 19:58 |
| thumper | alexisb: it is tuesday here now | 19:58 |
| alexisb | nevermind | 19:58 |
| thumper | :) | 19:58 |
| thumper | heh | 19:58 |
| thumper | although if you like, I can take it off | 19:58 |
| thumper | NZ is in the cricket world cup semi-final | 19:58 |
| thumper | game starts 2pm | 19:58 |
| katco | thumper: wow gl | 19:59 |
| thumper | playing south africa | 19:59 |
| thumper | and we are not favourites | 19:59 |
| alexisb | thumper, I was just excited about getting an hour back :) | 19:59 |
| thumper | so we'll need luck I think | 19:59 |
| thumper | heh | 19:59 |
| thumper | alexisb: sorry | 19:59 |
| * katco knows almost nothing about cricket | 19:59 | |
| natefinch | thumper: does loggo cache/batch writes? It seems like I'm having problems getting log output right before jujud shuts down. | 20:00 |
| thumper | natefinch: no, all log writes are synchronous | 20:00 |
| natefinch | (or I'm just doing something else wrong, which is also quite possible) | 20:00 |
| natefinch | thumper: ok good, I hoped so. | 20:00 |
| thumper | natefinch: although, the writers themselves may be buffering | 20:00 |
| natefinch | thumper: ahh, yeah, file writes are buffered, most likely, either by the software or the OS. | 20:01 |
| thumper | right | 20:01 |
| natefinch | (or both) | 20:01 |
| natefinch | wish we had a mechanism to flush logs before shutdown... seems like an easy way to lose log information (like what appears to be happening to me). | 20:04 |
| katco | natefinch: it seems like a useful concept in general: jujud is going down. anyone need anything? | 20:06 |
| natefinch | katco: yeah... I think we only recently started even thinking about jujud getting stopped | 20:09 |
| katco | natefinch: you could expand that even further and it would fix one of our long-standing issues: jujud takes in commands (i.e. command pattern) for major lifecycle events. it would fix jujud having to import the world | 20:11 |
| davecheney | cmars: lgtm, thanks | 20:19 |
| davecheney | it's now clear what the fix is | 20:19 |
| cmars | davecheney, ty | 20:21 |
| davecheney | cmars: send a follow up PR for those bits i told you to leave out | 20:24 |
| davecheney | i'll review it asap | 20:24 |
| === kadams54-away is now known as kadams54 | ||
| natefinch | oh well, fmt.Printf gets around the logging issue, but I still filed a bug about it, since it very well could bite us in the butt https://bugs.launchpad.net/juju-core/+bug/1435543 | 20:50 |
| mup | Bug #1435543: juju loses log messages written near jujud shutdown <juju-core:New> <https://launchpad.net/bugs/1435543> | 20:51 |
| mup | Bug #1435543 was opened: juju loses log messages written near jujud shutdown <juju-core:New> <https://launchpad.net/bugs/1435543> | 20:54 |
| jw4 | cmars: fyi I had a PR up for that bug on Friday: http://reviews.vapour.ws/r/1222/ , but I didn't do any of the changes you did in the follow up 1240 | 20:54 |
| cmars | jw4, aw man, sorry, didn't catch that | 21:01 |
| jw4 | cmars: no worries - you took it further anyway :) | 21:02 |
| cmars | too far, some might say :) | 21:02 |
| jw4 | haha | 21:02 |
| cmars | so master is broken because landing is blocked because master is broken | 21:07 |
| jw4 | cmars: JFDI ? :) | 21:13 |
| fwereade_ | / AgentConf handles command-line flags shared by all agents. | 21:17 |
| fwereade_ | ...yes, that is one of the things it does | 21:17 |
| fwereade_ | type AgentConf struct { | 21:17 |
| fwereade_ | DataDir string | 21:17 |
| fwereade_ | mu sync.Mutex | 21:17 |
| fwereade_ | _config agent.ConfigSetterWriter | 21:17 |
| fwereade_ | } | 21:17 |
| fwereade_ | ...but I would venture perhaps not its primary purpose | 21:17 |
| === kadams54 is now known as kadams54-away | ||
| niedbalski | ericsnow, ping | 21:53 |
| ericsnow | niedbalski: hi | 21:53 |
| niedbalski | hi ericsnow , could you review http://reviews.vapour.ws/r/1242/ (whenever you have a chance) ? | 21:54 |
| ericsnow | niedbalski: sure | 21:54 |
| niedbalski | ericsnow, thanks | 21:54 |
| ericsnow | niedbalski: interestingly, I'm actually working on a patch to use a config file that may be shared by the different mongo executables | 21:55 |
| ericsnow | niedbalski: but that will only apply to 1.24+ | 21:56 |
| niedbalski | ericsnow, yep, this is targeting 1.22 | 21:56 |
| ericsnow | niedbalski: isn't the --journal option not supported on mongodump without the --dbpath option? | 22:27 |
| niedbalski | ericsnow, the --help says 'is not relevant without specifying' | 22:33 |
| ericsnow | niedbalski: ah, fair enough | 22:33 |
| ericsnow | wallyworld: how hard do you think it will be to support the cloud-images "downloads" metadata in our simplestreams implementation | 22:49 |
| wallyworld | ericsnow: one sec, on a call | 22:50 |
| ericsnow | wallyworld: np | 22:50 |
| === kadams54-away is now known as kadams54 | ||
| jw4 | sinzui: isn't bug 1434680 fixed in master now since https://github.com/juju/juju/pull/1907 merged? | 23:15 |
| mup | Bug #1434680: Certificate generated by certupdater worker cannot be used by MongoDB <ci> <regression> <upgrade-juju> <juju-core:Fix Committed by wallyworld> <juju-core 1.22:Fix Released by menno.smits> <juju-core 1.23:Fix Released by wallyworld> <https://launchpad.net/bugs/1434680> | 23:15 |
| wallyworld | axw_: you around for standup? | 23:19 |
| === kadams54 is now known as kadams54-away | ||
| ericsnow | wallyworld: look like the joyent upgrade test is running for master right now | 23:29 |
| ericsnow | wallyworld: looks like it passed \o/ | 23:31 |
| wallyworld | yay | 23:32 |
| wallyworld | ericsnow: it won't be too hard to support cloud-images downloads in simplestreams - we probably do now in fact, but juju doesn't use information as i recall (i'd need to check) | 23:34 |
| ericsnow | wallyworld: it was hard to tell how much support for the index and products formats that we implemented beyond what we need for juju | 23:35 |
| ericsnow | wallyworld: I'm guessing we make assumptions in that regard | 23:36 |
| wallyworld | ericsnow: yeah, we do, we implemented what we needed, there's likely some stuff we left out, but it's not hard to add | 23:36 |
| ericsnow | wallyworld: at what point would it make sense to factor out our Go simplestreams "bindings" into its own repo? | 23:38 |
| wallyworld | could be done now really | 23:39 |
| ericsnow | (i.e. how much are we missing?) | 23:39 |
| wallyworld | there's a separate simplestreams package that's juju independent | 23:39 |
| wallyworld | and then we build on that for tools and images in separate packages | 23:39 |
| wallyworld | not sure how much we're missing though | 23:40 |
| wallyworld | and it was done with juju in mind so would likely need cleanup | 23:40 |
| === kadams54-away is now known as kadams54 | ||
| ericsnow | right | 23:40 |
| ericsnow | wallyworld: it's basically too big a project for what I need at the moment :( | 23:41 |
| ericsnow | wallyworld: for the vmware provider we are using the "downloads" metadata | 23:41 |
| ericsnow | wallyworld: but currently our simplestreams code doesn't work with that | 23:41 |
| wallyworld | oh i see | 23:41 |
| ericsnow | wallyworld: so I've been looking at the simplestreams "spec" and at our code and trying to make sense of it all | 23:42 |
| wallyworld | there's also python bindings that could be looked at | 23:43 |
| ericsnow | wallyworld: that's where I found the "spec" :) | 23:43 |
| ericsnow | wallyworld: http://bazaar.launchpad.net/~smoser/simplestreams/trunk/files, right? | 23:45 |
| wallyworld | think so, i don't know eanywhere else | 23:45 |
| ericsnow | k | 23:45 |
| ericsnow | BTW, I'm not convinced there is a bug in restore | 23:45 |
| ericsnow | but I need more info to be sure | 23:46 |
| wallyworld | ok | 23:46 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!