| davecheney | what does bzr mean when it returns error code 3 ? | 02:15 |
|---|---|---|
| davecheney | rogpeppe: i think i have a solution to maek the ssh tests much cleaner | 06:02 |
| davecheney | if you want it | 06:02 |
| davecheney | -- but i know you're busy tearing that out | 06:02 |
| rogpeppe | fwereade_: morning! | 06:43 |
| fwereade_ | rogpeppe, heyhey | 06:43 |
| fwereade_ | rogpeppe, how's it going? | 06:43 |
| rogpeppe | fwereade_: sorry about trunk breakage last night | 06:43 |
| rogpeppe | fwereade_: i didn't realise i had an old version of mongodb | 06:43 |
| fwereade_ | rogpeppe, no worries | 06:44 |
| rogpeppe | fwereade_: not bad at all, thanks. | 06:44 |
| rogpeppe | fwereade_: and you? | 06:44 |
| fwereade_ | rogpeppe, yeah, not bad :) | 06:44 |
| TheMue | morning | 07:00 |
| rogpeppe | fwereade_: i'm adding admin-secret to the environment configuration, and i *think* it should be part of config.Config, not specific to the ec2 config. it's an interesting case though, because it's *really* secret - it doesn't get pushed with the other secrets. | 07:05 |
| rogpeppe | fwereade_: so i'm not sure whether to have a "ReallySecretAttrs" method in environs, or just to make a special case for admin-secret | 07:06 |
| * fwereade_ thinks | 07:06 | |
| fwereade_ | morning TheMue btw | 07:06 |
| TheMue | fwereade_: heya | 07:07 |
| fwereade_ | rogpeppe, sorry, no strong feelings either way, just a sense of unease | 07:07 |
| rogpeppe | fwereade_: yeah, special cases make me uncomfortable too | 07:08 |
| rogpeppe | TheMue: hiya | 07:08 |
| TheMue | rogpeppe: a wonderful morning to you too (even if it is raining here) ;) | 07:09 |
| rogpeppe | TheMue: beautifully sunny this morning. (sun just up) | 07:09 |
| TheMue | rogpeppe: we had a nice october so far until yesterday. since then mostly rain, too much rain | 07:10 |
| fwereade_ | rogpeppe, wow, massive filter change works as expected; simplifies uniter noticeably; but pushes the tests up over a minute :/ | 07:43 |
| rogpeppe | fwereade_: cool, but... | 07:44 |
| rogpeppe | fwereade_: what's taking all the time in the tests, out of interest? | 07:44 |
| fwereade_ | rogpeppe, thus far I know not, I will do some poking around | 07:44 |
| rogpeppe | fwereade_: worth doing, i think. | 07:44 |
| rogpeppe | fwereade_: go test -gocheck.vv | timestamp is always a good start | 07:45 |
| fwereade_ | rogpeppe, some of it is probably attributable to just doing more work more often but iyt's a bit of a big change for that I think | 07:45 |
| rogpeppe | fwereade_: a small CL: https://codereview.appspot.com/6589072/ | 07:55 |
| rogpeppe | fwereade_: for context, here's the authentication sketch: http://paste.ubuntu.com/1259533/ | 07:56 |
| rogpeppe | davecheney: it would be good to have some feedback from you about the authentication scheme too, if poss | 07:56 |
| davecheney | rogpeppe: sure thing | 07:56 |
| davecheney | rogpeppe: did you see my comment about improving the ssh tests ? | 07:57 |
| rogpeppe | davecheney: i did | 07:57 |
| davecheney | althought I realise the horse has bolted | 07:57 |
| rogpeppe | davecheney: yeah, no point in flogging that one | 07:57 |
| davecheney | rogpeppe: right-o | 07:57 |
| fwereade_ | rogpeppe, so --initial-password will be ignored if a password file exists | 07:57 |
| rogpeppe | davecheney: i'm interested to know what your plan was though | 07:57 |
| rogpeppe | fwereade_: yes | 07:57 |
| davecheney | rogpeppe: http://codereview.appspot.com/6601043/diff/3011/ssh/sshtest/sshtest_unix_test.go | 07:57 |
| davecheney | ~ line 150 | 07:57 |
| rogpeppe | fwereade_: well actually, it might not be | 07:57 |
| davecheney | rogpeppe: which CL is the authn Cl ? | 07:58 |
| rogpeppe | fwereade_: if we fail to connect with the password, we'll try initial-password | 07:58 |
| davecheney | the paste ? | 07:58 |
| rogpeppe | davecheney: yeah | 07:58 |
| davecheney | kk | 07:58 |
| rogpeppe | fwereade_: because we might have written the password file but failed to change it | 07:58 |
| fwereade_ | rogpeppe, ah, sensible | 07:59 |
| davecheney | rogpeppe: i'm not sure how helpful this is | 07:59 |
| davecheney | but the primary customer inside canonical is Elmo | 08:00 |
| davecheney | so if he doesn't like the smell of this | 08:00 |
| davecheney | irrespective of its other merits | 08:00 |
| davecheney | it's game over | 08:00 |
| davecheney | not saying he won't like it, or that what you have is not correct | 08:00 |
| fwereade_ | rogpeppe, CL LGTM, I will try to get myself into a suitably adversarial mode before tackling the auth overview | 08:00 |
| rogpeppe | davecheney: i'm not sure who Elmo is | 08:00 |
| davecheney | me neither | 08:01 |
| davecheney | but I hear he's the big cheese of the internal sysadmin team | 08:01 |
| rogpeppe | davecheney: ok, i'll make a write-up and put it on juju-dev | 08:02 |
| davecheney | rogpeppe: that sounds like an excellent plan, then others can distribute as necessary | 08:03 |
| davecheney | rogpeppe: my only comment of note, is the storing of the key on disk per machine agent | 08:04 |
| davecheney | i don't have a solution to this | 08:04 |
| rogpeppe | davecheney: indeed. we need to store it somewhere | 08:04 |
| davecheney | only observe that others will see it as a potential loophole | 08:04 |
| rogpeppe | davecheney: we must be able to connect after reboot | 08:04 |
| davecheney | yeah, | 08:04 |
| davecheney | is there a concept of differeing levels of privilige ? | 08:04 |
| rogpeppe | davecheney: ssh has exactly the same issue | 08:04 |
| davecheney | rogpeppe: yup, it sure does | 08:05 |
| rogpeppe | davecheney: in fact any autonomous agent must have the same issue | 08:05 |
| rogpeppe | davecheney: the only solution is authenticated h/w | 08:05 |
| rogpeppe | davecheney: which we don't have. | 08:05 |
| davecheney | rogpeppe: and i'm not sure if that would actually solve the problem | 08:06 |
| davecheney | the issue, as i understand it is | 08:06 |
| davecheney | user X on machine Y can get root, then get whatever details they need to connect to the state, rip off the AWS keys .. | 08:06 |
| davecheney | is that correct ? | 08:06 |
| rogpeppe | davecheney: if there was a way of propagating a secret from bootstrap stage to the agent itself, then we could use the secret, then destroy it. then even if you were root, you couldn't get it. | 08:07 |
| rogpeppe | davecheney: but of course the agent needs to keep the secret around, so even then we're vulnerable | 08:08 |
| davecheney | rogpeppe: the secrets are in the /e document in the /e collection, right ? | 08:08 |
| rogpeppe | davecheney: it's a pity everyone has root access | 08:08 |
| rogpeppe | davecheney: no | 08:08 |
| rogpeppe | davecheney: these secrets are not | 08:08 |
| davecheney | rogpeppe: but the AWS creds we're trying to protect are | 08:08 |
| rogpeppe | davecheney: yes, they are currently | 08:09 |
| rogpeppe | davecheney: but they won't be when we use this scheme to leverage principal-specific access controls | 08:09 |
| davecheney | so, if by some mech, the /e document could be protected from access by the machine agent, would that be a solution ? | 08:09 |
| rogpeppe | davecheney: it's not a solution to malicious entities on the machine being able to impersonate the machine agent | 08:10 |
| rogpeppe | davecheney: but that is necessary too, yes | 08:10 |
| davecheney | rogpeppe: is their a spec for the security model ? | 08:10 |
| rogpeppe | davecheney: no | 08:10 |
| rogpeppe | davecheney: sigh | 08:11 |
| davecheney | rogpeppe: i'm not sure how to proceed without this | 08:11 |
| davecheney | at best you'll implement whatever is inside gustavos head | 08:11 |
| davecheney | and worse, you won't | 08:11 |
| davecheney | and niether case may be what customres want | 08:11 |
| rogpeppe | davecheney: i see what we're doing here as a necessary prelude to implementing the final security model, which is unspecified | 08:12 |
| rogpeppe | davecheney: we're adding the notion of a principal to the state info, which i think is always going to be necessary. | 08:12 |
| davecheney | rogpeppe: i'm concerned it is equiv to starting to walk in an unspecified direction, without picking a destination | 08:12 |
| davecheney | rogpeppe: please understand, i'm not have a go at your solution, | 08:13 |
| rogpeppe | davecheney: ok, the basic security model, as i understand it is: | 08:13 |
| davecheney | you know my pickyness for implementing security without a spec | 08:13 |
| rogpeppe | davecheney: agents identify themselves to the state; the state allows agents to agent-specific things. | 08:14 |
| rogpeppe | s/to agent/to do agent/ | 08:14 |
| davecheney | rogpeppe: but there is an unspoken requirement that agents cannot be impersonanted | 08:14 |
| rogpeppe | davecheney: yes... well, kinda. | 08:14 |
| rogpeppe | davecheney: damage limitation | 08:15 |
| rogpeppe | davecheney: we don't want any random non-root user on a machine to be able to impersonate that machine's machine agent. | 08:15 |
| rogpeppe | davecheney: but if you're root, you're going to be able to do what you damn please | 08:15 |
| davecheney | rogpeppe: then putting the per machine agent password in a 0600 file would work | 08:15 |
| davecheney | but i think further consultation with the customer is needed | 08:16 |
| rogpeppe | davecheney: yes, that's what we're doing | 08:16 |
| davecheney | i'm pretty sure that someone is going to say 'but what if they get root' | 08:16 |
| rogpeppe | davecheney: there's nothing we can do in that case. | 08:16 |
| davecheney | yup | 08:17 |
| rogpeppe | davecheney: the trickiness in the spec i pasted above is because there's no way of passing a secret to the initial machine agent that's not accessible by non-root users. | 08:17 |
| rogpeppe | davecheney: hence --initial-password | 08:17 |
| davecheney | but that is probably going to meen elmo rejects the idea, and we've done a lot of work for nohting | 08:17 |
| rogpeppe | davecheney: he can't ask the impossible | 08:17 |
| davecheney | he's a very powerful customer | 08:18 |
| rogpeppe | davecheney: this is significantly better than what we had before - entities get permissions on a need-to-have basis. | 08:18 |
| rogpeppe | davecheney: sure, but we're talking *impossible* here | 08:18 |
| rogpeppe | davecheney: noone else will be able to do better | 08:19 |
| davecheney | rogpeppe: i never said the customer was rational :) | 08:19 |
| rogpeppe | davecheney: also, you won't be able to do much by impersonating a machine agent, even if you are root. | 08:20 |
| rogpeppe | and malicious | 08:20 |
| rogpeppe | davecheney: in fact the machine agent doesn't need to be able to write to the state at all | 08:21 |
| rogpeppe | davecheney: you can do a little more by impersonating a unit agent, but again, not too bad, i think. | 08:22 |
| rogpeppe | davecheney: this all assumes an entity-aware API of course. | 08:23 |
| rogpeppe | davecheney: the thing i'm most concerned with is man-in-the-middle attacks. i don't see how we can protect against those unless we have some kind of key-distribution scheme. | 08:24 |
| davecheney | rogpeppe: them why does the MA need a password at all ?> | 08:26 |
| davecheney | i we treat the machine and the machine agent as untrusted | 08:26 |
| davecheney | you can ignore their credentials | 08:26 |
| rogpeppe | davecheney: we want to partition machines | 08:26 |
| davecheney | rogpeppe: can we trust the LXC security boundary ? | 08:27 |
| rogpeppe | davecheney: and i don't think we can entirely assume that non-root users can always obtain root. | 08:27 |
| rogpeppe | davecheney: apparently not. for root users within LXC anyway. | 08:27 |
| davecheney | rogpeppe: if that is a working assumption, then the job is a lot easier | 08:27 |
| rogpeppe | davecheney: i think our advice should be (as per usual) don't run untrusted stuff as root. | 08:29 |
| davecheney | seconded | 08:29 |
| rogpeppe | davecheney: this means we've essentially got a two-tier security model. primary layer: TLS-based authentication; secondary layer: entity name/password authentication | 08:30 |
| rogpeppe | davecheney: we assume that a malicious root user can bypass the secondary layer, but not the primary layer. | 08:31 |
| davecheney | rogpeppe: is the tls layer using client side certs ? | 08:32 |
| rogpeppe | davecheney: it'd better! | 08:32 |
| davecheney | otherwise it isn't a security authn mech :) | 08:32 |
| rogpeppe | davecheney: indeed | 08:32 |
| rogpeppe | davecheney: and vulnerable to man-in-the-middle too | 08:32 |
| rogpeppe | davecheney: we need a way of securely passing a cert to a new machine | 08:33 |
| rogpeppe | davecheney: ISTR that amz supports this | 08:33 |
| rogpeppe | davecheney: dunno about others | 08:33 |
| davecheney | rogpeppe: in the puppet model the client generates the cert and sends it to the server for signing | 08:33 |
| rogpeppe | davecheney: how does the server know where the cert is coming from? | 08:34 |
| davecheney | the admin is expected to manage that out of band | 08:34 |
| davecheney | just like gpg | 08:34 |
| rogpeppe | davecheney: we can't do that - it's all autonomous | 08:34 |
| davecheney | ie, if you weren't expecting to see a cert request, then don't sign it | 08:35 |
| davecheney | of course, most envs turn on automatic cert signing | 08:35 |
| rogpeppe | davecheney: of course. | 08:35 |
| rogpeppe | davecheney: i don't even see how an admin can know | 08:35 |
| davecheney | security is hard, shall we go shopping ? | 08:35 |
| rogpeppe | davecheney: ooh shiny | 08:35 |
| davecheney | rogpeppe: generally you install machine x, insgtall puppet, then tell it to join your puppet server | 08:36 |
| davecheney | go to the server, accept the request | 08:36 |
| davecheney | then profit | 08:36 |
| rogpeppe | davecheney: what if someone got in there between the install and going to the server? | 08:36 |
| rogpeppe | davecheney: of course, it may be improbable, but... | 08:36 |
| davecheney | puppet assumes you control the security of your environment | 08:37 |
| davecheney | it's a cfg management tool | 08:37 |
| davecheney | and there is a reason juju exists :) | 08:37 |
| davecheney | rogpeppe: my friends that work in hosting companies | 08:37 |
| davecheney | run one puppet server per customer | 08:37 |
| rogpeppe | davecheney: the way i did this when we had a similar thing was i installed manually on a machine, including a private key. | 08:37 |
| davecheney | the idea of a single puppet instnace for all customers is impractical | 08:37 |
| rogpeppe | davecheney: then when the machine dials in, you *know* that it's the right machine. | 08:38 |
| rogpeppe | davecheney: but that assumes a manual install, of course. | 08:38 |
| davecheney | rogpeppe: yup | 08:38 |
| rogpeppe | davecheney: which is why you have to assume that the cloud provider can do something similar | 08:38 |
| davecheney | my friends in hosting companies use vlans and shit | 08:38 |
| davecheney | to separate customers environments | 08:38 |
| rogpeppe | davecheney: worst comes to worst you're vulnerable to mitm, but if the hosting co is compromised, you're fucked anyway | 08:39 |
| rogpeppe | davecheney: "For Linux instances, you can provide an optional key pair ID in the launch request (created using the CreateKeyPair or ImportKeyPair operation). The instances will have access to the public key at boot. You can use this key to provide secure access to an instance of an image on a per-instance basis. Amazon EC2 public images use this feature to provide secure access without passwords." | 08:42 |
| rogpeppe | davecheney: i *think* we can leverage that | 08:43 |
| davecheney | rogpeppe: ooooooooooh | 08:46 |
| davecheney | but, are we likely to end up in the same 'too many firewall groups' quagmire ? | 08:46 |
| rogpeppe | davecheney: i don't think there should be a problem creating a keypair per machine, but i may well be wrong :-) | 08:47 |
| davecheney | another stupid amazon limitation | 08:47 |
| davecheney | and you'll probably get asked how to do it in the openstack/azure/hp world | 08:47 |
| rogpeppe | davecheney: the main problem is that this mechanism is designed for allowing you to connect to a new machine securely, not the other way around. | 08:50 |
| davecheney | mmm | 08:51 |
| rogpeppe | davecheney: i think we'd need to create a cert based on a hash of the key pair's public key or something. | 08:51 |
| rogpeppe | davechen1y: actually, i think we could probably go quite a long way by moving the environ config into a separate database. | 08:58 |
| rogpeppe | davechen1y: then we could at least restrict access to the AWS keys without needing an API | 08:59 |
| davechen1y | rogpeppe: yes | 08:59 |
| davechen1y | which sounds like the juju-as-a-service plan | 08:59 |
| davechen1y | customers have MA's, we run the PA | 08:59 |
| rogpeppe | davechen1y: yeah, that's certainly part of it | 08:59 |
| rogpeppe | davechen1y: by separate database, i didn't mean a separate mongo server, as it happens | 08:59 |
| rogpeppe | davechen1y: i meant a separate mgo.Database | 09:00 |
| rogpeppe | davechen1y: (we already use two) | 09:00 |
| davechen1y | rogpeppe: yup | 09:00 |
| davechen1y | i thought we did that already | 09:00 |
| davechen1y | or is that just different connections, same db ? | 09:01 |
| rogpeppe | davechen1y: same connection, different dbs | 09:01 |
| rogpeppe | davechen1y: but each db has its own set of users | 09:01 |
| rogpeppe | davechen1y: did you see this CL BTW? https://codereview.appspot.com/6587060 | 09:02 |
| * davechen1y looks | 09:05 | |
| davechen1y | rogpeppe: fwereade_ trivial: http://codereview.appspot.com/6601056/ | 09:09 |
| davechen1y | looking for a LGTM | 09:10 |
| * fwereade_ looks | 09:10 | |
| rogpeppe | davechen1y: LGTM, although i suppose the question is: why --format and not something else? | 09:11 |
| davechen1y | rogpeppe: this is the thing that broke | 09:12 |
| davechen1y | and it's intended to get people to update their gnuflag instance | 09:12 |
| rogpeppe | davechen1y: fair enough. | 09:12 |
| rogpeppe | davechen1y: it's gotta be somewhere; why not there? | 09:12 |
| rogpeppe | davechen1y: (rhetorical question) | 09:13 |
| fwereade_ | davechen1y, LGTM | 09:16 |
| davechen1y | thanks folks | 09:17 |
| * fwereade_ has made the uniter tests fast again by fixing a huge obvious repeated 500ms sleep | 09:18 | |
| * fwereade_ knows that has been there for ages | 09:18 | |
| * davechen1y applauds | 09:18 | |
| * rogpeppe applauds loudly | 09:19 | |
| rogpeppe | fwereade_: duration of uniter tests now? | 09:19 |
| fwereade_ | rogpeppe, back to ~45s | 09:20 |
| rogpeppe | fwereade_: hmm, still slow then | 09:20 |
| fwereade_ | rogpeppe, well, yeah, the bit I fixed was pre-existing, so there's probably another 20s to be extracted somewhere, but it's really not obvious | 09:21 |
| rogpeppe | fwereade_: 20s shorter would be much more reasonable... but i know the feeling. | 09:21 |
| rogpeppe | davechen1y: here's a draft of the first part of a heads-up email: http://paste.ubuntu.com/1259631/ | 09:22 |
| * davechen1y reads | 09:23 | |
| davechen1y | i don't see why we need both certs and usernames/passwords | 09:24 |
| davechen1y | certs can already be associated with a principal | 09:25 |
| rogpeppe | davechen1y: in mongodb? | 09:25 |
| rogpeppe | davechen1y: (i tend to agree - i feel that passwords are a bit retro) | 09:25 |
| fwereade_ | rogpeppe, davechen1y: I need to pop out for a while, but I have this: https://codereview.appspot.com/6588053 | 09:28 |
| fwereade_ | rogpeppe, davechen1y: still WIP | 09:28 |
| davechen1y | rogpeppe: not sure how mongo does it | 09:29 |
| fwereade_ | rogpeppe, davechen1y: but it incorporates some significant changes after niemeyer's suggestions in various places | 09:29 |
| rogpeppe | fwereade_: will have another look | 09:29 |
| davechen1y | but if you have a TLS cert, then you can request client authentication (ie, they need a sub cert signed by the same CA that issued your cert) | 09:29 |
| fwereade_ | rogpeppe, davechen1y: and if you have time to cast a quick eye over it for general sanity that would be great | 09:29 |
| davechen1y | or the TLS handshake fails | 09:29 |
| fwereade_ | rogpeppe, issues like "all the filter tests are broken" are not what I'm looking for ;p | 09:29 |
| fwereade_ | bbiab | 09:29 |
| rogpeppe | davechen1y: i'm wondering how/whether mongo converts client certs into mongo users | 09:30 |
| davechen1y | sounds complicate | 09:30 |
| davechen1y | compilcated | 09:30 |
| davechen1y | i reckon drop it | 09:30 |
| davechen1y | just use TLS for a secure channel to transmit creds over | 09:31 |
| rogpeppe | davechen1y: so don't connect direct to mongo? | 09:31 |
| rogpeppe | davechen1y: but use a forwarder? | 09:31 |
| rogpeppe | davechen1y: i don't know if that's easy | 09:32 |
| rogpeppe | (although at least the mongo client is written in Go) | 09:32 |
| rogpeppe | oh why, oh why are certificates so horribly useless in this world? | 09:36 |
| davechen1y | and hard, dont' forget that | 09:37 |
| rogpeppe | davechen1y: indeed | 09:37 |
| rogpeppe | davechen1y: so unnecessary | 09:37 |
| rogpeppe | davechen1y: it looks to me as if mongodb can't do client-certificate verification | 09:38 |
| davechen1y | scratch that | 09:39 |
| Aram | yo. | 10:26 |
| rogpeppe | Aram: hiya | 11:01 |
| davecheney | rogpeppe: fwereade_ : comments ? https://codereview.appspot.com/6591080 | 11:09 |
| rogpeppe | davecheney: looks reasonable to me, but i'm not really familiar with the issue, i'm afraid | 11:15 |
| rogpeppe | davecheney: "I want to ask that it be accepted." - you can always just *ask*, y'know :-) | 11:15 |
| Aram | rogpeppe: what mongodb version do you use? | 11:17 |
| rogpeppe | Aram: i've just started using a different version | 11:17 |
| rogpeppe | Aram: i was using... 2.0.3 i think | 11:17 |
| Aram | aha. | 11:17 |
| rogpeppe | Aram: now i've downloaded the version we use on ec2 and am using that | 11:17 |
| Aram | I'm kind of nervous that behavior changes so often between versions so close to eachother. | 11:18 |
| rogpeppe | Aram: me too. it's a pretty shitty thing to get wrong. | 11:18 |
| davecheney | slow cloud-init, is slow | 11:18 |
| davecheney | Aram: rogpeppe: i think we should switch to using the version from the public bucket, exclusively | 11:19 |
| rogpeppe | davecheney: and download it each time we run a test? | 11:19 |
| davecheney | no, i'm sure there is a way to avoid that expense | 11:21 |
| rogpeppe | davecheney: it's important that we be able to run tests on an aeroplane too. | 11:21 |
| davecheney | rogpeppe: i think you're reading too much into my suggestion | 11:22 |
| rogpeppe | davecheney: i suppose we could check the mongodb binaries into the repo | 11:22 |
| davecheney | i'm just thinking of unpacking the version into somewhere inside the juju-tree | 11:22 |
| davecheney | then just call that path directly | 11:22 |
| davecheney | if it's there 'win' | 11:22 |
| davecheney | if not, fail | 11:22 |
| * davecheney wishes we have juju destroy-service | 11:23 | |
| rogpeppe | davecheney: is there no such command in the original juju? | 11:23 |
| davecheney | yeah, but we don't have it in cmd/juju yet | 11:23 |
| davecheney | makes testing harder :) | 11:23 |
| rogpeppe | davecheney: if we don't check it into the repo (and i'm not sure we want to clutter the repo with an 8MB binary) i'm not sure that running from a different path buys us that much | 11:25 |
| davecheney | rogpeppe: we don't have to check it in | 11:25 |
| rogpeppe | davecheney: we'd be better off running mongod --version to check that we get the expected version | 11:26 |
| rogpeppe | davecheney: if we don't check it in, then we have the same problem of possible version skew | 11:26 |
| davecheney | just change the mongo tests to call to a specific path, one where we have already downloaded the mongodb version, rather than just calling any mongod in th epath | 11:26 |
| rogpeppe | ha, it looks like niemeyer didn't build mongod with ssl support | 11:28 |
| Aram | I thought that building with SSL was the reason why we needed to build it. | 11:29 |
| davecheney | rogpeppe: crap -- that was the _ENTIRE_ reason for cmd/builddb | 11:29 |
| rogpeppe | Aram: me too | 11:29 |
| rogpeppe | davecheney: try mongod --help 2>&1 | grep -i ssl | 11:30 |
| davecheney | % juju deploy couchbase &error: cannot assign unit "cf-mongodb/0" to machine: cannot assign unit "cf-mongodb/0" to machine 8: duplicate key insert for unique index of capped collection | 11:31 |
| rogpeppe | davecheney: hmm, it *looks* as if builddb builds it with ssl | 11:31 |
| rogpeppe | i wish we prefixed our errors with "juju: " rather than "error: ". i've been meaning to fix that for ages. | 11:32 |
| davecheney | here is some good news --> http://paste.ubuntu.com/1259806/ | 11:40 |
| rogpeppe | davecheney: woo! | 11:40 |
| rogpeppe | fwereade_: check it out! | 11:41 |
| davecheney | and most of them worked ! | 11:41 |
| rogpeppe | davecheney: that's almost like a real installation :-) | 11:41 |
| davecheney | i don't think I can start any more machines | 11:42 |
| davecheney | amazon will chide me | 11:42 |
| rogpeppe | davecheney: the security description gets longer (still haven't got to the bit that was the whole point yet though!) http://paste.ubuntu.com/1259811/ | 11:42 |
| davecheney | the buildbot charm failures, somethign happened with apt, it wasn't us | 11:43 |
| TheMue | davecheney: nice environment ;) | 11:43 |
| davecheney | % juju ssh ceph/0 -- -t 'less /var/log/juju/unit*' | 11:43 |
| davecheney | nice | 11:43 |
| rogpeppe | davecheney: what does -t do? | 11:43 |
| davecheney | tells ssh'd to allocate a pty | 11:44 |
| davecheney | normally if you do ssh host /some/command | 11:44 |
| davecheney | no pty is allocated | 11:44 |
| rogpeppe | davecheney: ah of course | 11:44 |
| davecheney | which sucks | 11:44 |
| davecheney | ssh cmd does a lot of things for you | 11:44 |
| rogpeppe | davecheney: no, that's a good thing :-) | 11:44 |
| davecheney | yeah, but then you have to figure out why it is how it is | 11:45 |
| rogpeppe | davecheney: for most commands a pty gets in the way | 11:45 |
| davecheney | true | 11:45 |
| davecheney | so, the ceph charm is flat broke, not our fault | 11:45 |
| rogpeppe | davecheney: paste the log? | 11:45 |
| davecheney | two secs | 11:46 |
| davecheney | most of them are missing debs | 11:46 |
| davecheney | rogpeppe: http://paste.ubuntu.com/1259818/ << ceph | 11:47 |
| rogpeppe | davecheney: i wonder what radosgw is and if it was installed by default before. | 11:48 |
| davecheney | 2012/10/04 11:35:31 JUJU HOOK ERROR: command: cluster-init: 10.190.42.228:8091, [Errno 111] Connection refused | 11:48 |
| davecheney | 2012/10/04 11:35:32 JUJU HOOK + /opt/couchbase/bin/couchbase-cli bucket-create -c 10.190.42.228:8091 -u Administrator -p administrator --bucket=jienaigo --bucket-type=couchbase --bucket-password= --bucket-ramsize=1607 --bucket-replica=1 | 11:48 |
| davecheney | 2012/10/04 11:35:32 JUJU HOOK ERROR: command: bucket-create: 10.190.42.228:8091, [Errno 111] Connection refused | 11:48 |
| davecheney | ^ couchbase | 11:48 |
| davecheney | i wonder if some charms install the py juju tools by accident ... | 11:49 |
| davecheney | rogpeppe: re your email | 11:51 |
| davecheney | drop the bit about a machine cert | 11:51 |
| davecheney | i thought we weren't/couldn't do that | 11:51 |
| rogpeppe | davecheney: i think we have to do that, somehow. | 11:52 |
| rogpeppe | davecheney: although we can't currently. | 11:52 |
| davecheney | rogpeppe: yup, good point | 11:52 |
| rogpeppe | davecheney: one way of doing it is to have a server that exchanges one-time tokens for certificate signing. | 11:54 |
| rogpeppe | davecheney: then we can pass a one-time token into cloudinit | 11:54 |
| rogpeppe | davecheney: the machine agent leverages that to get its own certificate signed. | 11:55 |
| davecheney | the couchdb charm requires a ppa which is broken | 11:56 |
| rogpeppe | davecheney: i wonder what your duplicate key insert problem was about | 11:56 |
| davecheney | mongo bug | 11:56 |
| davecheney | happens a lot | 11:56 |
| davecheney | niemeyer has raised a bug upstream | 11:56 |
| davecheney | couchdb | 11:57 |
| davecheney | 2012/10/04 11:37:39 JUJU HOOK * Starting database server couchdb | 11:57 |
| davecheney | 2012/10/04 11:37:39 JUJU HOOK ...done. | 11:57 |
| davecheney | 2012/10/04 11:37:39 JUJU hook failed: exit status 1 | 11:57 |
| davecheney | 2012/10/04 11:37:39 JUJU reading uniter state from disk... | 11:57 |
| davecheney | nice | 11:57 |
| Aram | yeah, mongodb is... less than stellar. | 11:58 |
| rogpeppe | Aram: it's better than zk though | 11:58 |
| davecheney | 2012/10/04 11:37:32 JUJU HOOK + sed -e 's/^STARTDISTCC=.*/STARTDISTCC="true"/' -i /etc/default/distcc | 11:59 |
| Aram | the products are so different they can't be compared like that. | 11:59 |
| davecheney | 2012/10/04 11:37:32 JUJU HOOK + '[' -x /usr/bin/open-port ']' | 11:59 |
| davecheney | 2012/10/04 11:37:32 JUJU hook failed: exit status 1 | 11:59 |
| Aram | they solve a diferent problem. | 11:59 |
| davecheney | ^ hard coded tools | 11:59 |
| rogpeppe | Aram: you're probably right. | 11:59 |
| rogpeppe | Aram: from my brief look at the mongo source earlier today, i wasn't enormously overjoyed. | 12:00 |
| Aram | oh, it's bad, I had to look through it when solving various quirks and bugs. | 12:00 |
| davecheney | 2012/10/04 11:39:15 JUJU HOOK ldconfig deferred processing now taking place | 12:01 |
| davecheney | 2012/10/04 11:39:17 JUJU HOOK install: cannot stat `files/php/php_conf.d_apc.ini': No such file or directory | 12:01 |
| davecheney | 2012/10/04 11:39:17 JUJU hook failed: exit status 1 | 12:01 |
| davecheney | 2012/10/04 11:39:17 JUJU reading uniter state from disk... | 12:01 |
| davecheney | ^ drupal6 expects a file not owned by a deb that it installed | 12:01 |
| rogpeppe | Aram: i'd forgotten that people still like ifdefs. ugh. | 12:01 |
| davecheney | so, short summary, 19 charms, 9 working | 12:02 |
| davecheney | none are our fault (directly) | 12:02 |
| Aram | open source stuff is plagued by ifdefs. | 12:02 |
| davecheney | someone please pass on to mramm and gustavo | 12:03 |
| davecheney | i | 12:03 |
| davecheney | i'm off to bed | 12:03 |
| rogpeppe | davecheney: good work, man | 12:04 |
| rogpeppe | davecheney: enjoy your rest | 12:04 |
| davecheney | no, congratulations to all of you | 12:04 |
| davecheney | with the exceptoin of juju-log -l | 12:04 |
| rogpeppe | davecheney: what was the issue with that? | 12:04 |
| davecheney | i haven't found a charm that is broken because we are incompatible with py juju | 12:04 |
| davecheney | rogpeppe: we didn't support -l $LEVEL | 12:04 |
| rogpeppe | davecheney: ah | 12:05 |
| rogpeppe | davecheney: well, gnuflag was broken too... | 12:05 |
| davecheney | rogpeppe: https://codereview.appspot.com/6584069 | 12:05 |
| davecheney | % juju destroy-environment | 12:06 |
| davecheney | error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. (SignatureDoesNotMatch) | 12:06 |
| davecheney | FUCK YOU AWS | 12:06 |
| davecheney | not tonight | 12:06 |
| davecheney | not with 20 machines running | 12:06 |
| davecheney | it always does this when I have more than a few machines running | 12:06 |
| rogpeppe | davecheney: hmm, that might be our bug, i suppose | 12:07 |
| rogpeppe | davecheney: just got to the aws console :-) | 12:08 |
| rogpeppe | s/got/go/ | 12:08 |
| davecheney | right, i have ceased hemorhaging money | 12:09 |
| rogpeppe | davecheney: i hope you claim it back on expenses | 12:09 |
| rogpeppe | davecheney: (i've been a bit crap at doing that recently) | 12:09 |
| davecheney | night all | 12:09 |
| rogpeppe | davecheney: but i had $200 bill last month, so it's worth doing | 12:09 |
| bigjools | fwereade_: howdy, can I bug you for a bit please? got a vexing problem with juju | 12:33 |
| fwereade_ | bigjools, heyhey | 12:45 |
| bigjools | how's it going? | 12:45 |
| fwereade_ | bigjools, ah, not bad thanks, not sure if I can *actually* remember anything about python but I'm game for a try | 12:45 |
| bigjools | ah you're a Goer | 12:45 |
| fwereade_ | bigjools, that's what they tell me | 12:46 |
| bigjools | :) | 12:46 |
| * fwereade_ maintains a perfectly straight face | 12:46 | |
| bigjools | I'm trying use juju to test deployment on maas and it's saying it did it, yet there's no attempt at all to start a machine | 12:46 |
| bigjools | I can't see start_machine being called, not sure how to debug this and even why it started misbehaving | 12:46 |
| fwereade_ | bigjools, hmmmm -- you're bootstrapping? | 12:47 |
| bigjools | after that | 12:47 |
| bigjools | bootstrap is ok | 12:47 |
| bigjools | just deploy going wrong | 12:47 |
| fwereade_ | bigjools, what does status tell you? is it that there "should be" a machine, but it's just not started? or that the machine never gets added to state in the first place? | 12:49 |
| rogpeppe | bigjools: have you looked at the logs on the bootstrap machine? | 12:49 |
| rogpeppe | bigjools: (something to do after fwereade_'s suggestion, possibly) | 12:49 |
| fwereade_ | bigjools, yeah, the start_machines will be called by the provisioning agent | 12:49 |
| bigjools | fwereade_: I see no attempt at all to even try to start a machine, but status says it's waiting for it | 12:49 |
| fwereade_ | bigjools, assuming they're making it into state, which you can check with status, your best bet is the PA logs | 12:50 |
| bigjools | rogpeppe: I didn't look there, will do so! | 12:50 |
| bigjools | fwereade_: PA? | 12:50 |
| fwereade_ | bigjools, provisioning agent | 12:50 |
| rogpeppe | bigjools: /var/log/juju/*.log i think | 12:51 |
| fwereade_ | bigjools, upstart job called somehign beginning with juju-pro IIRC | 12:51 |
| bigjools | ok | 12:51 |
| rogpeppe | fwereade_: here's the text of an email that i'm considering sending to juju-dev: http://paste.ubuntu.com/1259902/ | 12:51 |
| * fwereade_ reads | 12:51 | |
| bigjools | well, there's a traceback | 12:52 |
| rogpeppe | bigjools: that sounds fairly indicative :-) | 12:55 |
| bigjools | http://pastebin.ubuntu.com/1259905/ | 12:55 |
| rogpeppe | niemeyer: yo! | 12:57 |
| niemeyer | rogpeppe: Heya! | 12:57 |
| niemeyer | Hello all! | 12:57 |
| fwereade_ | bigjools, sorry, never seen that before | 12:57 |
| bigjools | fwereade_: darn :( | 12:58 |
| bigjools | fwereade_: happens on packaged or trunk version, I suspect some config problem but I can't work out what exactly from that traceback, it's not very helpful :( | 12:58 |
| fwereade_ | bigjools, indeed not :( | 12:58 |
| rogpeppe | niemeyer: i was getting feedback from dfc this morning about the security stuff, and he suggested sensibly that we should make sure that our potential users (e.g. Elmo) are happy with the direction we're going, so i put an email together trying to explain things. i don't know whether it's actually worth sending, but it focused my mind helpfully anyway. http://paste.ubuntu.com/1259902/ | 12:59 |
| fss | niemeyer: i sent the cl yesterday https://codereview.appspot.com/6586073/ :) | 13:00 |
| bigjools | fwereade_: I think I know what it is .... :/ | 13:00 |
| niemeyer | rogpeppe: I don't think it's worth sending because this is not our end goal | 13:01 |
| fwereade_ | bigjools, go on... | 13:01 |
| rogpeppe | niemeyer: ok. i'm not sure what our end goal is then. | 13:01 |
| bigjools | fwereade_: the ZK machine can't reach the maas server. why that is, I don't know | 13:01 |
| fwereade_ | bigjools, oh, hell | 13:02 |
| niemeyer | Oct 01 11:54:59 <niemeyer> rogpeppe: In a future universe, we'll then introduce an HTTPS API to which everyone will talk to | 13:02 |
| rogpeppe | niemeyer: that's what i talk about in the email | 13:02 |
| rogpeppe | niemeyer: as an intro | 13:02 |
| rogpeppe | niemeyer: then i say that these are steps in that direction | 13:02 |
| bigjools | fwereade_: well, that "integer is required" error is the fundamental problem causing that in fact. So still not closer to working it out. | 13:03 |
| fwereade_ | bigjools, oh, for real tracebacks in twisted | 13:03 |
| niemeyer | rogpeppe: That's what I mean.. our focus is on implementing these steps. If you want to discuss future with James, that sounds great, but I suggest getting hold of him in two weeks and talking to him | 13:03 |
| bigjools | fwereade_: quite :/ | 13:04 |
| bigjools | fwereade_: you have to turn on deferred debugging to get them | 13:04 |
| fwereade_ | bigjools, I don't think we expose a switch for that | 13:04 |
| fwereade_ | bigjools, I guess you can always hack at the code the PA runs :/ | 13:04 |
| niemeyer | rogpeppe: Meanwhile, I hope we *implement* the steps, rather than just discuss how a perfect future looks like | 13:05 |
| bigjools | fwereade_: awesome :) | 13:05 |
| rogpeppe | niemeyer: ok. i was thinking that it might be useful to know if we're stepping in the right direction, but if you think that's fine, i won't send anything. | 13:05 |
| niemeyer | rogpeppe: I'm not sure I understand your concerns | 13:06 |
| niemeyer | rogpeppe: If authenticating is a step in the right direction? Of course it is.. if transport security is a step in the right direction? Of course it is | 13:06 |
| rogpeppe | niemeyer: it may be that we don't need anything of what we're doing now. | 13:06 |
| niemeyer | rogpeppe: We're not doing anything fancy.. we're implementing what should be in place from day zero | 13:06 |
| rogpeppe | niemeyer: we already authenticate and do transport security. | 13:06 |
| niemeyer | rogpeppe: Transport security? Authentication? | 13:06 |
| rogpeppe | ssh | 13:06 |
| niemeyer | rogpeppe: Please read the code of our agents :) | 13:07 |
| rogpeppe | niemeyer: ok, we don't currently use ssh intra-cloud, but we could. | 13:07 |
| rogpeppe | niemeyer: we could do SSL security without any of the SetPassword stuff. | 13:08 |
| niemeyer | rogpeppe: How do you put a client SSL certificate in place? | 13:08 |
| rogpeppe | niemeyer: that's a question for us now too | 13:10 |
| niemeyer | rogpeppe: Heh | 13:10 |
| rogpeppe | niemeyer: perhaps we're not concerned with man-in-the-middle attacks though | 13:11 |
| niemeyer | rogpeppe: What man-in-the-middle attacks? | 13:12 |
| niemeyer | rogpeppe: Do you have man-in-the-middle attacks when you use a password on gmail? | 13:13 |
| rogpeppe | niemeyer: yes, potentially. | 13:13 |
| niemeyer | rogpeppe: No, you don't unless you ignore the security warnings from your browser | 13:13 |
| niemeyer | rogpeppe: I'm happy to hear proposals that are better than the one I've explained. I'm not greatly interested in stopping progress to hunt for a proposal without clear articulation of what is the problem, the solution, and the way we'll get there in time. | 13:17 |
| rogpeppe | niemeyer: i suppose that's what i was trying to articulate. | 13:17 |
| niemeyer | rogpeppe: I haven't noticed that yet.. you just told me we already authenticate and do transport security | 13:18 |
| rogpeppe | niemeyer: i wanted to put my sketch up on juju-dev to see if anyone could see obvious flaws in it, as we'll probably be using this model for a while. do you think that's a bad idea? | 13:21 |
| niemeyer | rogpeppe: Yes, I personally think it is. I'd like to see progress being made instead of exposing a half-baked plan. This is not the end goal.. we'll *not* use database constraint to secure data. | 13:24 |
| rogpeppe | niemeyer: ok, fair enough | 13:24 |
| niemeyer | rogpeppe: It's up to you, though | 13:24 |
| rogpeppe | niemeyer: changing the subject, what do we do about admin-secret vs the environment config? | 13:24 |
| rogpeppe | niemeyer: it's secret, but we don't want to push it with the rest of the secrets | 13:24 |
| niemeyer | rogpeppe: I personally don't mind that you're talking about it with people, of course. Feel free to do contact James, juju-dev, or whoever else. | 13:24 |
| niemeyer | rogpeppe: I'll be doing pressure for progress, though. | 13:25 |
| niemeyer | rogpeppe: I want to see code being merged that improves the situation. | 13:25 |
| rogpeppe | niemeyer: i'm not going to delay things at all | 13:25 |
| niemeyer | rogpeppe: Also, doing homework is good.. | 13:26 |
| niemeyer | """ | 13:26 |
| niemeyer | It is not clear to me whether it is possible to make MongoDB | 13:26 |
| niemeyer | perform client-certificate verification; if it cannot, for the time being | 13:26 |
| niemeyer | we will remain vulnerable to man-in-the-middle attacks within the cloud. | 13:26 |
| niemeyer | """ | 13:26 |
| niemeyer | """ | 13:26 |
| niemeyer | Even within this interim | 13:26 |
| niemeyer | model, we can significantly improve things by separating concerns within | 13:26 |
| niemeyer | the database. For example the environment configuration (containing | 13:26 |
| niemeyer | the cloud access keys), the machines collection (allowing the creation | 13:26 |
| niemeyer | of new instances), and the unit-related collections could each be in | 13:26 |
| niemeyer | """ | 13:26 |
| niemeyer | We won't do that. | 13:26 |
| rogpeppe | niemeyer: no? | 13:26 |
| niemeyer | rogpeppe: No | 13:26 |
| rogpeppe | niemeyer: do we have transaction that span machines and units? | 13:27 |
| rogpeppe | transactions | 13:27 |
| niemeyer | rogpeppe: Erm? | 13:27 |
| rogpeppe | niemeyer: sorry, i jumped to conclusions. why won't we do that? | 13:28 |
| rogpeppe | niemeyer: it seemed to me like keeping the environ config separate might be a cheap and easy way to make things more secure. | 13:28 |
| niemeyer | rogpeppe: Cheap? How do you separate out everyone that needs the environment configuration? | 13:29 |
| niemeyer | rogpeppe: Everybody uses it right now | 13:29 |
| rogpeppe | niemeyer: hmm. i suppose we do need to read the private bucket. | 13:30 |
| niemeyer | rogpeppe: How does MongoDB authentication will permit you to do it in a single database, or alternatively how do you span multiple databases with transactions? | 13:30 |
| niemeyer | rogpeppe: The solution is not to hack together such change.. the solution is to have a real API to which clients talk to, instead of communicating with the database | 13:31 |
| rogpeppe | niemeyer: depends whether we need to span those things in a single transaction. i thought perhaps we did not. it's true i didn't check, though. | 13:31 |
| rogpeppe | niemeyer: yeah that's true at least. | 13:31 |
| rogpeppe | niemeyer: i did try to do my homework regarding mongod client-side certificate verification, but got lost in a) the source code b) the openssl docs. | 13:32 |
| === TheMue_ is now known as TheMue | ||
| rogpeppe | niemeyer: please disregard the proposed email. i wanted feedback and i got it, thanks. | 13:34 |
| rogpeppe | niemeyer: currently i am wondering whether to make admin-secret a special case, or to have a VerySecretAttrs method on EnvironProvider. | 13:35 |
| rogpeppe | niemeyer: i'm tending towards the former, and putting admin-secret in config.Config. | 13:36 |
| niemeyer | rogpeppe: Transport security and authentication, in place, working.. that's what we have to focus on for the moment. The need for the API is being strongly requested, and won't take long. What we're doing is a good step towards supporting it. | 13:37 |
| rogpeppe | niemeyer: that's cool. i understand that now. let's move on. | 13:37 |
| niemeyer | rogpeppe: Special case in which sense? | 13:38 |
| rogpeppe | niemeyer: we don't want to push it to the state | 13:38 |
| rogpeppe | niemeyer: at least i *think* we don't want to push it to the state | 13:39 |
| rogpeppe | niemeyer: otherwise it makes a mockery of our careful password management | 13:39 |
| niemeyer | rogpeppe: Hmm | 13:39 |
| niemeyer | rogpeppe: Indeed | 13:39 |
| niemeyer | rogpeppe: This will likely be somewhat boring, in fact.. | 13:40 |
| rogpeppe | niemeyer: yeah | 13:40 |
| niemeyer | rogpeppe: Since we replace the local config with the remote one regularly | 13:40 |
| niemeyer | rogpeppe: and the remote one won't have the secret | 13:40 |
| rogpeppe | niemeyer: i'm tempted to make it a special case | 13:41 |
| rogpeppe | niemeyer: and never push an attribute named "admin-secret" | 13:41 |
| niemeyer | rogpeppe: It is a special case in either case.. I'm just wondering what that means in practice | 13:41 |
| niemeyer | rogpeppe: Well, I guess we only need the password when connecting, so the regular replacement may not be much of an issue | 13:42 |
| rogpeppe | niemeyer: i'm not sure which piece you're thinking of when you say "we replace the local config with the remote one regularly" | 13:43 |
| rogpeppe | niemeyer: which "local" and which "remote"? | 13:43 |
| niemeyer | rogpeppe: the one in memory vs. the one in the database | 13:43 |
| rogpeppe | niemeyer: i don't think that's a problem - we'd never have the admin-secret attribute in the cloud | 13:44 |
| niemeyer | rogpeppe: Exactly | 13:44 |
| rogpeppe | niemeyer: ah, when *the client is* connecting. yeah. | 13:44 |
| niemeyer | rogpeppe: Means we'll lose the local password from the configuration.. but I think that's ok | 13:44 |
| rogpeppe | niemeyer: i don't think we *need* to remove the password from the client-side Config object, but perhaps that's not what you're thinking of | 13:45 |
| niemeyer | rogpeppe: I'm thinking it is going to be removed even if we don't need it | 13:46 |
| niemeyer | rogpeppe: Because we load the environment configuration from the remote side | 13:46 |
| rogpeppe | niemeyer: we just have to change BootstrapConfig to remove admin-secret too | 13:50 |
| rogpeppe | niemeyer: but it would seem a bit odd if SecretAttrs didn't return admin-secret actually | 13:51 |
| bigjools | fwereade_: I restarted the PA and it made the error go away ... wtf! | 13:51 |
| rogpeppe | niemeyer: so a better fix would be to remove admin-secret from the secrets within juju.Conn.updateSecrets | 13:52 |
| niemeyer | rogpeppe: Not sure.. that's a setting for the provider itself | 13:52 |
| rogpeppe | niemeyer: is it? | 13:52 |
| rogpeppe | niemeyer: aren't we doing provider non-specific stuff with it? | 13:53 |
| niemeyer | rogpeppe: EnvironProvider.SecretAttrs is | 13:53 |
| rogpeppe | niemeyer: ah, yeah | 13:53 |
| rogpeppe | niemeyer: in which case changing BootstrapConfig would seem better | 13:53 |
| niemeyer | rogpeppe: Yeah, I think it's quite fitting | 13:54 |
| niemeyer | rogpeppe: We should also add a panic to State.SetEnvironConfig | 13:54 |
| niemeyer | rogpeppe: In case it ever sees an admin-secret | 13:54 |
| rogpeppe | niemeyer: that seems reasonable | 13:55 |
| niemeyer | rogpeppe: Or perhaps just an error.. I think we might reach the panic with "juju set admin-secret=foo" | 13:55 |
| fwereade_ | bigjools, grar | 13:55 |
| rogpeppe | niemeyer: hmm yeah | 13:55 |
| rogpeppe | niemeyer: i was thinking a panic seemed a bit harsh actually | 13:55 |
| niemeyer | rogpeppe: +1 | 13:55 |
| rogpeppe | niemeyer: actually, we could make juju set admin-secret work if we wanted, i think | 13:56 |
| niemeyer | rogpeppe: Yeah, sounds sane, but it's a different code path anyway | 13:58 |
| rogpeppe | niemeyer: yeah | 13:58 |
| rogpeppe | niemeyer: the error case would remain | 13:59 |
| rogpeppe | niemeyer: oh yeah, small CL from this morning: https://codereview.appspot.com/6589072/ | 14:15 |
| TheMue | niemeyer: morning btw from me too | 14:33 |
| TheMue | niemeyer: and also a CL regarding the firewall mode in EC2: https://codereview.appspot.com/6589073/ | 14:33 |
| niemeyer | TheMue: Heya | 14:44 |
| niemeyer | rogpeppe: ping | 14:49 |
| rogpeppe | niemeyer: pong | 14:49 |
| niemeyer | rogpeppe: Have a moment for a call? An idea just crossed my mind | 14:49 |
| rogpeppe | niemeyer: sure | 14:49 |
| niemeyer | rogpeppe: I'm not sure if it's crack or not, or if it'd take longer or not, so would appreciate some brainstorm | 14:50 |
| rogpeppe | niemeyer: i'll fetch the other computer, so it doesn't die half way through | 14:50 |
| rogpeppe | niemeyer: on mo | 14:50 |
| niemeyer | rogpeppe: Cool | 14:50 |
| niemeyer | TheMue: Quick question before the review: is the *-global group needed? Couldn't we just use the one group? | 14:53 |
| fwereade_ | niemeyer, this should be a trivial: https://codereview.appspot.com/6607043 | 14:54 |
| TheMue | niemeyer: started that way too, but then I found it could make sense that our machines internally share one group with possible pure internal needed ports for machine to machine communication while the global group contains those ports open to the public. | 14:55 |
| niemeyer | TheMue: COol | 14:56 |
| niemeyer | TheMue: And is there any practical benefits with that? | 14:57 |
| TheMue | niemeyer: currently our firewall model doesn't use it, everything is done on the global group. but we should discuss it. | 15:00 |
| TheMue | niemeyer: have been interrupted here, sorry | 15:01 |
| TheMue | niemeyer: so we can distiinguish between the source | 15:03 |
| niemeyer | TheMue: Cool, sounds ok | 15:03 |
| TheMue | niemeyer: glad you like it | 15:04 |
| rogpeppe | niemeyer: admin-secret: https://codereview.appspot.com/6587085 | 15:32 |
| niemeyer | rogpeppe: Cheers | 15:33 |
| niemeyer | fwereade_: Is it just moving the package without any semantic changes? | 15:33 |
| fwereade_ | niemeyer, yes, + package comment tweaking | 15:33 |
| niemeyer | fwereade_: Beautiful, LGTM | 15:34 |
| fwereade_ | niemeyer, cheers | 15:34 |
| niemeyer | rogpeppe: Great stuff | 15:38 |
| rogpeppe | niemeyer: thanks! unfortunately i accidentally made it dependent on a prereq, so perhaps you could have a look at that too (it's very small) https://codereview.appspot.com/6589072/ | 15:39 |
| niemeyer | rogpeppe: Looking | 15:39 |
| niemeyer | rogpeppe: Hmm.. that's reviewed already | 15:39 |
| rogpeppe | niemeyer: oh really? cool! | 15:39 |
| niemeyer | Lunch.. biab | 15:52 |
| niemeyer | fwereade_: Back on uniter here | 16:44 |
| fwereade_ | niemeyer, ah, cool, thanks | 16:52 |
| niemeyer | fwereade_: Looking pretty good | 16:53 |
| niemeyer | fwereade_: Looks like you've managed to get the upgrade decision entirely inside the filter | 16:53 |
| fwereade_ | niemeyer, yeah, I think it's reasonably clean | 16:54 |
| fwereade_ | niemeyer, and it does make for a very nice interface | 16:54 |
| niemeyer | fwereade_: Very true | 16:55 |
| niemeyer | fwereade_: I'm thinking through the relationship between ModeInstalling and the follow up continuation | 16:55 |
| niemeyer | fwereade_: In terms of possible races given the different origins of the charm | 16:55 |
| fwereade_ | niemeyer, not sure I follow... this is rarely a good sign ;) what race? | 16:56 |
| niemeyer | fwereade_: Well, that's what I'm trying to find :) | 16:56 |
| fwereade_ | niemeyer, ok, well, I don't *think* the changes matter re upgrading at all: at some single point in time we write the current service charm in an install op, and that is the charm that gets installed, full stop | 16:58 |
| fwereade_ | s/upgrading/installing/ | 16:58 |
| niemeyer | fwereade_: When a delta is taken from foo to bar, and then we pick current state from baz, that's always an eye-opener | 16:59 |
| niemeyer | fwereade_: Okay, imagine this: | 16:59 |
| niemeyer | 1) Unit starts up | 16:59 |
| niemeyer | 2) filter goroutine starts up, and blocks | 17:00 |
| niemeyer | 3) ModeInstalling runs, and picks charm C1 | 17:00 |
| niemeyer | 4) filter runs, and picks charm C2 as current charm | 17:00 |
| niemeyer | fwereade_: What happens next? | 17:00 |
| fwereade_ | niemeyer, at some point after this we will be in a mode that needs charms, so it will ask for charm events relative to a baseline of C1, and (assuming appropriate forcing) get the *Charm corresponding to C2 next time it reads from charmEvents | 17:02 |
| niemeyer | fwereade_: Why? C1 was never seen by the filter | 17:02 |
| fwereade_ | niemeyer, yeah, we tell the filter what charm events we're interested in | 17:02 |
| niemeyer | fwereade_: Ah, maybe that's what I'm missing | 17:03 |
| fwereade_ | niemeyer, wantCharmEvent now takes something like (upfradeFrom *state.Charm, mustForce bool) | 17:03 |
| niemeyer | fwereade_: Aha, makes sense, thanks | 17:03 |
| niemeyer | fwereade_: Quite nice | 17:04 |
| niemeyer | fwereade_: Very nice, in fact | 17:05 |
| fwereade_ | niemeyer, cheers :) | 17:05 |
| niemeyer | fwereade_: Okay, so.. | 17:06 |
| niemeyer | fwereade_: New branch/logic/etc is *awesome* | 17:06 |
| niemeyer | fwereade_: You did changes inside the filter that look like a great direction too | 17:07 |
| niemeyer | fwereade_: The select statement is neat and tight | 17:07 |
| fwereade_ | niemeyer, it seemed worth trying again, and it seemed to work out ok this time :) | 17:07 |
| niemeyer | fwereade_: There's one thing I think we can improve slightly, but it's just in terms of how to do exactly the same thing, rather than changing it | 17:08 |
| fwereade_ | niemeyer, great, improvements always welcome :) | 17:08 |
| fwereade_ | [in the background, a wail of shock: no! mum! I don't want to eat salad!] | 17:08 |
| niemeyer | ROTFL | 17:09 |
| niemeyer | fwereade_: These new closures are great in terms of naming and isolating logic, but they're begging to be real methods | 17:09 |
| fwereade_ | niemeyer, yeah, I felt the pull | 17:09 |
| fwereade_ | niemeyer, and then I felt like putting them on the same type as the chans would end up obscuring its purpose, and that a separate type wasn't quite right | 17:10 |
| fwereade_ | niemeyer, which would you favour? | 17:10 |
| niemeyer | fwereade_: This, I suspect, will also reduce a bit the massive namespacing that we have within that one method | 17:10 |
| fwereade_ | niemeyer, OTOH the same type as the chans is clearly the right place, given that they manipulate them | 17:10 |
| fwereade_ | niemeyer, yeah, indeed | 17:11 |
| fwereade_ | niemeyer, incidentally, obscuring the field namespace was also a concern | 17:11 |
| niemeyer | fwereade_: I'd put them in the same type.. they're a different category of methods, guaranteed | 17:11 |
| fwereade_ | niemeyer, if I'm going to have a busy namespace a single function scope sometimes seems like the right place ;p | 17:11 |
| niemeyer | fwereade_: They are private methods, except everything is private at the moment because the whole type is private | 17:12 |
| fwereade_ | niemeyer, how would yu feel about me exposing filter as Filter, and making the Events methods public? | 17:12 |
| fwereade_ | niemeyer, or just the methods even | 17:12 |
| niemeyer | fwereade_: I was about to suggest the latter half only | 17:12 |
| niemeyer | fwereade_: +1 | 17:12 |
| fwereade_ | niemeyer, convergence :) | 17:12 |
| niemeyer | fwereade_: ftw :) | 17:12 |
| fwereade_ | niemeyer, great | 17:13 |
| niemeyer | fwereade_: The field names so far are very clear.. we have well known fields, plus want* and out*.. we can then share a few private things with proper names, and I suspect the methods will hide a few variables within their own scope | 17:14 |
| fwereade_ | niemeyer, yeah, I think it'll work out pretty nice | 17:15 |
| fwereade_ | niemeyer, it's just the duplication of out* names that bugs me | 17:18 |
| niemeyer | fwereade_: Hmm, which duplication? | 17:20 |
| fwereade_ | niemeyer, the field that holds the real chan and the one that's the same but sometimes nil | 17:22 |
| fwereade_ | niemeyer, which is manipulated in those closures | 17:22 |
| fwereade_ | niemeyer, outCharm = nil; outCharm = f.outCharm | 17:22 |
| niemeyer | fwereade_: Hmm.. that doesn't feel too bad to me | 17:24 |
| fwereade_ | niemeyer, well, I certainly can't think of good names for the pair | 17:24 |
| fwereade_ | niemeyer, the outCharm above will need t be a field,right? | 17:24 |
| niemeyer | fwereade_: Ah, okay, I see | 17:25 |
| fwereade_ | niemeyer, f.maybeOutCharm = f.outCharm | 17:25 |
| fwereade_ | ;p | 17:25 |
| niemeyer | fwereade_: f.outCharm and f.outCharmOn? | 17:25 |
| fwereade_ | niemeyer, f.outCharm = f.outCharmOn | 17:26 |
| fwereade_ | niemeyer, I like it | 17:26 |
| fwereade_ | niemeyer, ty | 17:26 |
| niemeyer | fwereade_: np | 17:26 |
| rogpeppe | niemeyer: i'm thinking about putting some encoding of the admin-secret (secure hash or b64) into cloudinit, rather than the raw password, as using the raw password exposes us to awkward upstart quoting issues. | 17:27 |
| rogpeppe | niemeyer: do you think that's reasonable? | 17:27 |
| niemeyer | rogpeppe: I don't think we ever want the raw password in cloud init | 17:28 |
| rogpeppe | niemeyer: good point. doh. | 17:28 |
| rogpeppe | niemeyer: so we can assume that the Password in the StateInfo passed to cloud-init is always nicely formed. | 17:30 |
| niemeyer | rogpeppe: As in, contains reasonable text? Yeah | 17:34 |
| rogpeppe | niemeyer: yeah | 17:34 |
| rogpeppe | i'm off for the day. see y'all tomorrow. | 17:40 |
| niemeyer | rogpeppe: Cheers man | 17:48 |
| niemeyer | fwereade_: Sent some other minor comments on the review of that same branch | 17:48 |
| niemeyer | fwereade_: Only one point there may need some further talking | 17:49 |
| fwereade_ | niemeyer, heyhey | 17:49 |
| fwereade_ | niemeyer, I'll take a look | 17:50 |
| fwereade_ | niemeyer, about the config events? | 17:51 |
| niemeyer | fwereade_: Yeah | 17:51 |
| fwereade_ | niemeyer, it just means that if we get a really early wantConfig, we don't end up sending an event not corresponding to a real change when the initial watcher change shows up | 17:51 |
| fwereade_ | niemeyer, clearly it's not expressed very well :) | 17:51 |
| niemeyer | fwereade_: I don't think I get it still | 17:53 |
| fwereade_ | niemeyer, and re "Upgrading?" -- yes, only ModeUpgrading checks for ErrConflict, because all ModeInstalling does is pull into a new empty dir, and is therefore unlikely to experience conflicts ;) | 17:53 |
| niemeyer | fwereade_: How does it prevent anything.. the order is still unpredictable, and the block in the early select is the same block on the later select | 17:53 |
| fwereade_ | niemeyer, the order is not unpredictable: doing what I do guarantees that the config.Changes() chan will be read at least once before the want chan | 17:54 |
| fwereade_ | niemeyer, and that therefore it is impossible to (re?)activate the outConfig chan via a want *before* getting the initial event which is actually un *un*requested resend of the "original" event | 17:55 |
| fwereade_ | niemeyer, yeah, the above is not clear | 17:56 |
| fwereade_ | niemeyer, consider without that block | 17:56 |
| fwereade_ | niemeyer, 1) start config watch | 17:56 |
| fwereade_ | niemeyer, 2) get a wantConfig | 17:56 |
| fwereade_ | niemeyer, 3) send a config event | 17:57 |
| fwereade_ | niemeyer, 4) get the initial config change | 17:57 |
| fwereade_ | niemeyer, 5) send a config event | 17:57 |
| fwereade_ | niemeyer, the above sequence STM to be possible -- two events from a single config after only one want | 17:58 |
| niemeyer | fwereade_: Makes sense, thanks | 17:59 |
| fwereade_ | niemeyer, np, I'll try to make it clear in the code | 17:59 |
| niemeyer | fwereade_: Thanks! | 17:59 |
| fwereade_ | niemeyer, did the Upgrading? explanation make sense? | 17:59 |
| niemeyer | fwereade_: Does as well, thank you | 18:00 |
| fwereade_ | niemeyer, cool | 18:00 |
| fwereade_ | niemeyer, (just to check: you're ok with wantUpgradeEvent etc over wantCharmEvent?) | 18:02 |
| fwereade_ | niemeyer, that's what it actually *is* now after all :) | 18:03 |
| niemeyer | fwereade_: Yeah, definitely | 18:03 |
| niemeyer | fwereade_: Happy with it, actually | 18:03 |
| fwereade_ | niemeyer, sudden hare-brained idea, but since you're here: | 18:04 |
| fwereade_ | niemeyer, u.f.wantConfigEvent() is not really so bad | 18:04 |
| fwereade_ | niemeyer, I don't really think we get much from emebedding | 18:04 |
| niemeyer | fwereade_: +1 | 18:05 |
| fwereade_ | niemeyer, cool | 18:05 |
| fwereade_ | niemeyer, I should manage to propose again later tonight | 18:05 |
| niemeyer | fwereade_: Superb | 18:05 |
| niemeyer | fwereade_: I'll have to step out in a bit to sign routine bank papers, but will be back working later too | 18:06 |
| fwereade_ | niemeyer, have fun :) | 18:06 |
| niemeyer | fwereade_: "fun" :) | 18:07 |
| * niemeyer steps out | 18:30 | |
| === fss is now known as flaviamissi_ | ||
| === flaviamissi_ is now known as fss | ||
| * niemeyer is back | 20:16 | |
| fwereade_ | niemeyer, https://codereview.appspot.com/6588053 reproposed | 21:57 |
| niemeyer | fwereade_: Brilliant.. on the phone with mramm, but will be there soon | 21:57 |
| fwereade_ | niemeyer, cool, thanks | 21:57 |
| niemeyer | fwereade_: Looking | 22:15 |
| fwereade_ | niemeyer, cheers | 22:15 |
| niemeyer | fwereade_: done | 22:40 |
| niemeyer | fwereade_: LGTM, with a few last suggestions for your consideration | 22:40 |
| fwereade_ | niemeyer, awesome :D | 22:40 |
| fwereade_ | niemeyer, I'll take a look | 22:40 |
| fwereade_ | niemeyer, I'm a bit uncomfortable about serviceCharm.force sometimes meaning "force" and sometimes meaning "must force" depending on the var... but the only reason I didn't succumb to its convenience was because I though you wouldn't like it :) | 22:43 |
| fwereade_ | niemeyer, not sure I'll quite get that merged tonight, I'm thinking sleep sounds interesting once I get my current jujuc butchery building again | 22:43 |
| niemeyer | fwereade_: I can definitely understand.. it took me some time to suggest variable names properly for this to not be awkward | 22:43 |
| fwereade_ | niemeyer, yeah, those names help a lot | 22:44 |
| niemeyer | fwereade_: upgradeRequested.force and upgradeAvailable.force both sound about right, if not ideal | 22:44 |
| fwereade_ | niemeyer, definitely better than anything I could think of :) | 22:45 |
| niemeyer | fwereade_: About sleep, I can only say have a great one! :-) | 22:45 |
| niemeyer | davecheney: Morning! | 22:54 |
| niemeyer | We've got an empty review queue again | 22:54 |
| davecheney | niemeyer: nice one | 22:54 |
| davecheney | i'll see what I can do about that | 22:54 |
| davecheney | just trying to debug this atm | 22:54 |
| davecheney | lucky(~/src/launchpad.net/juju-core) % juju destroy-environment | 22:54 |
| davecheney | error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. (SignatureDoesNotMatch) | 22:54 |
| niemeyer | davecheney: Uh oh ;) | 22:54 |
| niemeyer | davecheney: Curious | 22:55 |
| davecheney | ^ happens when I have more than a few machines' running | 22:55 |
| niemeyer | davecheney: Up-to-date goamz/ec2? | 22:55 |
| * davecheney shrugs | 22:55 | |
| davecheney | it's the one i've been using since hte last goamz fix | 22:55 |
| * davecheney mumbles something about having to add tests to catch people using old pkg versions | 22:56 | |
| niemeyer | davecheney: Yeah, seems interesting | 22:57 |
| davecheney | niemeyer: did you see the results from my charm testing last night | 22:58 |
| davecheney | 9 out of 19 charms worked | 22:59 |
| niemeyer | davecheney: Woah, I hadn't seen that | 22:59 |
| davecheney | non failed because of compatability issues between py and go | 22:59 |
| davecheney | niemeyer: check the logs for ~12 hours ago | 22:59 |
| davecheney | niemeyer: http://paste.ubuntu.com/1259806/ | 23:03 |
| niemeyer | davecheney: Woah | 23:04 |
| niemeyer | This is *awesome* | 23:04 |
| niemeyer | davecheney: Deserves a mail to juju@ | 23:04 |
| davecheney | i don't have it setup on this machine | 23:05 |
| davecheney | i was going to do a bit more triage on the failing charms | 23:05 |
| davecheney | niemeyer: https://bugs.launchpad.net/juju-core/+bug/1061941 | 23:05 |
| davecheney | bug report for aws destroy-environment failure | 23:05 |
| niemeyer | davecheney: Can you post some more details of the problem there? | 23:07 |
| niemeyer | davecheney: This is indeed a critical issue | 23:07 |
| niemeyer | davecheney: and probably easy to solve too | 23:08 |
| niemeyer | davecheney: There's a private debug flag withing goamz/ec2 | 23:08 |
| niemeyer | davecheney: I'm working a bit on multi-config meanwhile | 23:08 |
| davecheney | niemeyer: ok, i'll rebuild goamz now i have a test case | 23:14 |
| davecheney | niemeyer: please review the irc logs from yesterday | 23:16 |
| davecheney | re: the failed charms | 23:16 |
| niemeyer | davecheney: Can you paste the section you'd like me to look at so I don't have to second guess? | 23:21 |
| davecheney | niemeyer: http://irclogs.ubuntu.com/2012/10/04/%23juju-dev.html#t11:48 | 23:25 |
| davecheney | ^ are we missing a command ? | 23:25 |
| davecheney | http://irclogs.ubuntu.com/2012/10/04/%23juju-dev.html#t11:57 | 23:25 |
| davecheney | ^ dud charm | 23:25 |
| davecheney | http://irclogs.ubuntu.com/2012/10/04/%23juju-dev.html#t11:59 | 23:25 |
| davecheney | ^ hard coded tools path | 23:25 |
| davecheney | http://irclogs.ubuntu.com/2012/10/04/%23juju-dev.html#t12:01 | 23:26 |
| davecheney | ^ unhygenic charm | 23:26 |
| davecheney | and that was as far as I got before I realised destroy-environment wasn't working :) | 23:26 |
| niemeyer | davecheney: Which command are we missing? | 23:27 |
| davecheney | is bucket-create a juju command ? | 23:28 |
| niemeyer | davecheney: It's a bit hard to read it all and guess what you think is going on | 23:28 |
| davecheney | niemeyer: yeah, i'll do more triage today | 23:29 |
| niemeyer | davecheney: " /opt/couchbase/bin/couchbase-cli bucket-create" | 23:29 |
| niemeyer | davecheney: That's couchbase, not juju | 23:29 |
| davecheney | right, then the tally is 9 out of 19 charms make it to started, non fail because of compatibility between us and py juju | 23:29 |
| davecheney | niemeyer: http://jujucharms.com/charms/precise | 23:30 |
| davecheney | ^ can i get this data in xml/json ? | 23:30 |
| niemeyer | davecheney: The list of charms? | 23:31 |
| davecheney | yup | 23:31 |
| niemeyer | davecheney: Let me see.. | 23:31 |
| davecheney | just the charm names | 23:31 |
| davecheney | niemeyer: thank you for your review on http://codereview.appspot.com/6591080/ | 23:37 |
| davecheney | i know it's not the right solution, but it lets me at least have a working deploy | 23:37 |
| niemeyer | davecheney: np, the interim solution looks good | 23:38 |
| niemeyer | davecheney: Doesn't corrupt the state, and easy to rollback | 23:38 |
| davecheney | and it will be fairly obvious for others coming alone later | 23:39 |
| davecheney | niemeyer: what do you think about adding so documentation to the top level, INSTALL and CONTRIBUTE | 23:47 |
| davecheney | also adding a scripts directory for some of the things like the stress test | 23:47 |
| davecheney | and running gocov will need some scripting support | 23:47 |
| niemeyer | davecheney: +1 on docs.. for the stress test, I'd prefer for it to have its own code base | 23:48 |
| davecheney | even if it's just a bash script ? | 23:49 |
| niemeyer | davecheney: Okay, that sounds fine to have in | 23:49 |
| niemeyer | davecheney: Unless the bash script is doing while true; do go test; done | 23:50 |
| niemeyer | ;) | 23:50 |
| davecheney | well, there is a little bit more, where it sets a random GOMAXPROCS | 23:50 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!