[02:15] what does bzr mean when it returns error code 3 ? [06:02] rogpeppe: i think i have a solution to maek the ssh tests much cleaner [06:02] if you want it [06:02] -- but i know you're busy tearing that out [06:43] fwereade_: morning! [06:43] rogpeppe, heyhey [06:43] rogpeppe, how's it going? [06:43] fwereade_: sorry about trunk breakage last night [06:43] fwereade_: i didn't realise i had an old version of mongodb [06:44] rogpeppe, no worries [06:44] fwereade_: not bad at all, thanks. [06:44] fwereade_: and you? [06:44] rogpeppe, yeah, not bad :) [07:00] morning [07:05] fwereade_: i'm adding admin-secret to the environment configuration, and i *think* it should be part of config.Config, not specific to the ec2 config. it's an interesting case though, because it's *really* secret - it doesn't get pushed with the other secrets. [07:06] fwereade_: so i'm not sure whether to have a "ReallySecretAttrs" method in environs, or just to make a special case for admin-secret [07:06] * fwereade_ thinks [07:06] morning TheMue btw [07:07] fwereade_: heya [07:07] rogpeppe, sorry, no strong feelings either way, just a sense of unease [07:08] fwereade_: yeah, special cases make me uncomfortable too [07:08] TheMue: hiya [07:09] rogpeppe: a wonderful morning to you too (even if it is raining here) ;) [07:09] TheMue: beautifully sunny this morning. (sun just up) [07:10] rogpeppe: we had a nice october so far until yesterday. since then mostly rain, too much rain [07:43] rogpeppe, wow, massive filter change works as expected; simplifies uniter noticeably; but pushes the tests up over a minute :/ [07:44] fwereade_: cool, but... [07:44] fwereade_: what's taking all the time in the tests, out of interest? [07:44] rogpeppe, thus far I know not, I will do some poking around [07:44] fwereade_: worth doing, i think. [07:45] fwereade_: go test -gocheck.vv | timestamp is always a good start [07:45] rogpeppe, some of it is probably attributable to just doing more work more often but iyt's a bit of a big change for that I think [07:55] fwereade_: a small CL: https://codereview.appspot.com/6589072/ [07:56] fwereade_: for context, here's the authentication sketch: http://paste.ubuntu.com/1259533/ [07:56] davecheney: it would be good to have some feedback from you about the authentication scheme too, if poss [07:56] rogpeppe: sure thing [07:57] rogpeppe: did you see my comment about improving the ssh tests ? [07:57] davecheney: i did [07:57] althought I realise the horse has bolted [07:57] davecheney: yeah, no point in flogging that one [07:57] rogpeppe: right-o [07:57] rogpeppe, so --initial-password will be ignored if a password file exists [07:57] davecheney: i'm interested to know what your plan was though [07:57] fwereade_: yes [07:57] rogpeppe: http://codereview.appspot.com/6601043/diff/3011/ssh/sshtest/sshtest_unix_test.go [07:57] ~ line 150 [07:57] fwereade_: well actually, it might not be [07:58] rogpeppe: which CL is the authn Cl ? [07:58] fwereade_: if we fail to connect with the password, we'll try initial-password [07:58] the paste ? [07:58] davecheney: yeah [07:58] kk [07:58] fwereade_: because we might have written the password file but failed to change it [07:59] rogpeppe, ah, sensible [07:59] rogpeppe: i'm not sure how helpful this is [08:00] but the primary customer inside canonical is Elmo [08:00] so if he doesn't like the smell of this [08:00] irrespective of its other merits [08:00] it's game over [08:00] not saying he won't like it, or that what you have is not correct [08:00] rogpeppe, CL LGTM, I will try to get myself into a suitably adversarial mode before tackling the auth overview [08:00] davecheney: i'm not sure who Elmo is [08:01] me neither [08:01] but I hear he's the big cheese of the internal sysadmin team [08:02] davecheney: ok, i'll make a write-up and put it on juju-dev [08:03] rogpeppe: that sounds like an excellent plan, then others can distribute as necessary [08:04] rogpeppe: my only comment of note, is the storing of the key on disk per machine agent [08:04] i don't have a solution to this [08:04] davecheney: indeed. we need to store it somewhere [08:04] only observe that others will see it as a potential loophole [08:04] davecheney: we must be able to connect after reboot [08:04] yeah, [08:04] is there a concept of differeing levels of privilige ? [08:04] davecheney: ssh has exactly the same issue [08:05] rogpeppe: yup, it sure does [08:05] davecheney: in fact any autonomous agent must have the same issue [08:05] davecheney: the only solution is authenticated h/w [08:05] davecheney: which we don't have. [08:06] rogpeppe: and i'm not sure if that would actually solve the problem [08:06] the issue, as i understand it is [08:06] user X on machine Y can get root, then get whatever details they need to connect to the state, rip off the AWS keys .. [08:06] is that correct ? [08:07] davecheney: if there was a way of propagating a secret from bootstrap stage to the agent itself, then we could use the secret, then destroy it. then even if you were root, you couldn't get it. [08:08] davecheney: but of course the agent needs to keep the secret around, so even then we're vulnerable [08:08] rogpeppe: the secrets are in the /e document in the /e collection, right ? [08:08] davecheney: it's a pity everyone has root access [08:08] davecheney: no [08:08] davecheney: these secrets are not [08:08] rogpeppe: but the AWS creds we're trying to protect are [08:09] davecheney: yes, they are currently [08:09] davecheney: but they won't be when we use this scheme to leverage principal-specific access controls [08:09] so, if by some mech, the /e document could be protected from access by the machine agent, would that be a solution ? [08:10] davecheney: it's not a solution to malicious entities on the machine being able to impersonate the machine agent [08:10] davecheney: but that is necessary too, yes [08:10] rogpeppe: is their a spec for the security model ? [08:10] davecheney: no [08:11] davecheney: sigh [08:11] rogpeppe: i'm not sure how to proceed without this [08:11] at best you'll implement whatever is inside gustavos head [08:11] and worse, you won't [08:11] and niether case may be what customres want [08:12] davecheney: i see what we're doing here as a necessary prelude to implementing the final security model, which is unspecified [08:12] davecheney: we're adding the notion of a principal to the state info, which i think is always going to be necessary. [08:12] rogpeppe: i'm concerned it is equiv to starting to walk in an unspecified direction, without picking a destination [08:13] rogpeppe: please understand, i'm not have a go at your solution, [08:13] davecheney: ok, the basic security model, as i understand it is: [08:13] you know my pickyness for implementing security without a spec [08:14] davecheney: agents identify themselves to the state; the state allows agents to agent-specific things. [08:14] s/to agent/to do agent/ [08:14] rogpeppe: but there is an unspoken requirement that agents cannot be impersonanted [08:14] davecheney: yes... well, kinda. [08:15] davecheney: damage limitation [08:15] davecheney: we don't want any random non-root user on a machine to be able to impersonate that machine's machine agent. [08:15] davecheney: but if you're root, you're going to be able to do what you damn please [08:15] rogpeppe: then putting the per machine agent password in a 0600 file would work [08:16] but i think further consultation with the customer is needed [08:16] davecheney: yes, that's what we're doing [08:16] i'm pretty sure that someone is going to say 'but what if they get root' [08:16] davecheney: there's nothing we can do in that case. [08:17] yup [08:17] davecheney: the trickiness in the spec i pasted above is because there's no way of passing a secret to the initial machine agent that's not accessible by non-root users. [08:17] davecheney: hence --initial-password [08:17] but that is probably going to meen elmo rejects the idea, and we've done a lot of work for nohting [08:17] davecheney: he can't ask the impossible [08:18] he's a very powerful customer [08:18] davecheney: this is significantly better than what we had before - entities get permissions on a need-to-have basis. [08:18] davecheney: sure, but we're talking *impossible* here [08:19] davecheney: noone else will be able to do better [08:19] rogpeppe: i never said the customer was rational :) [08:20] davecheney: also, you won't be able to do much by impersonating a machine agent, even if you are root. [08:20] and malicious [08:21] davecheney: in fact the machine agent doesn't need to be able to write to the state at all [08:22] davecheney: you can do a little more by impersonating a unit agent, but again, not too bad, i think. [08:23] davecheney: this all assumes an entity-aware API of course. [08:24] davecheney: the thing i'm most concerned with is man-in-the-middle attacks. i don't see how we can protect against those unless we have some kind of key-distribution scheme. [08:26] rogpeppe: them why does the MA need a password at all ?> [08:26] i we treat the machine and the machine agent as untrusted [08:26] you can ignore their credentials [08:26] davecheney: we want to partition machines [08:27] rogpeppe: can we trust the LXC security boundary ? [08:27] davecheney: and i don't think we can entirely assume that non-root users can always obtain root. [08:27] davecheney: apparently not. for root users within LXC anyway. [08:27] rogpeppe: if that is a working assumption, then the job is a lot easier [08:29] davecheney: i think our advice should be (as per usual) don't run untrusted stuff as root. [08:29] seconded [08:30] davecheney: this means we've essentially got a two-tier security model. primary layer: TLS-based authentication; secondary layer: entity name/password authentication [08:31] davecheney: we assume that a malicious root user can bypass the secondary layer, but not the primary layer. [08:32] rogpeppe: is the tls layer using client side certs ? [08:32] davecheney: it'd better! [08:32] otherwise it isn't a security authn mech :) [08:32] davecheney: indeed [08:32] davecheney: and vulnerable to man-in-the-middle too [08:33] davecheney: we need a way of securely passing a cert to a new machine [08:33] davecheney: ISTR that amz supports this [08:33] davecheney: dunno about others [08:33] rogpeppe: in the puppet model the client generates the cert and sends it to the server for signing [08:34] davecheney: how does the server know where the cert is coming from? [08:34] the admin is expected to manage that out of band [08:34] just like gpg [08:34] davecheney: we can't do that - it's all autonomous [08:35] ie, if you weren't expecting to see a cert request, then don't sign it [08:35] of course, most envs turn on automatic cert signing [08:35] davecheney: of course. [08:35] davecheney: i don't even see how an admin can know [08:35] security is hard, shall we go shopping ? [08:35] davecheney: ooh shiny [08:36] rogpeppe: generally you install machine x, insgtall puppet, then tell it to join your puppet server [08:36] go to the server, accept the request [08:36] then profit [08:36] davecheney: what if someone got in there between the install and going to the server? [08:36] davecheney: of course, it may be improbable, but... [08:37] puppet assumes you control the security of your environment [08:37] it's a cfg management tool [08:37] and there is a reason juju exists :) [08:37] rogpeppe: my friends that work in hosting companies [08:37] run one puppet server per customer [08:37] davecheney: the way i did this when we had a similar thing was i installed manually on a machine, including a private key. [08:37] the idea of a single puppet instnace for all customers is impractical [08:38] davecheney: then when the machine dials in, you *know* that it's the right machine. [08:38] davecheney: but that assumes a manual install, of course. [08:38] rogpeppe: yup [08:38] davecheney: which is why you have to assume that the cloud provider can do something similar [08:38] my friends in hosting companies use vlans and shit [08:38] to separate customers environments [08:39] davecheney: worst comes to worst you're vulnerable to mitm, but if the hosting co is compromised, you're fucked anyway [08:42] davecheney: "For Linux instances, you can provide an optional key pair ID in the launch request (created using the CreateKeyPair or ImportKeyPair operation). The instances will have access to the public key at boot. You can use this key to provide secure access to an instance of an image on a per-instance basis. Amazon EC2 public images use this feature to provide secure access without passwords." [08:43] davecheney: i *think* we can leverage that [08:46] rogpeppe: ooooooooooh [08:46] but, are we likely to end up in the same 'too many firewall groups' quagmire ? [08:47] davecheney: i don't think there should be a problem creating a keypair per machine, but i may well be wrong :-) [08:47] another stupid amazon limitation [08:47] and you'll probably get asked how to do it in the openstack/azure/hp world [08:50] davecheney: the main problem is that this mechanism is designed for allowing you to connect to a new machine securely, not the other way around. [08:51] mmm [08:51] davecheney: i think we'd need to create a cert based on a hash of the key pair's public key or something. [08:58] davechen1y: actually, i think we could probably go quite a long way by moving the environ config into a separate database. [08:59] davechen1y: then we could at least restrict access to the AWS keys without needing an API [08:59] rogpeppe: yes [08:59] which sounds like the juju-as-a-service plan [08:59] customers have MA's, we run the PA [08:59] davechen1y: yeah, that's certainly part of it [08:59] davechen1y: by separate database, i didn't mean a separate mongo server, as it happens [09:00] davechen1y: i meant a separate mgo.Database [09:00] davechen1y: (we already use two) [09:00] rogpeppe: yup [09:00] i thought we did that already [09:01] or is that just different connections, same db ? [09:01] davechen1y: same connection, different dbs [09:01] davechen1y: but each db has its own set of users [09:02] davechen1y: did you see this CL BTW? https://codereview.appspot.com/6587060 [09:05] * davechen1y looks [09:09] rogpeppe: fwereade_ trivial: http://codereview.appspot.com/6601056/ [09:10] looking for a LGTM [09:10] * fwereade_ looks [09:11] davechen1y: LGTM, although i suppose the question is: why --format and not something else? [09:12] rogpeppe: this is the thing that broke [09:12] and it's intended to get people to update their gnuflag instance [09:12] davechen1y: fair enough. [09:12] davechen1y: it's gotta be somewhere; why not there? [09:13] davechen1y: (rhetorical question) [09:16] davechen1y, LGTM [09:17] thanks folks [09:18] * fwereade_ has made the uniter tests fast again by fixing a huge obvious repeated 500ms sleep [09:18] * fwereade_ knows that has been there for ages [09:18] * davechen1y applauds [09:19] * rogpeppe applauds loudly [09:19] fwereade_: duration of uniter tests now? [09:20] rogpeppe, back to ~45s [09:20] fwereade_: hmm, still slow then [09:21] rogpeppe, well, yeah, the bit I fixed was pre-existing, so there's probably another 20s to be extracted somewhere, but it's really not obvious [09:21] fwereade_: 20s shorter would be much more reasonable... but i know the feeling. [09:22] davechen1y: here's a draft of the first part of a heads-up email: http://paste.ubuntu.com/1259631/ [09:23] * davechen1y reads [09:24] i don't see why we need both certs and usernames/passwords [09:25] certs can already be associated with a principal [09:25] davechen1y: in mongodb? [09:25] davechen1y: (i tend to agree - i feel that passwords are a bit retro) [09:28] rogpeppe, davechen1y: I need to pop out for a while, but I have this: https://codereview.appspot.com/6588053 [09:28] rogpeppe, davechen1y: still WIP [09:29] rogpeppe: not sure how mongo does it [09:29] rogpeppe, davechen1y: but it incorporates some significant changes after niemeyer's suggestions in various places [09:29] fwereade_: will have another look [09:29] but if you have a TLS cert, then you can request client authentication (ie, they need a sub cert signed by the same CA that issued your cert) [09:29] rogpeppe, davechen1y: and if you have time to cast a quick eye over it for general sanity that would be great [09:29] or the TLS handshake fails [09:29] rogpeppe, issues like "all the filter tests are broken" are not what I'm looking for ;p [09:29] bbiab [09:30] davechen1y: i'm wondering how/whether mongo converts client certs into mongo users [09:30] sounds complicate [09:30] compilcated [09:30] i reckon drop it [09:31] just use TLS for a secure channel to transmit creds over [09:31] davechen1y: so don't connect direct to mongo? [09:31] davechen1y: but use a forwarder? [09:32] davechen1y: i don't know if that's easy [09:32] (although at least the mongo client is written in Go) [09:36] oh why, oh why are certificates so horribly useless in this world? [09:37] and hard, dont' forget that [09:37] davechen1y: indeed [09:37] davechen1y: so unnecessary [09:38] davechen1y: it looks to me as if mongodb can't do client-certificate verification [09:39] scratch that [10:26] yo. [11:01] Aram: hiya [11:09] rogpeppe: fwereade_ : comments ? https://codereview.appspot.com/6591080 [11:15] davecheney: looks reasonable to me, but i'm not really familiar with the issue, i'm afraid [11:15] davecheney: "I want to ask that it be accepted." - you can always just *ask*, y'know :-) [11:17] rogpeppe: what mongodb version do you use? [11:17] Aram: i've just started using a different version [11:17] Aram: i was using... 2.0.3 i think [11:17] aha. [11:17] Aram: now i've downloaded the version we use on ec2 and am using that [11:18] I'm kind of nervous that behavior changes so often between versions so close to eachother. [11:18] Aram: me too. it's a pretty shitty thing to get wrong. [11:18] slow cloud-init, is slow [11:19] Aram: rogpeppe: i think we should switch to using the version from the public bucket, exclusively [11:19] davecheney: and download it each time we run a test? [11:21] no, i'm sure there is a way to avoid that expense [11:21] davecheney: it's important that we be able to run tests on an aeroplane too. [11:22] rogpeppe: i think you're reading too much into my suggestion [11:22] davecheney: i suppose we could check the mongodb binaries into the repo [11:22] i'm just thinking of unpacking the version into somewhere inside the juju-tree [11:22] then just call that path directly [11:22] if it's there 'win' [11:22] if not, fail [11:23] * davecheney wishes we have juju destroy-service [11:23] davecheney: is there no such command in the original juju? [11:23] yeah, but we don't have it in cmd/juju yet [11:23] makes testing harder :) [11:25] davecheney: if we don't check it into the repo (and i'm not sure we want to clutter the repo with an 8MB binary) i'm not sure that running from a different path buys us that much [11:25] rogpeppe: we don't have to check it in [11:26] davecheney: we'd be better off running mongod --version to check that we get the expected version [11:26] davecheney: if we don't check it in, then we have the same problem of possible version skew [11:26] just change the mongo tests to call to a specific path, one where we have already downloaded the mongodb version, rather than just calling any mongod in th epath [11:28] ha, it looks like niemeyer didn't build mongod with ssl support [11:29] I thought that building with SSL was the reason why we needed to build it. [11:29] rogpeppe: crap -- that was the _ENTIRE_ reason for cmd/builddb [11:29] Aram: me too [11:30] davecheney: try mongod --help 2>&1 | grep -i ssl [11:31] % juju deploy couchbase &error: cannot assign unit "cf-mongodb/0" to machine: cannot assign unit "cf-mongodb/0" to machine 8: duplicate key insert for unique index of capped collection [11:31] davecheney: hmm, it *looks* as if builddb builds it with ssl [11:32] i wish we prefixed our errors with "juju: " rather than "error: ". i've been meaning to fix that for ages. [11:40] here is some good news --> http://paste.ubuntu.com/1259806/ [11:40] davecheney: woo! [11:41] fwereade_: check it out! [11:41] and most of them worked ! [11:41] davecheney: that's almost like a real installation :-) [11:42] i don't think I can start any more machines [11:42] amazon will chide me [11:42] davecheney: the security description gets longer (still haven't got to the bit that was the whole point yet though!) http://paste.ubuntu.com/1259811/ [11:43] the buildbot charm failures, somethign happened with apt, it wasn't us [11:43] davecheney: nice environment ;) [11:43] % juju ssh ceph/0 -- -t 'less /var/log/juju/unit*' [11:43] nice [11:43] davecheney: what does -t do? [11:44] tells ssh'd to allocate a pty [11:44] normally if you do ssh host /some/command [11:44] no pty is allocated [11:44] davecheney: ah of course [11:44] which sucks [11:44] ssh cmd does a lot of things for you [11:44] davecheney: no, that's a good thing :-) [11:45] yeah, but then you have to figure out why it is how it is [11:45] davecheney: for most commands a pty gets in the way [11:45] true [11:45] so, the ceph charm is flat broke, not our fault [11:45] davecheney: paste the log? [11:46] two secs [11:46] most of them are missing debs [11:47] rogpeppe: http://paste.ubuntu.com/1259818/ << ceph [11:48] davecheney: i wonder what radosgw is and if it was installed by default before. [11:48] 2012/10/04 11:35:31 JUJU HOOK ERROR: command: cluster-init: 10.190.42.228:8091, [Errno 111] Connection refused [11:48] 2012/10/04 11:35:32 JUJU HOOK + /opt/couchbase/bin/couchbase-cli bucket-create -c 10.190.42.228:8091 -u Administrator -p administrator --bucket=jienaigo --bucket-type=couchbase --bucket-password= --bucket-ramsize=1607 --bucket-replica=1 [11:48] 2012/10/04 11:35:32 JUJU HOOK ERROR: command: bucket-create: 10.190.42.228:8091, [Errno 111] Connection refused [11:48] ^ couchbase [11:49] i wonder if some charms install the py juju tools by accident ... [11:51] rogpeppe: re your email [11:51] drop the bit about a machine cert [11:51] i thought we weren't/couldn't do that [11:52] davecheney: i think we have to do that, somehow. [11:52] davecheney: although we can't currently. [11:52] rogpeppe: yup, good point [11:54] davecheney: one way of doing it is to have a server that exchanges one-time tokens for certificate signing. [11:54] davecheney: then we can pass a one-time token into cloudinit [11:55] davecheney: the machine agent leverages that to get its own certificate signed. [11:56] the couchdb charm requires a ppa which is broken [11:56] davecheney: i wonder what your duplicate key insert problem was about [11:56] mongo bug [11:56] happens a lot [11:56] niemeyer has raised a bug upstream [11:57] couchdb [11:57] 2012/10/04 11:37:39 JUJU HOOK * Starting database server couchdb [11:57] 2012/10/04 11:37:39 JUJU HOOK ...done. [11:57] 2012/10/04 11:37:39 JUJU hook failed: exit status 1 [11:57] 2012/10/04 11:37:39 JUJU reading uniter state from disk... [11:57] nice [11:58] yeah, mongodb is... less than stellar. [11:58] Aram: it's better than zk though [11:59] 2012/10/04 11:37:32 JUJU HOOK + sed -e 's/^STARTDISTCC=.*/STARTDISTCC="true"/' -i /etc/default/distcc [11:59] the products are so different they can't be compared like that. [11:59] 2012/10/04 11:37:32 JUJU HOOK + '[' -x /usr/bin/open-port ']' [11:59] 2012/10/04 11:37:32 JUJU hook failed: exit status 1 [11:59] they solve a diferent problem. [11:59] ^ hard coded tools [11:59] Aram: you're probably right. [12:00] Aram: from my brief look at the mongo source earlier today, i wasn't enormously overjoyed. [12:00] oh, it's bad, I had to look through it when solving various quirks and bugs. [12:01] 2012/10/04 11:39:15 JUJU HOOK ldconfig deferred processing now taking place [12:01] 2012/10/04 11:39:17 JUJU HOOK install: cannot stat `files/php/php_conf.d_apc.ini': No such file or directory [12:01] 2012/10/04 11:39:17 JUJU hook failed: exit status 1 [12:01] 2012/10/04 11:39:17 JUJU reading uniter state from disk... [12:01] ^ drupal6 expects a file not owned by a deb that it installed [12:01] Aram: i'd forgotten that people still like ifdefs. ugh. [12:02] so, short summary, 19 charms, 9 working [12:02] none are our fault (directly) [12:02] open source stuff is plagued by ifdefs. [12:03] someone please pass on to mramm and gustavo [12:03] i [12:03] i'm off to bed [12:04] davecheney: good work, man [12:04] davecheney: enjoy your rest [12:04] no, congratulations to all of you [12:04] with the exceptoin of juju-log -l [12:04] davecheney: what was the issue with that? [12:04] i haven't found a charm that is broken because we are incompatible with py juju [12:04] rogpeppe: we didn't support -l $LEVEL [12:05] davecheney: ah [12:05] davecheney: well, gnuflag was broken too... [12:05] rogpeppe: https://codereview.appspot.com/6584069 [12:06] % juju destroy-environment [12:06] error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. (SignatureDoesNotMatch) [12:06] FUCK YOU AWS [12:06] not tonight [12:06] not with 20 machines running [12:06] it always does this when I have more than a few machines running [12:07] davecheney: hmm, that might be our bug, i suppose [12:08] davecheney: just got to the aws console :-) [12:08] s/got/go/ [12:09] right, i have ceased hemorhaging money [12:09] davecheney: i hope you claim it back on expenses [12:09] davecheney: (i've been a bit crap at doing that recently) [12:09] night all [12:09] davecheney: but i had $200 bill last month, so it's worth doing [12:33] fwereade_: howdy, can I bug you for a bit please? got a vexing problem with juju [12:45] bigjools, heyhey [12:45] how's it going? [12:45] bigjools, ah, not bad thanks, not sure if I can *actually* remember anything about python but I'm game for a try [12:45] ah you're a Goer [12:46] bigjools, that's what they tell me [12:46]