[01:37] <_mup_> ensemble/expose-cleanup r337 committed by jim.baker@canonical.com
[01:37] <_mup_> Test new securitygroup function
[02:23] <_mup_> ensemble/expose-cleanup r338 committed by jim.baker@canonical.com
[02:23] <_mup_> Merged trunk & resolved conflicts
[03:03] <_mup_> ensemble/expose-cleanup r339 committed by jim.baker@canonical.com
[03:03] <_mup_> Fixed merge issues
[13:40] <niemeyer> Greetings!
[13:55] <hazmat> niemeyer, greetings
[13:57] <hazmat> i do find the abbreviations to make reading the go branches a bit harder to read
[13:57] <hazmat> it would help if the abbreviations where at least documented once
[13:58] <hazmat> +type ifaceExpC struct {
[13:58] <hazmat> is interface expander c?
[13:59] <hazmat> the variable declaration for the type uses stringC which i assume to be string constant.. but thats a different usage than the preceding statement that uses C for ?
[14:09] <niemeyer> hazmat: Checker
[14:09] <niemeyer> hazmat: That convention was used on every single type within schema.go
[14:10] <niemeyer> hazmat: I can rename it if it feels so tricky
[14:10] <niemeyer> hazmat: It's akin to _state, though
[14:10] <hazmat> niemeyer, its just that without that context it becomes harder to understand, i don't know that it needs renaming but a comment would be useful
[14:11] <niemeyer> hazmat: I'm happy to rename or to add the comment
[14:11] <niemeyer> hazmat: // The *C convention is used within schema.go for Checkers
[14:11] <niemeyer> ?
[14:12] <hazmat> niemeyer, what's schema.M?
[14:13] <niemeyer> hazmat: That's documented
[14:13] <niemeyer> hazmat: 
[14:13] <niemeyer> / All map types used in the schema package are of type M.
[14:13] <niemeyer> type M map[interface{}]interface{}
[14:13] <hazmat> niemeyer, ic
[14:14] <niemeyer> hazmat: This is just a handy alias to this type that allows building maps as
[14:14] <niemeyer> hazmat: schema.M{"a": 1}, etc
[14:14] <niemeyer> hazmat: map[interface{}]interface{}{"a": 1} would work, but.. :)
[14:15] <hazmat> indeeds its an improvement
[14:15] <niemeyer> hazmat: Yeah, this became clear in tests
[14:15] <niemeyer> hazmat: I wasn't using it before, but when I started to test the results in the formula work, got boring
[14:16] <hazmat> niemeyer, is this sort of multi-file lookup for abbreviations common in go?
[14:16] <niemeyer> hazmat: multi-file lookup?
[14:17] <hazmat> niemeyer, abbreviation used in one place, but needs to be tracked down to a different file to understand naming?
[14:17] <niemeyer> hazmat: Not a different file.. this is a package, and the type is defined and has meaning within the package itself
[14:18] <niemeyer> hazmat: schema.M is used for schemas only
[14:18] <niemeyer> hazmat: Just for comprehension, note that the file could be named foo.go
[14:18] <niemeyer> hazmat: The "schema" in "schema.M" comes from the package name, not the file name
[14:19] <niemeyer> hazmat: In other words, I wouldn't use schema.M in things not related to the schema itself
[14:20] <hazmat> niemeyer, their not different packages? one is ensemble/go/schema and ensemble/go/formula? 
[14:20] <niemeyer> hazmat: These are different packages, yeah
[14:22] <niemeyer> hazmat: In these terms it's quite similar to Python.. if you use ensemble.environment.Environment, you may go to ensemble.environment to see what's that
[14:22] <hazmat> niemeyer, so in this case we have one package's abbreviation being used in another package, the question is if that sort of usage is common?
[14:23] <niemeyer> hazmat: The distinction, as I was pointing out, is that unlike Python packages are multi-file
[14:23] <niemeyer> hazmat: It's not an abbreviation.. it's a type
[14:24] <niemeyer> hazmat: and the reason why the type is used in a different file is because we're using _schema_ in the other package
[14:24] <hazmat> niemeyer, the name itself is an abbreviation, its the name for a type
[14:24] <niemeyer> hazmat: It's not an abbreviation, it's a real type
[14:24] <niemeyer> hazmat: reflect.TypeOf(schema.M{}) != reflect.TypeOf(map[interface{}]interface{}{})
[14:25] <hazmat> "M" is not an abbreviation for MapType?
[14:25] <niemeyer> hazmat: Sorry, I see what you mean
[14:25] <niemeyer> hazmat: Yes
[14:25] <niemeyer> hazmat: Let me try to drive a parallel to show you what I mean
[14:26] <hazmat> i think this would be much readable if we avoided cross package abbreviations, i'm just trying to understand if this usage is common in go
[14:28] <niemeyer> hazmat: Alright.. I'll put map[interface{}]interface{}{} back then
[14:29] <hazmat> niemeyer, i don't think thats any better, but is there a better name for the type?
[14:29] <niemeyer> hazmat: schema.M?
[14:29] <niemeyer> hazmat: That was my take on it.. makes building the map easier
[14:29] <niemeyer> hazmat: I've used the same convention in other marshalling modules
[14:30] <niemeyer> hazmat: But I won't bikeshed on that.. if you have other suggestions, I can change
[14:34] <hazmat> niemeyer, schema.SchemaMap ?
[14:34] <niemeyer> hazmat: Heh
[14:34] <niemeyer> hazmat: Duplicating the schema name won't make it handier or more readable in any way
[14:35] <niemeyer> hazmat: Also, what's the difference between schema.SchemaMap and schema.Map?
[14:36] <niemeyer> hazmat: I can rename schema.Map to ...
[14:36] <hazmat> MapChecker
[14:36] <niemeyer> hazmat: schema.GenericMap
[14:36] <hazmat> yeah.. that works as well, and M->Map
[14:36] <niemeyer> hazmat: I can do that as well, as long as you never complain again about naming things with _state.. ;-D
[14:37] <niemeyer> hazmat: Imagine how the schema definition would look like..
[14:37] <hazmat> niemeyer, sounds good as long as there no complaints about a -(_state) branch ;-)
[14:37] <niemeyer> hazmat: schema.MapChecker(schema.StringChecker(), schema.InterfaceChecker(....))
[14:38] <hazmat> yeah.. that's lame
[14:38] <niemeyer> hazmat: Ok, I'll do the name change
[14:38] <niemeyer> hazmat: But in a follow up branch if that's alright
[14:38] <niemeyer> Since it'll touch everything else
[14:39] <hazmat> niemeyer, feel to change it a different name, i just think having a common entry point into a package be an abbreviation also feels like a violation of encapsulation
[14:40] <niemeyer> hazmat: I don't understand how that could be the case
[14:40] <niemeyer> hazmat: There's zero encapsulation involved
[14:40] <hazmat> because it tends to require reading the relevant implementation to understand its use
[14:40] <niemeyer> hazmat: The only reason we use schema.M in formula is because we're dealing with the schema there
[14:42] <niemeyer> hazmat: I also don't think this is true.. you don't have to read the implementation to understand its use.. you have to read the documentation, which is a pretty general problem 
[14:42] <niemeyer> hazmat: It's like saying we need to read Python's json.dumps prototype to understand how to call it
[14:43] <hazmat> what it is and what its doing are fairly obvious by name alone, schema.M is not 
[14:43] <niemeyer> hazmat: Agreed.. it's a convention to be aware of
[14:43] <hazmat> so if your reading code that uses both, one is pretty clear without additional context and one is not
[14:43] <niemeyer> hazmat: The same convention is used in gobson and goyaml
[14:44] <niemeyer> Actually, I lie.. gobson only it seems
[14:45] <niemeyer> Anyway.. I was explaining the POV
[14:45] <niemeyer> I'm fine with renaming it
[14:48] <niemeyer> hazmat: and answering your very first and simple question, no, this is not a general convention, and if it becomes one it's my fault ;-0
[14:48] <niemeyer> ;-)
[14:50] <hazmat> niemeyer, thanks, good to know
[14:52] <_mup_> ensemble/go-formulas r8 committed by gustavo@niemeyer.net
[14:52] <_mup_> Renamed schema.M/L to schema.Map/ListType
[14:55] <_mup_> Bug #836753 was filed: schema.M/L are not readable <Ensemble:In Progress by niemeyer> < https://launchpad.net/bugs/836753 >
[15:16] <niemeyer> fwereade: Gosh.. is all of that really needed for adding _http auth_?
[15:38] <SpamapS> http://ec2-107-20-64-136.compute-1.amazonaws.com:8080/job/tested%20PPA/42/console
[15:39] <SpamapS> So, other than the failure at the end because I forgot to quote the 'cd' ... first iteration of integration tests went well.
[15:39] <SpamapS> lp:~clint-fewbar/ensemble/jenkins-test-suite has the code under misc/jenkins
[15:40] <SpamapS> I need to break it up into 3 or 4 bash scripts.. but the interesting parts are wait4state and unit2machine .. which I think should become subcommands or at least helper args to some of the commands
[15:40] <SpamapS> wait4state especially would be best written as a zookeeper watch
[15:45] <SpamapS> One thing I was struggling with was whether there was a way to address yaml nodes similar to xpath for xml.
[15:46] <SpamapS> The way that wait4state works is pretty weird and crazy.. but it works for this specific job
[15:47] <jimbaker> SpamapS, taking a look
[15:48] <jimbaker> SpamapS, we should definitely make wait4state easier to do. as you said, it really should be using zk watch support
[15:50] <jimbaker> SpamapS, re some sort of xpath - really this is some sort of query syntax for working w/ python collections. there must be a number of packages out there to do just that
[15:51] <jimbaker> biab, i have to go to a quick doctor's appt
[15:51]  * SpamapS will have to vacate his house and go work at a co-working space in about an hour while the exterminators work. UGH
[15:54] <fwereade> niemeyer: sorry, had to pop out -- and, as far as I can tell, yes it is
[15:54] <fwereade> niemeyer: I looked pretty hard for alternatives I could piggyback on, but nothing seemed cleanly doable
[15:58] <SpamapS> fwereade: were you at least able to configure the webdav service to auth against cobbler, so the auth credentials can remain the same?
[16:01] <fwereade> SpamapS: hm, I didn't think to try that -- if you don't specif separate cobbler credentials, it falls back to the orchestra ones
[16:02] <fwereade> SpamapS: same model as the configurable storage-url -- if you don't include one, it assumes http://%(orchestra-server)s/webdav
[16:03] <fwereade> SpamapS: but I have no idea how to set up cobbler and webdav with a shared auth source
[16:04] <SpamapS> fwereade: I forget what our default is, but https://fedorahosted.org/cobbler/wiki/CustomizableAuthentication has a run down on the auth methods possible.. authn_passthru would be the simplest way to get this done.
[16:04] <SpamapS> fwereade: actually authn_configfile will also work because we can point the webdav service at the same config
[16:05] <fwereade> SpamapS: easier than I thought then :)
[16:08] <fwereade> niemeyer: https://code.launchpad.net/~fwereade/ensemble/webdav-unicode-paths/+merge/72579 -- did you mean to mark that as approved for merge, or should I get someone else to give it a look?
[16:13] <SpamapS> fwereade: how well does urllib2 handle https btw?
[16:13] <SpamapS> fwereade: is it going to squawk because of self signed certs or anything like that?
[16:14] <fwereade> SpamapS: hm, I have no idea
[16:15] <SpamapS> fwereade: I'd say that initially it *should* complain with invalid certs, but that we should make it easy to turn that off given Orchestra's "backoffice" role.
[16:15] <kirkland> SpamapS: certs for what?
[16:15] <fwereade> SpamapS: wounds reasonable, but I'm not sure what context you're asking about
[16:15] <SpamapS> kirkland: for the webdav file storage
[16:16] <kirkland> SpamapS: RoAkSoAx is already using orchestra + preseed to install SSL certs in deployed guests
[16:16] <kirkland> SpamapS: for SSL protected rsyslog
[16:16] <kirkland> SpamapS: you should be able to just force the webdav to use those same certs
[16:16] <SpamapS> Seeing as formulas run as root and have no cryptographic verification built in.. its fairly important to consider the security implications they carry.
[16:16] <kirkland> SpamapS: or use the same mechanism to install additional certs
[16:16] <SpamapS> Yeah that sounds good.
[16:17] <SpamapS> Give the admin a way to easily pull that in to his local cert store.
[16:18] <SpamapS> I think by default it should be easy and just work w/o intervention. I also think that if the admin needs/wants to crank up the security, they should be able to.
[16:19] <SpamapS> (And I'm sure some people will argue that by default it should be high security, and admins should be able to turn off security)
[16:24] <niemeyer> adam_g, negronjl, lynxman: ping
[16:25] <negronjl> niemeyer: pong
[16:25] <niemeyer> negronjl: Hey man
[16:25] <negronjl> niemeyer:  Happy Monday ( if there is such a thing ) :)
[16:26] <niemeyer> negronjl: Was just wondering about how you guys feel about how the package and code stability has been progressing
[16:26] <niemeyer> negronjl: Are you happy with it in general, or do you feel we've been breaking things too often somehow?
[16:27] <negronjl> niemeyer:  Althought I am very happy with the project in general, I would feel more confortable with some stable version of ensemble that, may not have all of the latest and greates features, but, it is stable.
[16:27] <niemeyer> negronjl: That's a bit different from the actual question.. I would like to have a stable version too :-)
[16:27] <negronjl> niemeyer:  I found myself with a very close call on NoSQL conference where I was scheduled to talk about Ensemble but, ensemble was broken
[16:28] <niemeyer> negronjl: The question is whether we've been breaking things often
[16:28] <negronjl> niemeyer:  not often no.
[16:28] <niemeyer> negronjl: Ok, cool.. I understand the desire for stability, and we're doing things to improve that
[16:28] <niemeyer> negronjl: I just want to make sure I understand the feeling
[16:30] <negronjl> niemeyer:  To clarify my earlier point.  It would help if there is a version in ensemble that would be considered stable enough and we can somehow pin it ( a /stable ppa or so ) until another version becomes stable enough and can replace it.  This would alleviate some of the issues that I experienced during NoSQL last week.
[16:30] <negronjl> niemeyer:  Just my thoughts.
[16:31] <niemeyer> negronjl: Sounds good.. I hope this is the version in Ubuntu
[16:36]  * SpamapS too
[16:37] <SpamapS> negronjl: to niemeyer's point, the version in Ubuntu is actually pretty stable. :)
[16:53] <jcastro> yes that's what I was using and it was working fine
[17:06] <adam_g> assuming whats in the archive is stable, it would good if nodes install straight from the archive by default, with the option to install a bleeding edge versin from ppa:ensemble/ppa (default now). 
[17:06] <adam_g> niemeyer: ^
[17:06] <niemeyer> fwereade: I did, sorry
[17:06] <fwereade> niemeyer: thanks, no worries
[17:06] <niemeyer> fwereade: Marked it now
[17:06] <niemeyer> fwereade: Sad about the auth complexity :(
[17:06] <fwereade> niemeyer: cheers
[17:06] <fwereade> niemeyer: me too :(
[17:07] <niemeyer> fwereade: It's _auth_, for dev's sake
[17:07] <fwereade> niemeyer: I came to terms with it eventually, but... quite
[17:08] <ahasenack> adam_g: you mean the nodes install something else? Like, trunk?
[17:09] <adam_g> ahasenack: i just bootstrapped a node using 0.5+bzr330-0ensemble1, and userdata sets up access to the ppa instead of installing out of the archive
[17:10] <ahasenack> adam_g: is that ppa "bleeding edge"?
[17:12] <adam_g> ahasenack: honestly do not know how/when/why packages get built and pushed there, but installing from there is where ive personally hit show stopper bugs.
[17:12] <hazmat> ahasenack, the ppa is a trunk daily build
[17:12] <ahasenack> ok
[17:15] <niemeyer> fwereade: Can't we use the same technique twisted.web._auth.digest uses, and rely on twisted.creds.credentials?
[17:16] <niemeyer> fwereade: It feels like you're reimplementing it
[17:19] <fwereade> niemeyer: hm, I was put off by the privateness of the module, and assumed that something would end up breaking if I used a _kindaprivate module
[17:19] <niemeyer> fwereade: The _auth module has pretty much nothings inside it
[17:19] <fwereade> hm, I thought that was where the meat of it was
[17:19] <fwereade> niemeyer: let me check again
[17:19] <niemeyer> fwereade: You're allowed to look inside private modules! ;-0
[17:19] <niemeyer> :-)
[17:20] <hazmat> "_auth" ;-)
[17:20] <hazmat> there's a bug open for it
[17:20] <hazmat> it might be nice to ask therve or free
[17:20] <hazmat> or on #twisted
[17:21] <niemeyer> hazmat: There's nothing interesting inside it
[17:21] <fwereade> niemeyer: well, crap, somehow I missed cred.credentials
[17:21] <niemeyer> hazmat: Besides a pointer to twisted.cred.credentials
[17:21] <fwereade> niemeyer: I, er, did look :/
[17:21] <fwereade> niemeyer: maybe I stopped looking at that point though :(
[17:22] <niemeyer> fwereade: Don't worry about it
[17:23] <niemeyer> fwereade: Maybe it's not even useful.. but certainly has very similar logic, so worth checking
[17:23] <fwereade> niemeyer: definitely so, I'll give it a look
[17:23] <niemeyer> fwereade: In either case, the auth logic itself should be factored out of FileStorage
[17:23] <niemeyer> fwereade: and the file
[17:23] <niemeyer> fwereade: and put into its own module, and tested in isolation
[17:23] <fwereade> niemeyer: it's not immediately apparent how to hook it up, but you're right, it's doing just the right thing
[17:23] <niemeyer> fwereade: This is a major missing piece in the http library, rather than part of the FileStorage problem
[17:24] <fwereade> niemeyer: it didn't feel quite big enough to do that at the time, but you're right
[17:24] <fwereade> cheers
[17:24] <hazmat> as another way of getting basic auth into twisted web client ... http://code.activestate.com/recipes/525493-simple-crawler-using-twisted/
[17:25] <niemeyer> hazmat: Not sure about what's the point there?
[17:26] <hazmat> its a much simpler layering of basic auth onto twisted web client
[17:26] <fwereade> hazmat: surely basic auth is barely worth bothering with?
[17:27] <niemeyer> hazmat: Yeah, I'm trying to understand what's your underlying suggestion.. is it that we should use client.HTTPDownload?
[17:27] <niemeyer> fwereade: Can we?
[17:28] <fwereade> niemeyer: use basic auth?
[17:28] <hazmat> fwereade, that's fair, its not secure over http, but over https its fine
[17:28] <niemeyer> hazmat: Use HTTPDownloader
[17:28] <niemeyer> Erm
[17:28] <niemeyer> fwereade: Use HTTPDownloader
[17:29] <fwereade> niemeyer: looking at it, we probably can, the only reason I didn't use getPage is because apparently I can't get at the reply headers
[17:30] <niemeyer> fwereade: gotHeaders?
[17:30] <fwereade> niemeyer: yes indeed, that's on HTTPDownloader
[17:30] <fwereade> niemeyer: so, looking at it, we probably can
[17:31] <fwereade> niemeyer: I saw things saying "if getPage isn't good enough, use Agent"... so I used Agent :/
[17:32] <fwereade> niemeyer: sounds like it's a solid "needs fixing" with a side helping of "lrn2google"
[17:32] <fwereade> niemeyer: at least the code'll end up smaller :)
[17:32] <niemeyer> fwereade: Both approaches are surprisingly involved, to be honest
[17:32] <fwereade> niemeyer: the creds stuff looks like it might be quite helpful, I'll look into that
[17:33] <niemeyer> fwereade: The approach referenced by hazmat seems slightly  better because it's more encapsulated than the one in your branch
[17:33] <fwereade> niemeyer: there are approaches I didn't consider, so I should definitely look into them
[17:33] <niemeyer> fwereade: It should be possible to merge your auth with the simpler approach
[17:33] <niemeyer> hazmat: Which is why I was asking what was the underlying suggestion
[17:33] <fwereade> niemeyer: ah, I see
[17:34] <fwereade> niemeyer: I think it's worth spending a little while trying to hook up the cred.credentials solution though, that would be best
[17:34] <niemeyer> fwereade: Sounds good.. I'm happy with any of these options
[17:34] <fwereade> niemeyer: anyway, I'm sorry to cut you off, but the calls for suppertime are becoming quite insistent
[17:35] <fwereade> niemeyer: I'll try to have something better tomorrow
[17:35] <niemeyer> fwereade: The only thing that we should necessarily do is to factor that problem auth of the FileStorage
[17:35] <niemeyer> s/auth of/out of/
[17:35] <niemeyer> Contextual typo :)
[17:47] <niemeyer> SpamapS: This is sitting on our review queue for 2 weeks untouched: https://code.launchpad.net/~clint-fewbar/txaws/fix-s3-port/+merge/71289
[17:48] <niemeyer> SpamapS: Can you please address the points for getting it merged, or put it back in Work In Progress otherwise?
[17:50] <SpamapS> niemeyer: I started working on that about 30 minutes ago actually. :)
[17:50] <niemeyer> SpamapS: WOohay
[17:50] <SpamapS> niemeyer: just writing some test code. I addressed the first point before Robert made it. :)
[17:50] <niemeyer> SpamapS: Great timing then :)
[17:51] <SpamapS> niemeyer: It seems that all of the tests in txaws are functional tests.. I see no mocking for AWS
[17:53] <niemeyer> SpamapS: I've tweaked tests there before
[17:53] <niemeyer> SpamapS: There are unittests
[18:12] <SpamapS> jimbaker: so.. security group cleanup...
[18:13] <jimbaker> SpamapS, yes, it's about ready to land
[18:13] <jimbaker> (in expose-cleanup branch)
[18:14] <SpamapS> jimbaker: http://ec2-107-20-64-136.compute-1.amazonaws.com:8080/job/tested%20PPA/43/console
[18:14] <SpamapS> jimbaker: is that a bug?
[18:15] <jimbaker> SpamapS, expose-cleanup fixes this issue
[18:16] <SpamapS> jimbaker: so I have to manually delete all the groups before I can re-use the environment?
[18:16] <jimbaker> what we see here is generally seen when doing an immediate bootstrap after shutdown - the instance originally associated with this group is still around (in 'shutting-down' state) and so cannot be deleted
[18:16] <jimbaker> SpamapS, once expose-cleanup lands in trunk, you won't have to anything
[18:16] <jimbaker> the other possibility is to just wait sufficiently long
[18:17] <SpamapS> jimbaker: the shutdown was 24 hours ago
[18:17] <m_3> I just wait it out
[18:17] <m_3> wow!
[18:17] <jimbaker> SpamapS, something is holding onto that security group then
[18:17] <m_3> usually cleans itself up after a few minutes
[18:18] <jimbaker> SpamapS, so expose-cleanup simply polls the instance state to see if they're terminated. if so, it can delete the associated security groups
[18:18] <jimbaker> or i should say, this polling is done in a revised shutdown_machines in ensemble.providers.ec2
[18:19] <jimbaker> SpamapS, please check whether or not you can manually delete that security group
[18:19] <jimbaker> using the AWS console
[18:19] <jimbaker> or tool of choice
[18:19] <SpamapS> how can I see what instances are in a group w/ the cmdline tools?
[18:20] <jimbaker> SpamapS, i don't believe that's directly possible, but i could be wrong
[18:20] <jimbaker> i know it's in the reservation part of the instance description
[18:21] <SpamapS> hmm must have been transient
[18:21] <_mup_> ensemble/expose-cleanup r340 committed by jim.baker@canonical.com
[18:21] <_mup_> Merged trunk
[18:22] <SpamapS> its odd, my testing does shutdown+bootstrap pretty constantly.. maybe 10 seconds of lag between those two steps
[18:23] <m_3> 10s is rarely enough time... I usually let it sit for a minute between shutdown and the next bootstrap
[18:23] <jimbaker> SpamapS, that testing regimen is not going to work well until this cleanup occurs - it needs at least a couple of minutes in between for ec2 to properly cleanup
[18:23] <SpamapS> jimbaker: but it has worked maybe 20 times so far
[18:23]  * m_3 surprised
[18:24] <jimbaker> SpamapS, i'm also surprised
[18:24] <jimbaker> :)
[18:24] <jimbaker> sounds like ec2 was especially fast in its cleanup
[18:24] <SpamapS> I think the way I'll make it work is just to put each target in its own "environment"
[18:24] <jimbaker> SpamapS, that makes much more sense
[18:24] <m_3> good test of other things too :)
[18:25] <SpamapS> I need to find a way to generate and dump an environments.yaml into the test chroots anyway. :p
[18:25] <jimbaker> that will definitely isolate
[18:34] <_mup_> ensemble/expose-cleanup r341 committed by jim.baker@canonical.com
[18:34] <_mup_> PEP8 & PyFlakes
[18:34] <lynxman> niemeyer: pong, sorry it was bank holiday here in the UK
[18:35] <niemeyer> lynxman: Hey
[18:35] <niemeyer> lynxman: No worries
[18:35] <jimbaker> hazmat, i briefly looked at autodoc, but there seems to be some problem in our docstrings which is causing it to crash when i run  the sphinx build
[18:35] <niemeyer> lynxman: Was just wondering about how Ensemble has been treating you.. I've heard a few claims that you guys were facing problems with us introducing new bugs daily, but it sounds like it wasn't really the case
[18:36] <lynxman> niemeyer: we had some stability issues in formulas indeed, but as soon as that happens we either ask here or open bug reports
[18:37] <niemeyer> lynxman: What kind of stability problem was that?  Just want to make sure I'm aware
[18:37] <hazmat> jimbaker, any tracebacks?
[18:37] <jimbaker> hazmat, sure, let me paste
[18:38] <jimbaker> hazmat, actually hold on that - i was going against my expose-cleanup branch, but i didn't want it to get mired in a bigger problem, so i reverted my sphinx conf changes
[18:39] <jimbaker> hazmat, just fairly simple - update conf.py, plus __init__.py (or use some third party gen scripts to do the walking)
[18:40] <lynxman> niemeyer: in my case was just trying to make Ensemble work against openstack, but hazmat helped a lot on it so I didn't need to file any bugs
[18:40] <jimbaker> hazmat, docutils was crashing in its statemachine.py code w/ a recursion limit issue, so something was indigestible
[18:40] <niemeyer> lynxman: Ahh, ok
[18:40] <lynxman> niemeyer: negronjl and iamfuzz are working on the cloudfoundry formulas which encountered more issues, which were raised on time
[18:40] <niemeyer> lynxman: Well.. OpenStack support is a minefield at the moment
[18:40] <SpamapS> so.. does ensemble make use of blueprints at all?
[18:40] <jimbaker> hazmat, anyway, i will recreate in a bit
[18:40] <hazmat> SpamapS, not really bcsaller1 and i where experimenting with them
[18:41] <hazmat> but pretty much zero
[18:41] <lynxman> niemeyer: it kinda is, but so far works pretty okay :)
[18:41] <hazmat> niemeyer, its not that bad
[18:41] <niemeyer> lynxman: That's good to know :-)
[18:41] <SpamapS> I was thinking of a suite of tools to make ensemble more command-line-useful ... waiting for states, outputting data about machines/services that is not yaml.. and exit codes to differentiate permanent and transient errrors.
[18:41] <lynxman> niemeyer: as said, hazmat helped a great deal
[18:42] <hazmat> niemeyer, the ec2 compatibility issue they have marked as a bug, although its unclear when they'll fix it.. but just as far as impact to the ec2 provider, its like a dozen lines
[18:42] <jimbaker> SpamapS, maybe "ensemble watch"
[18:42] <niemeyer> hazmat: Yeah, I'm more concerned in conceptual terms than in terms of the particular issue
[18:42] <hazmat> not including tests of course
[18:42] <jimbaker> SpamapS, another thing that might be useful is to have a wait-for ensemble is ready command
[18:42] <niemeyer> hazmat: It doesn't feel like there's serious interest in preserving the compatibility
[18:42] <hazmat> niemeyer, so they will fix ec2 bugs, but not s3 is what i got out of it
[18:43] <hazmat> niemeyer, agreed about the overall compatibility.. they have a layer, it has tests, they won't break it, but if something doesn't work, we'll probably need to make it do so
[18:43] <jimbaker> SpamapS, there are now at least two places that do that last bit for the ec2 provider (via polling ec2_describe_instances); in particular ftests have this functionality
[18:43] <SpamapS> jimbaker: right, there's really a swath of changes that need a binding purpose.. this is what blueprints are for
[18:44] <jimbaker> m_3, i would assume this wait-for-ensemble command would be useful for you too
[18:44] <hazmat> niemeyer, they have an ec2 compatibility bug for the security group work, that ensemble needs, its still marked as a bug as opposed to the s3 one.
[18:44] <hazmat> https://bugs.launchpad.net/nova/+bug/829609
[18:44] <_mup_> Bug #829609: EC2 compatibility describe security group returns erroneous value for group ip permissions <Ensemble:In Progress by hazmat> <OpenStack Compute (nova):Confirmed> < https://launchpad.net/bugs/829609 >
[18:44] <SpamapS> jimbaker: actually I'd say that zookeeper is the system of record... so polling zk is the only sensible thing to do.
[18:44] <jimbaker> SpamapS, also in ftests
[18:45] <jimbaker> specifically a watch for our ZK nodes created by ensemble
[18:45] <niemeyer> hazmat: Good to know about their feeling about that, thanks
[18:46] <jimbaker> let me push what i have for that - not yet sufficiently robust, but at least it usually works
[18:46] <hazmat> depending on the command probably an agent watch for unit or machine
[18:47] <_mup_> ensemble/fix-functional-testing r334 committed by jim.baker@canonical.com
[18:47] <_mup_> Better branch identification
[18:49] <jimbaker> SpamapS, i don't like ensemble.ftests.test_ec2_provider.EC2MachineTest.assert_node yet, but it's on the right path
[18:49] <jimbaker> note that we still need to poll ec2 first, then *watch* zk
[18:50] <SpamapS> jimbaker: well my point is that this is something to plan at a larger scale than solving each of these issues one by one.
[18:50] <SpamapS> jimbaker: there's a real use case for using ensemble in jenkins for instance.
[18:51] <SpamapS> jimbaker: you only need to ask the provider for the seed so you can find zookeeper. After that you wait for zookeeper, then you listen to zookeeper.
[18:51] <jimbaker> SpamapS, agreed with that approach
[18:56] <m_3> jimbaker: yes, something that gives us that functionality
[18:58] <SpamapS> ugh these test cases are really oddly organized
[19:03] <hazmat> SpamapS, yeah.. you need an output capture from openstack effectively
[19:03] <SpamapS> hazmat: for what I did, I just need to verify that the port is put back into the generated urls actually. :)
[19:04] <hazmat> SpamapS, that sounds much nicer ;-)
[19:04] <hazmat> fixing the security group parsing needs a response output capture to verify the implementation
[19:05] <SpamapS> hazmat: right, that one at least is being discussed upstream as being fixed .. tho probably not in diablo. :(
[19:13] <SpamapS> hrm.. txaws's test cases that use venusian fail when it is installed .. looks like they were written to a different venusian api
[19:14] <SpamapS> http://paste.ubuntu.com/677409/
[19:31] <_mup_> ensemble/trunk r335 committed by jim.baker@canonical.com
[19:31] <_mup_> merge expose-cleanup [r=niemeyer,fwereade][f=824219]
[19:31] <_mup_> For EC2: shutting down machines removes machine security groups;
[19:31] <_mup_> destroying the environment removes the environment security group.
[19:32] <niemeyer> jimbaker: Just double checking: have you tested it with a real interaction on _trunk_?
[19:33] <jimbaker> niemeyer, i will double check it one more time - it has worked every time, but i will check as pushed
[19:33] <niemeyer> jimbaker: Awesome, thanks a lot
[19:33] <ahasenack> SpamapS: txaws needs venusian 1.0a1 at least, it was just released
[19:33] <niemeyer> SpamapS: your opinion on this would also be welcome
[19:36] <jimbaker> niemeyer, looks good as pushed with bouncing repeatedly (ensemble bootstrap && ensemble shutdown && ...), this was the pathological case of what the cleanup now fixes
[19:38] <jimbaker> SpamapS, m_3 - but nothing like real usage. so please tell me if this doesn't fix the problem you've been seeing
[19:38] <m_3> jimbaker: will do
[19:39] <jimbaker> m_3, thanks!
[19:39] <SpamapS> so, I just realized a big problem
[19:40] <SpamapS> Ensemble by default pulls itself on deployed machines from the PPA
[19:40] <SpamapS> This means that if ensemble is being driven from, say, mac ports.. at revision 295 .. it may deploy incompatible versions of ensemble onto the machines.
[19:41] <SpamapS> I *think* have an answer for this.. but it may be a little bit weird...
[19:42] <jimbaker> SpamapS, there's an outstanding bug about this that i started looking into
[19:42] <jimbaker> bug 828147
[19:42] <_mup_> Bug #828147: Ensemble branch option needs to allow for distro pkg, ppa, and source branch install <Ensemble:New for jimbaker> < https://launchpad.net/bugs/828147 >
[19:43] <jimbaker> SpamapS, it's not quite what you described just now, but it does allow for precise specification of what it should be on the target environment
[19:45] <SpamapS> jimbaker: it has to automatically choose itself though
[19:45] <jimbaker> SpamapS, exactly, it would be nice for an automagic option in ensemble-origin
[19:45] <SpamapS> I can't think of any time where I want to, by default, deploy something other than what I'm running.
[19:45] <SpamapS> unless we guarantee backward compatibility
[19:46] <jimbaker> SpamapS, we actually are planning to do just that for client access
[19:46] <hazmat> jimbaker, we're not guaranteeing backward compatibility
[19:46] <jimbaker> there's now a protocol version that's stored in zk (in the topology node to be precise); it's checked for compatibility on every operation working w/ topology (which is most of them)
[19:46] <hazmat> we're just detecting a conflict
[19:47] <jimbaker> hazmat, sorry i've reinterpreted the problem so it works like you said :)
[19:47] <jimbaker> we only guarantee backwards compatibility if the protocol version is compatible
[19:48] <jimbaker> this still allows for some wiggle room
[19:49] <hazmat> SpamapS, the txaws tests run fine on natty fwiw
[19:49] <SpamapS> hazmat: yeah the tests run fine for me w/o venusian
[19:49] <SpamapS> on oneiric
[19:51]  * ahasenack is invisible
[19:54] <hazmat> SpamapS, ping me when you've got tests pushed, i'll merge to my updated my branch (~hazmat/txaws/fix-s3-port-and-bucket-op/) and push for review
[20:00] <niemeyer> hazmat: Does that look good:
[20:00] <niemeyer> c.Assert(meta.Provides["server"], Equals, formula.Relation{Interface: "mysql"})
[20:00] <niemeyer> ?
[20:01] <hazmat> niemeyer, that looks great
[20:01] <niemeyer> hazmat: Awesome, I'll move forward with it then, thanks
[20:03] <SpamapS> hazmat: I pushed over an hour ago
[20:03] <hazmat> SpamapS, awesome, good to know, i'll merge and push the additional fixes for review, thanks
[20:03] <OldSchool> does ensemble have to be used w/ EC2 or can you use it with, say, an in-house ESXi environment?
[20:04] <OldSchool> ah, wait, found the FAQs finally
[20:04] <hazmat> OldSchool, no esxi support at the moment, ensemble supports pluggable providers, and we're working on some additional ones at the moment (physical machine, openstack, local dev), but no plans atm for esxi 
[20:04] <SpamapS> hazmat: no, thank *you* :)
[20:06] <OldSchool> thank you
[20:06] <hazmat> OldSchool, it shouldn't be too bad to add one, implementing a provider has gotten cleaned up quite a bit, and made easier, but its probably going to need community contribution.. if your interested folks around here be happy to help you out
[20:07] <OldSchool> sweet, I just heard about it from a Canonical rep on Friday after doing some tinkering w/ Puppet and Landscape
[20:08] <OldSchool> I only admin 4-5 Ubuntu boxes at work and they're pretty self-sufficient, I'm just always interested in automation, more curious for future projects
[20:12] <hazmat> OldSchool, if you want to give ensemble a test drives, its probably to use ec2.. we'll be supporting a local development mode in the 11.10 release for ubuntu laptop/desktops which will also allow some experimentation and test driving
[20:13] <hazmat> s/probably/probably best
[20:22] <OldSchool> awesome
[20:22] <OldSchool> I'll have to try out EC2
[20:26] <OldSchool> thanks for your help, gotta run but I'm sure I'll be around
[21:34] <SpamapS> argh, why does vim think 'description:' needs to be un-indented in yaml?
[21:50] <_mup_> ensemble/go-formulas r9 committed by gustavo@niemeyer.net
[21:50] <_mup_> Adding sample formula repository content for tests.
[21:51] <_mup_> ensemble/go-formulas r10 committed by gustavo@niemeyer.net
[21:51] <_mup_> Added support for pasing relations within the formula metadata.
[21:51] <_mup_> This completes support for formula metadata parsing.
[21:55] <_mup_> Bug #837027 was filed: Formula metadata parsing in Go must be completed <Ensemble:New> < https://launchpad.net/bugs/837027 >
[21:58] <_mup_> ensemble/stack-crack r329 committed by kapil.thangavelu@canonical.com
[21:58] <_mup_> be flexible about int vs. str status codes, remove error test relying on error result response parsing.
[22:04] <_mup_> ensemble/stack-crack r330 committed by kapil.thangavelu@canonical.com
[22:04] <_mup_> remove the ec2 key name usage, to verify its nesc.
[22:28] <botchagalupe> ensemble: environments
[22:28] <botchagalupe> environments:
[22:28] <botchagalupe>   sample:
[22:28] <botchagalupe>     type: ec2
[22:28] <botchagalupe>     control-bucket: ensemble-9cb5c6eea8334c9780078fe63f6ccdb0
[22:28] <botchagalupe>     admin-secret: 06ae08af8e0f4dd79180401bc28824b0
[22:28] <botchagalupe> access-key: AKIAISCMJNNSSTGM4L2A
[22:28] <botchagalupe> secret-key: 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
[22:28] <botchagalupe> That does't work and this does...
[22:28] <botchagalupe> ensemble: environments
[22:28] <botchagalupe> environments:
[22:28] <botchagalupe>   sample:
[22:28] <botchagalupe>     type: ec2
[22:28] <botchagalupe>     control-bucket: ensemble-9cb5c6eea8334c9780078fe63f6ccdb0
[22:28] <botchagalupe>     admin-secret: 06ae08af8e0f4dd79180401bc28824b0
[22:28] <botchagalupe>     access-key: AKIAISCMJNNSSTGM4L2A
[22:28] <botchagalupe>     secret-key: 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
[22:29] <botchagalupe> also any help on how to pre-seed a environments.yaml file before the bootstrap?
[22:32] <niemeyer> botchagalupe: Hey
[22:33] <niemeyer> botchagalupe: I'm not sure I get the pastes.. do you mean you'd like to have global access/secret keys?
[22:33] <niemeyer> botchagalupe: The file format is yaml, so the indentation indeed matters
[22:34] <niemeyer> botchagalupe: In terms of pre-seeding, you can just drop it in place and Ensemble will rely on the pre-existing one
[22:34] <niemeyer> botchagalupe: pre-existing environments.yaml, that is
[22:34] <niemeyer> botchagalupe: You can hand-pick all the values.. Ensemble doesn't really need anything special in there as long as you provide valid data
[22:37] <botchagalupe> no if I indent the access-key in col 1 the bootstrap fails
[22:38] <botchagalupe> ugly in col 1 but ugly not to support it...
[22:38] <botchagalupe> my bad… sorry… 
[22:40] <botchagalupe> col… will it override the the bucket and oter key values… I will give it a try .. thanks… 
[22:47] <niemeyer> botchagalupe: Yeah, the key is part of the environment, so it has to be indented with the environment fields itself
[22:47] <niemeyer> botchagalupe: That said, you can also use environment variables for the keys
[22:47] <niemeyer> botchagalupe: As standard for the ec2-* tools
[22:47] <niemeyer> botchagalupe: and omit them from the env entirely
[22:48] <niemeyer> botchagalupe: AWS_ACCESS_KEY_ID etc
[22:51] <botchagalupe> Saw that in the doc.. I was trying to pre seed in a chef cookbook for quick setup…  
[23:06] <hazmat> botchagalupe, you can also set it in environment variables
[23:07] <niemeyer> botchagalupe: Quite interesting
[23:08] <hazmat> botchagalupe, ie. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
[23:09] <jcastro> it would be a nice convenience feature to spit out the link to the keys on AWS when you try to bootstrap and it doesn't find anything
[23:10] <niemeyer> hazmat: I just said that :)
[23:10] <hazmat> niemeyer, oh.. my proxy disconnected for a few minutes, i missed about 20m of the conversation it looks like
[23:11] <niemeyer> jcastro: You mentioning the env variables as well?
[23:11] <niemeyer> hazmat: Aha, ok :)
[23:27] <hazmat> hmm.. having some problems on canonistack connectivity
[23:35] <hazmat> looks like the s3server is hosed
[23:42] <_mup_> ensemble/stack-crack r331 committed by kapil.thangavelu@canonical.com
[23:42] <_mup_> verify txaws branch support