| _mup_ | ensemble/expose-cleanup r337 committed by jim.baker@canonical.com | 01:37 |
|---|---|---|
| _mup_ | Test new securitygroup function | 01:37 |
| _mup_ | ensemble/expose-cleanup r338 committed by jim.baker@canonical.com | 02:23 |
| _mup_ | Merged trunk & resolved conflicts | 02:23 |
| _mup_ | ensemble/expose-cleanup r339 committed by jim.baker@canonical.com | 03:03 |
| _mup_ | Fixed merge issues | 03:03 |
| niemeyer | Greetings! | 13:40 |
| hazmat | niemeyer, greetings | 13:55 |
| hazmat | i do find the abbreviations to make reading the go branches a bit harder to read | 13:57 |
| hazmat | it would help if the abbreviations where at least documented once | 13:57 |
| hazmat | +type ifaceExpC struct { | 13:58 |
| hazmat | is interface expander c? | 13:58 |
| hazmat | the variable declaration for the type uses stringC which i assume to be string constant.. but thats a different usage than the preceding statement that uses C for ? | 13:59 |
| niemeyer | hazmat: Checker | 14:09 |
| niemeyer | hazmat: That convention was used on every single type within schema.go | 14:09 |
| niemeyer | hazmat: I can rename it if it feels so tricky | 14:10 |
| niemeyer | hazmat: It's akin to _state, though | 14:10 |
| hazmat | niemeyer, its just that without that context it becomes harder to understand, i don't know that it needs renaming but a comment would be useful | 14:10 |
| niemeyer | hazmat: I'm happy to rename or to add the comment | 14:11 |
| niemeyer | hazmat: // The *C convention is used within schema.go for Checkers | 14:11 |
| niemeyer | ? | 14:11 |
| hazmat | niemeyer, what's schema.M? | 14:12 |
| niemeyer | hazmat: That's documented | 14:13 |
| niemeyer | hazmat: | 14:13 |
| niemeyer | / All map types used in the schema package are of type M. | 14:13 |
| niemeyer | type M map[interface{}]interface{} | 14:13 |
| hazmat | niemeyer, ic | 14:13 |
| niemeyer | hazmat: This is just a handy alias to this type that allows building maps as | 14:14 |
| niemeyer | hazmat: schema.M{"a": 1}, etc | 14:14 |
| niemeyer | hazmat: map[interface{}]interface{}{"a": 1} would work, but.. :) | 14:14 |
| hazmat | indeeds its an improvement | 14:15 |
| niemeyer | hazmat: Yeah, this became clear in tests | 14:15 |
| niemeyer | hazmat: I wasn't using it before, but when I started to test the results in the formula work, got boring | 14:15 |
| hazmat | niemeyer, is this sort of multi-file lookup for abbreviations common in go? | 14:16 |
| niemeyer | hazmat: multi-file lookup? | 14:16 |
| hazmat | niemeyer, abbreviation used in one place, but needs to be tracked down to a different file to understand naming? | 14:17 |
| niemeyer | hazmat: Not a different file.. this is a package, and the type is defined and has meaning within the package itself | 14:17 |
| niemeyer | hazmat: schema.M is used for schemas only | 14:18 |
| niemeyer | hazmat: Just for comprehension, note that the file could be named foo.go | 14:18 |
| niemeyer | hazmat: The "schema" in "schema.M" comes from the package name, not the file name | 14:18 |
| niemeyer | hazmat: In other words, I wouldn't use schema.M in things not related to the schema itself | 14:19 |
| hazmat | niemeyer, their not different packages? one is ensemble/go/schema and ensemble/go/formula? | 14:20 |
| niemeyer | hazmat: These are different packages, yeah | 14:20 |
| niemeyer | hazmat: In these terms it's quite similar to Python.. if you use ensemble.environment.Environment, you may go to ensemble.environment to see what's that | 14:22 |
| hazmat | niemeyer, so in this case we have one package's abbreviation being used in another package, the question is if that sort of usage is common? | 14:22 |
| niemeyer | hazmat: The distinction, as I was pointing out, is that unlike Python packages are multi-file | 14:23 |
| niemeyer | hazmat: It's not an abbreviation.. it's a type | 14:23 |
| niemeyer | hazmat: and the reason why the type is used in a different file is because we're using _schema_ in the other package | 14:24 |
| hazmat | niemeyer, the name itself is an abbreviation, its the name for a type | 14:24 |
| niemeyer | hazmat: It's not an abbreviation, it's a real type | 14:24 |
| niemeyer | hazmat: reflect.TypeOf(schema.M{}) != reflect.TypeOf(map[interface{}]interface{}{}) | 14:24 |
| hazmat | "M" is not an abbreviation for MapType? | 14:25 |
| niemeyer | hazmat: Sorry, I see what you mean | 14:25 |
| niemeyer | hazmat: Yes | 14:25 |
| niemeyer | hazmat: Let me try to drive a parallel to show you what I mean | 14:25 |
| hazmat | i think this would be much readable if we avoided cross package abbreviations, i'm just trying to understand if this usage is common in go | 14:26 |
| niemeyer | hazmat: Alright.. I'll put map[interface{}]interface{}{} back then | 14:28 |
| hazmat | niemeyer, i don't think thats any better, but is there a better name for the type? | 14:29 |
| niemeyer | hazmat: schema.M? | 14:29 |
| niemeyer | hazmat: That was my take on it.. makes building the map easier | 14:29 |
| niemeyer | hazmat: I've used the same convention in other marshalling modules | 14:29 |
| niemeyer | hazmat: But I won't bikeshed on that.. if you have other suggestions, I can change | 14:30 |
| hazmat | niemeyer, schema.SchemaMap ? | 14:34 |
| niemeyer | hazmat: Heh | 14:34 |
| niemeyer | hazmat: Duplicating the schema name won't make it handier or more readable in any way | 14:34 |
| niemeyer | hazmat: Also, what's the difference between schema.SchemaMap and schema.Map? | 14:35 |
| niemeyer | hazmat: I can rename schema.Map to ... | 14:36 |
| hazmat | MapChecker | 14:36 |
| niemeyer | hazmat: schema.GenericMap | 14:36 |
| hazmat | yeah.. that works as well, and M->Map | 14:36 |
| niemeyer | hazmat: I can do that as well, as long as you never complain again about naming things with _state.. ;-D | 14:36 |
| niemeyer | hazmat: Imagine how the schema definition would look like.. | 14:37 |
| hazmat | niemeyer, sounds good as long as there no complaints about a -(_state) branch ;-) | 14:37 |
| niemeyer | hazmat: schema.MapChecker(schema.StringChecker(), schema.InterfaceChecker(....)) | 14:37 |
| hazmat | yeah.. that's lame | 14:38 |
| niemeyer | hazmat: Ok, I'll do the name change | 14:38 |
| niemeyer | hazmat: But in a follow up branch if that's alright | 14:38 |
| niemeyer | Since it'll touch everything else | 14:38 |
| hazmat | niemeyer, feel to change it a different name, i just think having a common entry point into a package be an abbreviation also feels like a violation of encapsulation | 14:39 |
| niemeyer | hazmat: I don't understand how that could be the case | 14:40 |
| niemeyer | hazmat: There's zero encapsulation involved | 14:40 |
| hazmat | because it tends to require reading the relevant implementation to understand its use | 14:40 |
| niemeyer | hazmat: The only reason we use schema.M in formula is because we're dealing with the schema there | 14:40 |
| niemeyer | hazmat: I also don't think this is true.. you don't have to read the implementation to understand its use.. you have to read the documentation, which is a pretty general problem | 14:42 |
| niemeyer | hazmat: It's like saying we need to read Python's json.dumps prototype to understand how to call it | 14:42 |
| hazmat | what it is and what its doing are fairly obvious by name alone, schema.M is not | 14:43 |
| niemeyer | hazmat: Agreed.. it's a convention to be aware of | 14:43 |
| hazmat | so if your reading code that uses both, one is pretty clear without additional context and one is not | 14:43 |
| niemeyer | hazmat: The same convention is used in gobson and goyaml | 14:43 |
| niemeyer | Actually, I lie.. gobson only it seems | 14:44 |
| niemeyer | Anyway.. I was explaining the POV | 14:45 |
| niemeyer | I'm fine with renaming it | 14:45 |
| niemeyer | hazmat: and answering your very first and simple question, no, this is not a general convention, and if it becomes one it's my fault ;-0 | 14:48 |
| niemeyer | ;-) | 14:48 |
| hazmat | niemeyer, thanks, good to know | 14:50 |
| _mup_ | ensemble/go-formulas r8 committed by gustavo@niemeyer.net | 14:52 |
| _mup_ | Renamed schema.M/L to schema.Map/ListType | 14:52 |
| _mup_ | Bug #836753 was filed: schema.M/L are not readable <Ensemble:In Progress by niemeyer> < https://launchpad.net/bugs/836753 > | 14:55 |
| niemeyer | fwereade: Gosh.. is all of that really needed for adding _http auth_? | 15:16 |
| SpamapS | http://ec2-107-20-64-136.compute-1.amazonaws.com:8080/job/tested%20PPA/42/console | 15:38 |
| SpamapS | So, other than the failure at the end because I forgot to quote the 'cd' ... first iteration of integration tests went well. | 15:39 |
| SpamapS | lp:~clint-fewbar/ensemble/jenkins-test-suite has the code under misc/jenkins | 15:39 |
| SpamapS | I need to break it up into 3 or 4 bash scripts.. but the interesting parts are wait4state and unit2machine .. which I think should become subcommands or at least helper args to some of the commands | 15:40 |
| SpamapS | wait4state especially would be best written as a zookeeper watch | 15:40 |
| SpamapS | One thing I was struggling with was whether there was a way to address yaml nodes similar to xpath for xml. | 15:45 |
| SpamapS | The way that wait4state works is pretty weird and crazy.. but it works for this specific job | 15:46 |
| jimbaker | SpamapS, taking a look | 15:47 |
| jimbaker | SpamapS, we should definitely make wait4state easier to do. as you said, it really should be using zk watch support | 15:48 |
| jimbaker | SpamapS, re some sort of xpath - really this is some sort of query syntax for working w/ python collections. there must be a number of packages out there to do just that | 15:50 |
| jimbaker | biab, i have to go to a quick doctor's appt | 15:51 |
| * SpamapS will have to vacate his house and go work at a co-working space in about an hour while the exterminators work. UGH | 15:51 | |
| fwereade | niemeyer: sorry, had to pop out -- and, as far as I can tell, yes it is | 15:54 |
| fwereade | niemeyer: I looked pretty hard for alternatives I could piggyback on, but nothing seemed cleanly doable | 15:54 |
| SpamapS | fwereade: were you at least able to configure the webdav service to auth against cobbler, so the auth credentials can remain the same? | 15:58 |
| fwereade | SpamapS: hm, I didn't think to try that -- if you don't specif separate cobbler credentials, it falls back to the orchestra ones | 16:01 |
| fwereade | SpamapS: same model as the configurable storage-url -- if you don't include one, it assumes http://%(orchestra-server)s/webdav | 16:02 |
| === daker is now known as daker_ | ||
| fwereade | SpamapS: but I have no idea how to set up cobbler and webdav with a shared auth source | 16:03 |
| SpamapS | fwereade: I forget what our default is, but https://fedorahosted.org/cobbler/wiki/CustomizableAuthentication has a run down on the auth methods possible.. authn_passthru would be the simplest way to get this done. | 16:04 |
| SpamapS | fwereade: actually authn_configfile will also work because we can point the webdav service at the same config | 16:04 |
| fwereade | SpamapS: easier than I thought then :) | 16:05 |
| fwereade | niemeyer: https://code.launchpad.net/~fwereade/ensemble/webdav-unicode-paths/+merge/72579 -- did you mean to mark that as approved for merge, or should I get someone else to give it a look? | 16:08 |
| SpamapS | fwereade: how well does urllib2 handle https btw? | 16:13 |
| SpamapS | fwereade: is it going to squawk because of self signed certs or anything like that? | 16:13 |
| fwereade | SpamapS: hm, I have no idea | 16:14 |
| SpamapS | fwereade: I'd say that initially it *should* complain with invalid certs, but that we should make it easy to turn that off given Orchestra's "backoffice" role. | 16:15 |
| kirkland | SpamapS: certs for what? | 16:15 |
| fwereade | SpamapS: wounds reasonable, but I'm not sure what context you're asking about | 16:15 |
| SpamapS | kirkland: for the webdav file storage | 16:15 |
| kirkland | SpamapS: RoAkSoAx is already using orchestra + preseed to install SSL certs in deployed guests | 16:16 |
| kirkland | SpamapS: for SSL protected rsyslog | 16:16 |
| kirkland | SpamapS: you should be able to just force the webdav to use those same certs | 16:16 |
| SpamapS | Seeing as formulas run as root and have no cryptographic verification built in.. its fairly important to consider the security implications they carry. | 16:16 |
| kirkland | SpamapS: or use the same mechanism to install additional certs | 16:16 |
| SpamapS | Yeah that sounds good. | 16:16 |
| SpamapS | Give the admin a way to easily pull that in to his local cert store. | 16:17 |
| SpamapS | I think by default it should be easy and just work w/o intervention. I also think that if the admin needs/wants to crank up the security, they should be able to. | 16:18 |
| SpamapS | (And I'm sure some people will argue that by default it should be high security, and admins should be able to turn off security) | 16:19 |
| niemeyer | adam_g, negronjl, lynxman: ping | 16:24 |
| negronjl | niemeyer: pong | 16:25 |
| niemeyer | negronjl: Hey man | 16:25 |
| negronjl | niemeyer: Happy Monday ( if there is such a thing ) :) | 16:25 |
| niemeyer | negronjl: Was just wondering about how you guys feel about how the package and code stability has been progressing | 16:26 |
| niemeyer | negronjl: Are you happy with it in general, or do you feel we've been breaking things too often somehow? | 16:26 |
| negronjl | niemeyer: Althought I am very happy with the project in general, I would feel more confortable with some stable version of ensemble that, may not have all of the latest and greates features, but, it is stable. | 16:27 |
| niemeyer | negronjl: That's a bit different from the actual question.. I would like to have a stable version too :-) | 16:27 |
| negronjl | niemeyer: I found myself with a very close call on NoSQL conference where I was scheduled to talk about Ensemble but, ensemble was broken | 16:27 |
| niemeyer | negronjl: The question is whether we've been breaking things often | 16:28 |
| negronjl | niemeyer: not often no. | 16:28 |
| niemeyer | negronjl: Ok, cool.. I understand the desire for stability, and we're doing things to improve that | 16:28 |
| niemeyer | negronjl: I just want to make sure I understand the feeling | 16:28 |
| negronjl | niemeyer: To clarify my earlier point. It would help if there is a version in ensemble that would be considered stable enough and we can somehow pin it ( a /stable ppa or so ) until another version becomes stable enough and can replace it. This would alleviate some of the issues that I experienced during NoSQL last week. | 16:30 |
| negronjl | niemeyer: Just my thoughts. | 16:30 |
| niemeyer | negronjl: Sounds good.. I hope this is the version in Ubuntu | 16:31 |
| * SpamapS too | 16:36 | |
| SpamapS | negronjl: to niemeyer's point, the version in Ubuntu is actually pretty stable. :) | 16:37 |
| jcastro | yes that's what I was using and it was working fine | 16:53 |
| adam_g | assuming whats in the archive is stable, it would good if nodes install straight from the archive by default, with the option to install a bleeding edge versin from ppa:ensemble/ppa (default now). | 17:06 |
| adam_g | niemeyer: ^ | 17:06 |
| niemeyer | fwereade: I did, sorry | 17:06 |
| fwereade | niemeyer: thanks, no worries | 17:06 |
| niemeyer | fwereade: Marked it now | 17:06 |
| niemeyer | fwereade: Sad about the auth complexity :( | 17:06 |
| fwereade | niemeyer: cheers | 17:06 |
| fwereade | niemeyer: me too :( | 17:06 |
| niemeyer | fwereade: It's _auth_, for dev's sake | 17:07 |
| fwereade | niemeyer: I came to terms with it eventually, but... quite | 17:07 |
| ahasenack | adam_g: you mean the nodes install something else? Like, trunk? | 17:08 |
| adam_g | ahasenack: i just bootstrapped a node using 0.5+bzr330-0ensemble1, and userdata sets up access to the ppa instead of installing out of the archive | 17:09 |
| ahasenack | adam_g: is that ppa "bleeding edge"? | 17:10 |
| adam_g | ahasenack: honestly do not know how/when/why packages get built and pushed there, but installing from there is where ive personally hit show stopper bugs. | 17:12 |
| hazmat | ahasenack, the ppa is a trunk daily build | 17:12 |
| ahasenack | ok | 17:12 |
| niemeyer | fwereade: Can't we use the same technique twisted.web._auth.digest uses, and rely on twisted.creds.credentials? | 17:15 |
| niemeyer | fwereade: It feels like you're reimplementing it | 17:16 |
| fwereade | niemeyer: hm, I was put off by the privateness of the module, and assumed that something would end up breaking if I used a _kindaprivate module | 17:19 |
| niemeyer | fwereade: The _auth module has pretty much nothings inside it | 17:19 |
| fwereade | hm, I thought that was where the meat of it was | 17:19 |
| fwereade | niemeyer: let me check again | 17:19 |
| niemeyer | fwereade: You're allowed to look inside private modules! ;-0 | 17:19 |
| niemeyer | :-) | 17:19 |
| hazmat | "_auth" ;-) | 17:20 |
| hazmat | there's a bug open for it | 17:20 |
| hazmat | it might be nice to ask therve or free | 17:20 |
| hazmat | or on #twisted | 17:20 |
| niemeyer | hazmat: There's nothing interesting inside it | 17:21 |
| fwereade | niemeyer: well, crap, somehow I missed cred.credentials | 17:21 |
| niemeyer | hazmat: Besides a pointer to twisted.cred.credentials | 17:21 |
| fwereade | niemeyer: I, er, did look :/ | 17:21 |
| fwereade | niemeyer: maybe I stopped looking at that point though :( | 17:21 |
| niemeyer | fwereade: Don't worry about it | 17:22 |
| niemeyer | fwereade: Maybe it's not even useful.. but certainly has very similar logic, so worth checking | 17:23 |
| fwereade | niemeyer: definitely so, I'll give it a look | 17:23 |
| niemeyer | fwereade: In either case, the auth logic itself should be factored out of FileStorage | 17:23 |
| niemeyer | fwereade: and the file | 17:23 |
| niemeyer | fwereade: and put into its own module, and tested in isolation | 17:23 |
| fwereade | niemeyer: it's not immediately apparent how to hook it up, but you're right, it's doing just the right thing | 17:23 |
| niemeyer | fwereade: This is a major missing piece in the http library, rather than part of the FileStorage problem | 17:23 |
| fwereade | niemeyer: it didn't feel quite big enough to do that at the time, but you're right | 17:24 |
| fwereade | cheers | 17:24 |
| hazmat | as another way of getting basic auth into twisted web client ... http://code.activestate.com/recipes/525493-simple-crawler-using-twisted/ | 17:24 |
| niemeyer | hazmat: Not sure about what's the point there? | 17:25 |
| hazmat | its a much simpler layering of basic auth onto twisted web client | 17:26 |
| fwereade | hazmat: surely basic auth is barely worth bothering with? | 17:26 |
| niemeyer | hazmat: Yeah, I'm trying to understand what's your underlying suggestion.. is it that we should use client.HTTPDownload? | 17:27 |
| niemeyer | fwereade: Can we? | 17:27 |
| fwereade | niemeyer: use basic auth? | 17:28 |
| hazmat | fwereade, that's fair, its not secure over http, but over https its fine | 17:28 |
| niemeyer | hazmat: Use HTTPDownloader | 17:28 |
| niemeyer | Erm | 17:28 |
| niemeyer | fwereade: Use HTTPDownloader | 17:28 |
| fwereade | niemeyer: looking at it, we probably can, the only reason I didn't use getPage is because apparently I can't get at the reply headers | 17:29 |
| niemeyer | fwereade: gotHeaders? | 17:30 |
| fwereade | niemeyer: yes indeed, that's on HTTPDownloader | 17:30 |
| fwereade | niemeyer: so, looking at it, we probably can | 17:30 |
| fwereade | niemeyer: I saw things saying "if getPage isn't good enough, use Agent"... so I used Agent :/ | 17:31 |
| fwereade | niemeyer: sounds like it's a solid "needs fixing" with a side helping of "lrn2google" | 17:32 |
| fwereade | niemeyer: at least the code'll end up smaller :) | 17:32 |
| niemeyer | fwereade: Both approaches are surprisingly involved, to be honest | 17:32 |
| fwereade | niemeyer: the creds stuff looks like it might be quite helpful, I'll look into that | 17:32 |
| niemeyer | fwereade: The approach referenced by hazmat seems slightly better because it's more encapsulated than the one in your branch | 17:33 |
| fwereade | niemeyer: there are approaches I didn't consider, so I should definitely look into them | 17:33 |
| niemeyer | fwereade: It should be possible to merge your auth with the simpler approach | 17:33 |
| niemeyer | hazmat: Which is why I was asking what was the underlying suggestion | 17:33 |
| fwereade | niemeyer: ah, I see | 17:33 |
| fwereade | niemeyer: I think it's worth spending a little while trying to hook up the cred.credentials solution though, that would be best | 17:34 |
| niemeyer | fwereade: Sounds good.. I'm happy with any of these options | 17:34 |
| fwereade | niemeyer: anyway, I'm sorry to cut you off, but the calls for suppertime are becoming quite insistent | 17:34 |
| fwereade | niemeyer: I'll try to have something better tomorrow | 17:35 |
| niemeyer | fwereade: The only thing that we should necessarily do is to factor that problem auth of the FileStorage | 17:35 |
| niemeyer | s/auth of/out of/ | 17:35 |
| niemeyer | Contextual typo :) | 17:35 |
| niemeyer | SpamapS: This is sitting on our review queue for 2 weeks untouched: https://code.launchpad.net/~clint-fewbar/txaws/fix-s3-port/+merge/71289 | 17:47 |
| niemeyer | SpamapS: Can you please address the points for getting it merged, or put it back in Work In Progress otherwise? | 17:48 |
| SpamapS | niemeyer: I started working on that about 30 minutes ago actually. :) | 17:50 |
| niemeyer | SpamapS: WOohay | 17:50 |
| SpamapS | niemeyer: just writing some test code. I addressed the first point before Robert made it. :) | 17:50 |
| niemeyer | SpamapS: Great timing then :) | 17:50 |
| SpamapS | niemeyer: It seems that all of the tests in txaws are functional tests.. I see no mocking for AWS | 17:51 |
| niemeyer | SpamapS: I've tweaked tests there before | 17:53 |
| niemeyer | SpamapS: There are unittests | 17:53 |
| SpamapS | jimbaker: so.. security group cleanup... | 18:12 |
| jimbaker | SpamapS, yes, it's about ready to land | 18:13 |
| jimbaker | (in expose-cleanup branch) | 18:13 |
| SpamapS | jimbaker: http://ec2-107-20-64-136.compute-1.amazonaws.com:8080/job/tested%20PPA/43/console | 18:14 |
| SpamapS | jimbaker: is that a bug? | 18:14 |
| jimbaker | SpamapS, expose-cleanup fixes this issue | 18:15 |
| SpamapS | jimbaker: so I have to manually delete all the groups before I can re-use the environment? | 18:16 |
| jimbaker | what we see here is generally seen when doing an immediate bootstrap after shutdown - the instance originally associated with this group is still around (in 'shutting-down' state) and so cannot be deleted | 18:16 |
| jimbaker | SpamapS, once expose-cleanup lands in trunk, you won't have to anything | 18:16 |
| jimbaker | the other possibility is to just wait sufficiently long | 18:16 |
| SpamapS | jimbaker: the shutdown was 24 hours ago | 18:17 |
| m_3 | I just wait it out | 18:17 |
| m_3 | wow! | 18:17 |
| jimbaker | SpamapS, something is holding onto that security group then | 18:17 |
| m_3 | usually cleans itself up after a few minutes | 18:17 |
| jimbaker | SpamapS, so expose-cleanup simply polls the instance state to see if they're terminated. if so, it can delete the associated security groups | 18:18 |
| jimbaker | or i should say, this polling is done in a revised shutdown_machines in ensemble.providers.ec2 | 18:18 |
| jimbaker | SpamapS, please check whether or not you can manually delete that security group | 18:19 |
| jimbaker | using the AWS console | 18:19 |
| jimbaker | or tool of choice | 18:19 |
| SpamapS | how can I see what instances are in a group w/ the cmdline tools? | 18:19 |
| jimbaker | SpamapS, i don't believe that's directly possible, but i could be wrong | 18:20 |
| jimbaker | i know it's in the reservation part of the instance description | 18:20 |
| SpamapS | hmm must have been transient | 18:21 |
| _mup_ | ensemble/expose-cleanup r340 committed by jim.baker@canonical.com | 18:21 |
| _mup_ | Merged trunk | 18:21 |
| SpamapS | its odd, my testing does shutdown+bootstrap pretty constantly.. maybe 10 seconds of lag between those two steps | 18:22 |
| m_3 | 10s is rarely enough time... I usually let it sit for a minute between shutdown and the next bootstrap | 18:23 |
| jimbaker | SpamapS, that testing regimen is not going to work well until this cleanup occurs - it needs at least a couple of minutes in between for ec2 to properly cleanup | 18:23 |
| SpamapS | jimbaker: but it has worked maybe 20 times so far | 18:23 |
| * m_3 surprised | 18:23 | |
| jimbaker | SpamapS, i'm also surprised | 18:24 |
| jimbaker | :) | 18:24 |
| jimbaker | sounds like ec2 was especially fast in its cleanup | 18:24 |
| SpamapS | I think the way I'll make it work is just to put each target in its own "environment" | 18:24 |
| jimbaker | SpamapS, that makes much more sense | 18:24 |
| m_3 | good test of other things too :) | 18:24 |
| SpamapS | I need to find a way to generate and dump an environments.yaml into the test chroots anyway. :p | 18:25 |
| jimbaker | that will definitely isolate | 18:25 |
| _mup_ | ensemble/expose-cleanup r341 committed by jim.baker@canonical.com | 18:34 |
| _mup_ | PEP8 & PyFlakes | 18:34 |
| lynxman | niemeyer: pong, sorry it was bank holiday here in the UK | 18:34 |
| niemeyer | lynxman: Hey | 18:35 |
| niemeyer | lynxman: No worries | 18:35 |
| jimbaker | hazmat, i briefly looked at autodoc, but there seems to be some problem in our docstrings which is causing it to crash when i run the sphinx build | 18:35 |
| niemeyer | lynxman: Was just wondering about how Ensemble has been treating you.. I've heard a few claims that you guys were facing problems with us introducing new bugs daily, but it sounds like it wasn't really the case | 18:35 |
| lynxman | niemeyer: we had some stability issues in formulas indeed, but as soon as that happens we either ask here or open bug reports | 18:36 |
| niemeyer | lynxman: What kind of stability problem was that? Just want to make sure I'm aware | 18:37 |
| hazmat | jimbaker, any tracebacks? | 18:37 |
| jimbaker | hazmat, sure, let me paste | 18:37 |
| jimbaker | hazmat, actually hold on that - i was going against my expose-cleanup branch, but i didn't want it to get mired in a bigger problem, so i reverted my sphinx conf changes | 18:38 |
| jimbaker | hazmat, just fairly simple - update conf.py, plus __init__.py (or use some third party gen scripts to do the walking) | 18:39 |
| lynxman | niemeyer: in my case was just trying to make Ensemble work against openstack, but hazmat helped a lot on it so I didn't need to file any bugs | 18:40 |
| jimbaker | hazmat, docutils was crashing in its statemachine.py code w/ a recursion limit issue, so something was indigestible | 18:40 |
| niemeyer | lynxman: Ahh, ok | 18:40 |
| lynxman | niemeyer: negronjl and iamfuzz are working on the cloudfoundry formulas which encountered more issues, which were raised on time | 18:40 |
| niemeyer | lynxman: Well.. OpenStack support is a minefield at the moment | 18:40 |
| SpamapS | so.. does ensemble make use of blueprints at all? | 18:40 |
| jimbaker | hazmat, anyway, i will recreate in a bit | 18:40 |
| hazmat | SpamapS, not really bcsaller1 and i where experimenting with them | 18:40 |
| hazmat | but pretty much zero | 18:41 |
| lynxman | niemeyer: it kinda is, but so far works pretty okay :) | 18:41 |
| hazmat | niemeyer, its not that bad | 18:41 |
| niemeyer | lynxman: That's good to know :-) | 18:41 |
| SpamapS | I was thinking of a suite of tools to make ensemble more command-line-useful ... waiting for states, outputting data about machines/services that is not yaml.. and exit codes to differentiate permanent and transient errrors. | 18:41 |
| lynxman | niemeyer: as said, hazmat helped a great deal | 18:41 |
| hazmat | niemeyer, the ec2 compatibility issue they have marked as a bug, although its unclear when they'll fix it.. but just as far as impact to the ec2 provider, its like a dozen lines | 18:42 |
| jimbaker | SpamapS, maybe "ensemble watch" | 18:42 |
| niemeyer | hazmat: Yeah, I'm more concerned in conceptual terms than in terms of the particular issue | 18:42 |
| hazmat | not including tests of course | 18:42 |
| jimbaker | SpamapS, another thing that might be useful is to have a wait-for ensemble is ready command | 18:42 |
| niemeyer | hazmat: It doesn't feel like there's serious interest in preserving the compatibility | 18:42 |
| hazmat | niemeyer, so they will fix ec2 bugs, but not s3 is what i got out of it | 18:42 |
| hazmat | niemeyer, agreed about the overall compatibility.. they have a layer, it has tests, they won't break it, but if something doesn't work, we'll probably need to make it do so | 18:43 |
| jimbaker | SpamapS, there are now at least two places that do that last bit for the ec2 provider (via polling ec2_describe_instances); in particular ftests have this functionality | 18:43 |
| SpamapS | jimbaker: right, there's really a swath of changes that need a binding purpose.. this is what blueprints are for | 18:43 |
| jimbaker | m_3, i would assume this wait-for-ensemble command would be useful for you too | 18:44 |
| hazmat | niemeyer, they have an ec2 compatibility bug for the security group work, that ensemble needs, its still marked as a bug as opposed to the s3 one. | 18:44 |
| hazmat | https://bugs.launchpad.net/nova/+bug/829609 | 18:44 |
| _mup_ | Bug #829609: EC2 compatibility describe security group returns erroneous value for group ip permissions <Ensemble:In Progress by hazmat> <OpenStack Compute (nova):Confirmed> < https://launchpad.net/bugs/829609 > | 18:44 |
| SpamapS | jimbaker: actually I'd say that zookeeper is the system of record... so polling zk is the only sensible thing to do. | 18:44 |
| jimbaker | SpamapS, also in ftests | 18:44 |
| jimbaker | specifically a watch for our ZK nodes created by ensemble | 18:45 |
| niemeyer | hazmat: Good to know about their feeling about that, thanks | 18:45 |
| jimbaker | let me push what i have for that - not yet sufficiently robust, but at least it usually works | 18:46 |
| hazmat | depending on the command probably an agent watch for unit or machine | 18:46 |
| _mup_ | ensemble/fix-functional-testing r334 committed by jim.baker@canonical.com | 18:47 |
| _mup_ | Better branch identification | 18:47 |
| jimbaker | SpamapS, i don't like ensemble.ftests.test_ec2_provider.EC2MachineTest.assert_node yet, but it's on the right path | 18:49 |
| jimbaker | note that we still need to poll ec2 first, then *watch* zk | 18:49 |
| SpamapS | jimbaker: well my point is that this is something to plan at a larger scale than solving each of these issues one by one. | 18:50 |
| SpamapS | jimbaker: there's a real use case for using ensemble in jenkins for instance. | 18:50 |
| SpamapS | jimbaker: you only need to ask the provider for the seed so you can find zookeeper. After that you wait for zookeeper, then you listen to zookeeper. | 18:51 |
| jimbaker | SpamapS, agreed with that approach | 18:51 |
| m_3 | jimbaker: yes, something that gives us that functionality | 18:56 |
| SpamapS | ugh these test cases are really oddly organized | 18:58 |
| hazmat | SpamapS, yeah.. you need an output capture from openstack effectively | 19:03 |
| SpamapS | hazmat: for what I did, I just need to verify that the port is put back into the generated urls actually. :) | 19:03 |
| hazmat | SpamapS, that sounds much nicer ;-) | 19:04 |
| hazmat | fixing the security group parsing needs a response output capture to verify the implementation | 19:04 |
| SpamapS | hazmat: right, that one at least is being discussed upstream as being fixed .. tho probably not in diablo. :( | 19:05 |
| SpamapS | hrm.. txaws's test cases that use venusian fail when it is installed .. looks like they were written to a different venusian api | 19:13 |
| SpamapS | http://paste.ubuntu.com/677409/ | 19:14 |
| _mup_ | ensemble/trunk r335 committed by jim.baker@canonical.com | 19:31 |
| _mup_ | merge expose-cleanup [r=niemeyer,fwereade][f=824219] | 19:31 |
| _mup_ | For EC2: shutting down machines removes machine security groups; | 19:31 |
| _mup_ | destroying the environment removes the environment security group. | 19:31 |
| niemeyer | jimbaker: Just double checking: have you tested it with a real interaction on _trunk_? | 19:32 |
| jimbaker | niemeyer, i will double check it one more time - it has worked every time, but i will check as pushed | 19:33 |
| niemeyer | jimbaker: Awesome, thanks a lot | 19:33 |
| ahasenack | SpamapS: txaws needs venusian 1.0a1 at least, it was just released | 19:33 |
| niemeyer | SpamapS: your opinion on this would also be welcome | 19:33 |
| jimbaker | niemeyer, looks good as pushed with bouncing repeatedly (ensemble bootstrap && ensemble shutdown && ...), this was the pathological case of what the cleanup now fixes | 19:36 |
| jimbaker | SpamapS, m_3 - but nothing like real usage. so please tell me if this doesn't fix the problem you've been seeing | 19:38 |
| m_3 | jimbaker: will do | 19:38 |
| jimbaker | m_3, thanks! | 19:39 |
| SpamapS | so, I just realized a big problem | 19:39 |
| SpamapS | Ensemble by default pulls itself on deployed machines from the PPA | 19:40 |
| SpamapS | This means that if ensemble is being driven from, say, mac ports.. at revision 295 .. it may deploy incompatible versions of ensemble onto the machines. | 19:40 |
| SpamapS | I *think* have an answer for this.. but it may be a little bit weird... | 19:41 |
| jimbaker | SpamapS, there's an outstanding bug about this that i started looking into | 19:42 |
| jimbaker | bug 828147 | 19:42 |
| _mup_ | Bug #828147: Ensemble branch option needs to allow for distro pkg, ppa, and source branch install <Ensemble:New for jimbaker> < https://launchpad.net/bugs/828147 > | 19:42 |
| jimbaker | SpamapS, it's not quite what you described just now, but it does allow for precise specification of what it should be on the target environment | 19:43 |
| SpamapS | jimbaker: it has to automatically choose itself though | 19:45 |
| jimbaker | SpamapS, exactly, it would be nice for an automagic option in ensemble-origin | 19:45 |
| SpamapS | I can't think of any time where I want to, by default, deploy something other than what I'm running. | 19:45 |
| SpamapS | unless we guarantee backward compatibility | 19:45 |
| jimbaker | SpamapS, we actually are planning to do just that for client access | 19:46 |
| hazmat | jimbaker, we're not guaranteeing backward compatibility | 19:46 |
| jimbaker | there's now a protocol version that's stored in zk (in the topology node to be precise); it's checked for compatibility on every operation working w/ topology (which is most of them) | 19:46 |
| hazmat | we're just detecting a conflict | 19:46 |
| jimbaker | hazmat, sorry i've reinterpreted the problem so it works like you said :) | 19:47 |
| jimbaker | we only guarantee backwards compatibility if the protocol version is compatible | 19:47 |
| jimbaker | this still allows for some wiggle room | 19:48 |
| hazmat | SpamapS, the txaws tests run fine on natty fwiw | 19:49 |
| SpamapS | hazmat: yeah the tests run fine for me w/o venusian | 19:49 |
| SpamapS | on oneiric | 19:49 |
| * ahasenack is invisible | 19:51 | |
| hazmat | SpamapS, ping me when you've got tests pushed, i'll merge to my updated my branch (~hazmat/txaws/fix-s3-port-and-bucket-op/) and push for review | 19:54 |
| niemeyer | hazmat: Does that look good: | 20:00 |
| niemeyer | c.Assert(meta.Provides["server"], Equals, formula.Relation{Interface: "mysql"}) | 20:00 |
| niemeyer | ? | 20:00 |
| hazmat | niemeyer, that looks great | 20:01 |
| niemeyer | hazmat: Awesome, I'll move forward with it then, thanks | 20:01 |
| SpamapS | hazmat: I pushed over an hour ago | 20:03 |
| hazmat | SpamapS, awesome, good to know, i'll merge and push the additional fixes for review, thanks | 20:03 |
| OldSchool | does ensemble have to be used w/ EC2 or can you use it with, say, an in-house ESXi environment? | 20:03 |
| OldSchool | ah, wait, found the FAQs finally | 20:04 |
| hazmat | OldSchool, no esxi support at the moment, ensemble supports pluggable providers, and we're working on some additional ones at the moment (physical machine, openstack, local dev), but no plans atm for esxi | 20:04 |
| SpamapS | hazmat: no, thank *you* :) | 20:04 |
| OldSchool | thank you | 20:06 |
| hazmat | OldSchool, it shouldn't be too bad to add one, implementing a provider has gotten cleaned up quite a bit, and made easier, but its probably going to need community contribution.. if your interested folks around here be happy to help you out | 20:06 |
| OldSchool | sweet, I just heard about it from a Canonical rep on Friday after doing some tinkering w/ Puppet and Landscape | 20:07 |
| OldSchool | I only admin 4-5 Ubuntu boxes at work and they're pretty self-sufficient, I'm just always interested in automation, more curious for future projects | 20:08 |
| hazmat | OldSchool, if you want to give ensemble a test drives, its probably to use ec2.. we'll be supporting a local development mode in the 11.10 release for ubuntu laptop/desktops which will also allow some experimentation and test driving | 20:12 |
| hazmat | s/probably/probably best | 20:13 |
| OldSchool | awesome | 20:22 |
| OldSchool | I'll have to try out EC2 | 20:22 |
| OldSchool | thanks for your help, gotta run but I'm sure I'll be around | 20:26 |
| === daker_ is now known as daker | ||
| SpamapS | argh, why does vim think 'description:' needs to be un-indented in yaml? | 21:34 |
| _mup_ | ensemble/go-formulas r9 committed by gustavo@niemeyer.net | 21:50 |
| _mup_ | Adding sample formula repository content for tests. | 21:50 |
| _mup_ | ensemble/go-formulas r10 committed by gustavo@niemeyer.net | 21:51 |
| _mup_ | Added support for pasing relations within the formula metadata. | 21:51 |
| _mup_ | This completes support for formula metadata parsing. | 21:51 |
| _mup_ | Bug #837027 was filed: Formula metadata parsing in Go must be completed <Ensemble:New> < https://launchpad.net/bugs/837027 > | 21:55 |
| _mup_ | ensemble/stack-crack r329 committed by kapil.thangavelu@canonical.com | 21:58 |
| _mup_ | be flexible about int vs. str status codes, remove error test relying on error result response parsing. | 21:58 |
| _mup_ | ensemble/stack-crack r330 committed by kapil.thangavelu@canonical.com | 22:04 |
| _mup_ | remove the ec2 key name usage, to verify its nesc. | 22:04 |
| botchagalupe | ensemble: environments | 22:28 |
| botchagalupe | environments: | 22:28 |
| botchagalupe | sample: | 22:28 |
| botchagalupe | type: ec2 | 22:28 |
| botchagalupe | control-bucket: ensemble-9cb5c6eea8334c9780078fe63f6ccdb0 | 22:28 |
| botchagalupe | admin-secret: 06ae08af8e0f4dd79180401bc28824b0 | 22:28 |
| botchagalupe | access-key: AKIAISCMJNNSSTGM4L2A | 22:28 |
| botchagalupe | secret-key: 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 | 22:28 |
| botchagalupe | That does't work and this does... | 22:28 |
| botchagalupe | ensemble: environments | 22:28 |
| botchagalupe | environments: | 22:28 |
| botchagalupe | sample: | 22:28 |
| botchagalupe | type: ec2 | 22:28 |
| botchagalupe | control-bucket: ensemble-9cb5c6eea8334c9780078fe63f6ccdb0 | 22:28 |
| botchagalupe | admin-secret: 06ae08af8e0f4dd79180401bc28824b0 | 22:28 |
| botchagalupe | access-key: AKIAISCMJNNSSTGM4L2A | 22:28 |
| botchagalupe | secret-key: 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 | 22:28 |
| botchagalupe | also any help on how to pre-seed a environments.yaml file before the bootstrap? | 22:29 |
| niemeyer | botchagalupe: Hey | 22:32 |
| niemeyer | botchagalupe: I'm not sure I get the pastes.. do you mean you'd like to have global access/secret keys? | 22:33 |
| niemeyer | botchagalupe: The file format is yaml, so the indentation indeed matters | 22:33 |
| niemeyer | botchagalupe: In terms of pre-seeding, you can just drop it in place and Ensemble will rely on the pre-existing one | 22:34 |
| niemeyer | botchagalupe: pre-existing environments.yaml, that is | 22:34 |
| niemeyer | botchagalupe: You can hand-pick all the values.. Ensemble doesn't really need anything special in there as long as you provide valid data | 22:34 |
| botchagalupe | no if I indent the access-key in col 1 the bootstrap fails | 22:37 |
| botchagalupe | ugly in col 1 but ugly not to support it... | 22:38 |
| botchagalupe | my bad… sorry… | 22:38 |
| botchagalupe | col… will it override the the bucket and oter key values… I will give it a try .. thanks… | 22:40 |
| niemeyer | botchagalupe: Yeah, the key is part of the environment, so it has to be indented with the environment fields itself | 22:47 |
| niemeyer | botchagalupe: That said, you can also use environment variables for the keys | 22:47 |
| niemeyer | botchagalupe: As standard for the ec2-* tools | 22:47 |
| niemeyer | botchagalupe: and omit them from the env entirely | 22:47 |
| niemeyer | botchagalupe: AWS_ACCESS_KEY_ID etc | 22:48 |
| botchagalupe | Saw that in the doc.. I was trying to pre seed in a chef cookbook for quick setup… | 22:51 |
| hazmat | botchagalupe, you can also set it in environment variables | 23:06 |
| niemeyer | botchagalupe: Quite interesting | 23:07 |
| hazmat | botchagalupe, ie. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY | 23:08 |
| jcastro | it would be a nice convenience feature to spit out the link to the keys on AWS when you try to bootstrap and it doesn't find anything | 23:09 |
| niemeyer | hazmat: I just said that :) | 23:10 |
| hazmat | niemeyer, oh.. my proxy disconnected for a few minutes, i missed about 20m of the conversation it looks like | 23:10 |
| niemeyer | jcastro: You mentioning the env variables as well? | 23:11 |
| niemeyer | hazmat: Aha, ok :) | 23:11 |
| hazmat | hmm.. having some problems on canonistack connectivity | 23:27 |
| hazmat | looks like the s3server is hosed | 23:35 |
| _mup_ | ensemble/stack-crack r331 committed by kapil.thangavelu@canonical.com | 23:42 |
| _mup_ | verify txaws branch support | 23:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!