/srv/irclogs.ubuntu.com/2011/09/06/#ubuntu-ensemble.txt

* hazmat yawns09:57
_mup_Bug #842488 was filed: Enable cloud-init debug output to better support problem analysis <Ensemble:In Progress by smoser> < https://launchpad.net/bugs/842488 >10:09
_mup_ensemble/stack-crack r332 committed by kapil.thangavelu@canonical.com10:19
_mup_clean up bad merge from formula-provider-storage url integration10:19
hazmathmm.. one more openstack ec2 incompatiblity found.. only affects shutdown though10:21
_mup_Bug #842497 was filed: Error on openstack env shutdown <Ensemble:New for hazmat> < https://launchpad.net/bugs/842497 >10:22
fwereadehazmat: what's the distinction between a machine_id and a machine_state_id?10:25
fwereadehazmat: because it seems to me that they're the same, except that the requirement that it be an int is never expressed with reference to machine_id10:27
=== Hussain is now known as Guest46158
smoserSpamapS, i thiknm you're planning on getting a new ensemble into ubuntu, right? when you do i would really like to have bug 842488 in.14:02
_mup_Bug #842488: Enable cloud-init debug output to better support problem analysis <Ensemble:In Progress by smoser> < https://launchpad.net/bugs/842488 >14:02
smoserif people are going to use those packages, then this will really help debug.14:02
SpamapSsmoser: its trivial enough, I'll drop it in as a patch until it gets merged14:19
SpamapShazmat: since the txaws MP's were approved, but I'm not on that particular team, can you merge both branches into txaws?14:28
SpamapShazmat: I'll handle putting them in as a patch to the package14:28
jimbaker`fwereade, re machine_state_id - i only see one reference to that in trunk, in ensemble.agents.provision15:05
jimbaker`fwereade, in that case, it's certainly the same as machine_id15:05
fwereadejimbaker`: heh, I didn't actually look for other references, I thought I'd already seen it elsewhere ;)15:06
fwereadejimbaker`, cheers15:06
jimbaker`fwereade, re being constrained to an int, the one place that appears to be the case is in terminate_machine, in the corresponding argparse. normally it's something like 0 or 42, but we often test it as a "machine-0"15:08
jimbaker`we have tried not to overly constrain what these IDs are15:09
fwereadejimbaker`: it seems to me that it's tighter than that, just a sec15:09
fwereadejimbaker`, ensemble.state.machine, get_machine_state15:10
fwereadejimbaker`, if it's not an int (or a str that can be inted) it will fail15:10
jimbaker`fwereade, indeed that's the case15:11
fwereadejimbaker`, so while many things can still work if it's not an int, it feels kinda wrong to throw random data around elsewhere15:11
fwereadeif, say, bootstrap were to create a machine called "bootstrap-0", it would indeed start a machine, but the rest of the system wouldn't be able to use it properly15:12
jimbaker`fwereade, yeah, i'm not certain what distinction is intended here15:12
fwereadejimbaker`, it's the sort of thing that makes me want to just tighten things up a little by asserting that the machine-id field in machine_data actually is an int when we call start_machine15:13
fwereadethe tests imply you can pass anything :)15:13
jimbaker`fwereade, even if you were to remove this constraint? i have never asked hazmat about it, but the tests as we have discussed do use strings preferentially15:14
fwereadejimbaker`, I'm not sure I want to remove any constraints, just make the ones we already have (but I only just discovered) explicit15:15
jimbaker`fwereade, the external machine id generation is just a (convenient) artifact of the internal ZK representation15:15
fwereadejimbaker`, your point being that the constraint is itself unnecessary?15:15
jimbaker`fwereade, it's quite possible this constraint is rather old15:16
jimbaker`and we have never exercised it15:16
jimbaker`it's certainly the intent expressed in a lot of tests that the machine id could in fact be something besides an int15:16
fwereadejimbaker`, hmm, makes sense15:17
jimbaker`fwereade, fwiw, hazmat is credited with that code (or at least touching those lines) in bzr blame. but my bzr fu is currently too weak to know how to map it to the rev # on trunk15:25
fwereadejimbaker`, I'm just going to leave an XXX there for now, I don't want to make any logic changes while I'm documenting15:26
jimbaker`fwereade, sounds like a good plan15:26
jimbaker`fwereade, btw, are you doing the doc changes to use autodoc in one big branch?15:27
fwereadejimbaker`, I'm stacking them15:28
fwereadejimbaker`, I'm finding the documenting quite hard going tbh15:28
fwereadejimbaker`, but it's teaching me a lot about the bits ofthe code I don't know15:28
fwereadejimbaker`, but if I tried to do everything at once I think I'd go insane15:28
jimbaker`fwereade, good to split up. it would make reviewers insane too. just don't know if you should be the one doing all of these fixes as well15:30
jimbaker`fwereade, also, is it necessary to fully specify the full path in the rst? my understanding is that the full path is only necessary for resolving ambiguities15:34
jimbaker`eg :class:`ensemble.machine.ProviderMachine` vs :class:`ProviderMachine`. i guess it's worth experimenting with15:34
jimbaker`fwereade, regrettably it doesn't seem able to do the xref unless the path is explicit15:43
fwereadejimbaker`, indeed, I resisted the temptation to start trying to hack up sphinx to make it resolve non-ambiguous references ;)15:44
fwereadejimbaker`, and, well, it's teaching me a lot, but to be fair I'm not that likely to finish everything15:45
jimbaker`fwereade, well it does seem some sort of hacking up is required, as seen in generate_modules.py15:45
fwereadejimbaker` that's distinct from hacking sphinx itself ;)15:45
jimbaker`fwereade, indeed, there is that distinction15:46
fwereadejimbaker`, anyway, I have something else to do that's oddly uninspiring, and docs feel like a good thing to work on while I try to figure out what's bothering me in the other stuff 15:46
fwereadejimbaker`, I'm not committing to documenting everything, but I can make things a little better and teach myself stuff in the process :)15:47
jimbaker`fwereade, indeed, it's a very good way to learn the codebase by having to do some sort of explicit/formal pass over it, vs just reading code15:48
niemeyerHey folks15:54
jimbaker`niemeyer, hi15:54
niemeyerI'm heading out to lunch.. will be biab15:54
jimbaker`niemeyer, enjoy!15:55
jcastrolynxman: any word on macports?15:55
lynxmanjcastro: nope :( https://www.macports.org/ports.php?by=name&substr=ensemble16:01
jcastroso is there like someone we can ping?16:01
m_3jcastro, lynxman: what abpout homebrew?16:04
lynxmanm_3: if you get the port it works fine :)16:04
lynxmanjcastro: I'll try to ping them tomorrow in the irc channel, I'm supposedly off today (why am I on irc? I don't know)16:05
jcastroheh, ok16:05
m_3jcastro: thanks for the syndication!16:12
m_3we're missing some posts though16:12
jcastroyeah the plugin got upgraded16:13
m_3Juan's got a couple in that same timeperiod that've gone missing16:13
m_3membase16:13
m_3and hpcc16:13
jcastrook16:13
m_3lemme know if we should do something different with the formatting to make it easier16:15
m_3thanks man!16:15
jcastrook I think I see the problem16:15
jcastroyou need a "planet" tag AND the "featured" tag16:15
jcastroso using both those tags should autosyndicate you guys16:15
jcastronegronjl: ^16:15
m_3ok, I'll add them on future posts16:16
m_3jcastro: should I retroactively do it to old ones?16:16
jcastroI've been adding them by hand in the syndication thing, I guess that would make more sense16:16
negronjljcastro:  I'll add them to all of mine ... let's see what happens then16:17
jcastrok16:17
jcastrohey so m_3 16:17
m_3me216:17
jcastroabout that node.js "easy" article I linked you to16:18
jcastrothe one on HN.16:18
m_3I'm working on a post now16:18
jcastroI was thinking a nice response on how we're making all that easier16:18
jcastroand then when we pingback the guy I can send him a note16:18
jcastroand then he can mention it on the original article, that would be win16:18
m_3yup!16:19
kirklandhazmat: pong16:19
m_3one of the comments is by a friend16:19
m_3small world16:19
jcastrooh awesome16:19
negronjljcastro:  posts tagged16:25
* niemeyer waves17:22
niemeyerfwereade: ping17:23
fwereadeniemeyer: pong17:23
niemeyerfwereade: Heya!17:23
fwereadeniemeyer: hey :) how's it going?17:23
niemeyerfwereade: Nicer today :)17:23
fwereadeniemeyer: jolly good17:24
fwereadeniemeyer: as you say, yay, modern medicine :)17:24
niemeyerfwereade: How's the Orchestra stuff going?17:24
fwereadeniemeyer: I've been finding it oddly tricky to redo the put-auth stuff17:25
niemeyerfwereade: Hmm.. how so?17:26
fwereadeniemeyer: I ended up doing docs and poking at go today, I'm hoping whatever's bothering me will sort itself out by tomorrow17:26
fwereadeniemeyer: whenever I start, it feels like I'm reinventing the wheel17:26
niemeyerfwereade: Ok17:27
fwereadeniemeyer: HTTPClient doesn't seem to be quite right -- I need at least a couple of distinct classes that differ slightly from the ones they've provided 17:27
fwereadeniemeyer: really I just need to force myself to finish it off, and it'll probably turn out to be fine17:27
niemeyerfwereade: Cool.. I'm hoping to have a hand from you on the store stuff17:28
fwereadeniemeyer: cool, that sounds like fun :)17:28
niemeyerfwereade: But for that we need to have the Orchestra fully closed17:28
fwereadeniemeyer: well, I'll polish that auth stuff off as soon as I can and hassle RoAkSoAx for any more bugs he can think of ;)17:29
niemeyerfwereade: Awesome17:30
niemeyerfwereade: Once you've finished with that, please ping me so we can synchronize on the store17:30
fwereadeniemeyer: cool, will do :D17:31
fwereadeniemeyer: hopefully tomorrow :)17:31
niemeyerAwesome!17:33
jimbaker`fwereade, hazmat introduced that change in get_machine_state in jan 13 as part of the misc-demo-fixes branch. but it seems more to be able to parse internal ids17:47
hazmatSpamapS, yes re merge, post plumbers17:48
hazmatbus time.. bbiab17:49
hazmatfwereade, jimbaker` the machine i corresponds to the zk sequence value of the machine state node17:53
hazmats/i/id17:53
jimbaker`hazmat, correct17:53
hazmatjimbaker`, trying to back track through the conversation, not sure what the question was17:53
jimbaker`hazmat, fwereade was simply bringing up this restriction as part of his doc work17:53
hazmatah17:54
jimbaker`hazmat, in many tests, machine id is often something like "machine-0"17:54
jimbaker`but in a few places, like get_machine_state, we explicitly require it to be an integer, or something that has an integer portion17:54
jimbaker`so "machine-0" would in fact qualify in that case ;)17:55
hazmatyeah.. in retrospect, it would hav ebeen nice to have those namespaced to something more blantaly obvious.. ssh 0 .. vs ssh machine/017:55
hazmathotspot on a bus .. rocks17:55
jimbaker`hazmat, i always liked how aws does it, i-abcdef17:55
jimbaker`and of course we do that in the topology tests, s-0 or m-017:56
hazmatjimbaker`, at this point are namespace precedent is "qualifier/id" for service units, which is pretty nice overall i think... definitely true that tests violate those blatantly where they can17:57
jimbaker`hazmat, all vehicles should be a hotspot. i definitely enjoy my verizon router for that aspect17:57
hazmati was able to get the local dev provider bootstrapped on a plane17:58
jimbaker`hazmat, and maybe the tests should do just that17:58
hazmatSpamapS, is there a way to prime a debcache dir with additional packages.. i really want to have the ability to do completely offline demos, just trying to understand how to go about it.. afaics, i would also need to setup a file apt repo accessible the containers for formula pkg installs17:59
_mup_ensemble/local-ubuntu-provider r348 committed by kapil.thangavelu@canonical.com18:05
_mup_work around path issues when launching machine agent, better multi bootstrap/shutdown handling, incorporate unpack formula into start unit, fix up admin identity into hierarchy initialization, make unitdeploy.is-running return a deferred18:05
hazmatbcsaller, so i got about as far as creating the lxc container in response to ensemble deploy with the latest on the above branch, still lots of work, but i'm optimistic.. 18:07
bcsallerhazmat: I am as well18:07
bcsallerhazmat: where are you now?18:07
hazmatbcsaller, i'm wondering if we can do away with the ensemble-branch/origin stuff and try to always deploy into the container the version running the cli, via copying the source tree into the container.. maybe too aggressively implicit.18:08
hazmatbcsaller, on a bus just left sfo airport about 10m ago.. heading north on the 118:08
SpamapShazmat: if the right .deb is in /var/cache/apt/archives it will not be downloaded18:08
bcsallerhazmat: for local dev that might work, but the origin stuff would still be needed on EC2 I'd think18:08
SpamapS/var/cache/apt/archive actually18:09
bcsallerbus w/wifi, nice 18:09
hazmatbcsaller, personal t-mobile hotspot.. works just as well :-)18:09
SpamapShazmat: oh debootstrap.. hm18:09
hazmatbcsaller, its already outside the container on the host... its just a bit of an ugly issue when using multiple branches against the same env.. different deploys with different unit code.. and bad from an upgrade perspective.. probably makes a debian maintainer cry somewhere ;-)18:10
hazmatSpamapS, excellent thanks18:10
hazmatSpamapS,  i think the /var/cache/apt/archives priming should be able to do it.. the debcache thing isn't going to help per se, if we can't pass the additional packages to be installed when doing the debootstrap18:11
SpamapShazmat: you can also just build your own mirror w/ reprepro18:11
hazmatSpamapS, that looks great, i was figuring we could do something like a shared local package cache onto a block dev, we overlayfs into the containers18:22
SpamapShazmat: lxc-clone perhaps? ;)18:23
hazmatSpamapS, well i don't really want to clone the entire container, i just want a primed package cache for any given container18:23
SpamapShazmat: the default container builder only builds the mini-buntu once.18:24
hazmatSpamapS, i know.. but it would be nice to extend that to common packages we might install.. i mean if i have 20 units of wordpress... i really don't need to download apache/php/wordpress times18:24
SpamapSIMO there's a missing hook command which is the "I'm ready to be turned into an image" which is executed whenever any common operations (like package installs) are done.18:25
hazmatSpamapS, perhaps.. its a little tricky... an image implies a static base point, but from the moment of deploy a formula may be launched with different configs.. it might work within a service, for a quick new unit, on the same machine...18:31
SpamapShazmat: it should work within every service if the formula author identifies the point at which they have done no more unique configuration.18:32
SpamapSso during 'install' you say 'ready-for-imaging' right before you start services or record hostnames or anything like that.18:32
hazmatbcsaller, we totally have to hit avatar's again... life changing18:32
SpamapSThen the provisioning agent can snapshot your EBS, make an image of it, and every add-unit to that service is super fast.18:32
hazmatSpamapS, good point.. assuming volume management.. which we'll probably need to tackle next cycle18:34
SpamapSif there's no volume management, then the 'ready-for-imaging' command could simply do the bundling itself.18:34
SpamapSThinking in terms of real use cases vs. playing w/ ensemble.. the faster add-unit is, the more useful it is to people... they won't mind the first deploy being 10 minutes if every add-unit after that is 40 seconds.18:35
hazmatSpamapS, so there are faster tricks we can do to get there18:35
hazmater.. not faster in terms of clock time, but in terms of dev time18:36
hazmati figured out the easiest way to preallocate machines just using the existing infrastructure, is to have  our bootstrap setup the zk tree with however many nodes we want, and they'll be created as standby nodes for new deploys after the first deploy18:37
SpamapSanyway, to your point about containers.. I think lxc-clone is what you want18:37
SpamapShazmat: yeah that would be huge.. to be able to decouple machine creation from deploy/add-unit18:37
hazmatSpamapS, its basically in the vision of a min/max environment config18:38
hazmatwhere you spec min/max machines you want to pay for18:38
hazmatSpamapS, its starts to fall down because we're doing dynamic port management18:38
hazmatso we can't ever do placement to avoid conflicts, since we have no static analysis capabilities upfront at formula deploy time18:39
hazmati don't really see the need for dynamic port management.. but otoh.. getting true density is going to need some network trickery anyways18:39
robbiewokay folks....gotta fire in some nearby woods...railroad tracks are a good barrier between the woods and my neighborhood, but I need to go afk for a bit until I know it's safe18:42
hazmatrobbiew, good luck and be safe18:43
niemeyerrobbiew: Ugh :(18:43
SpamapSAustin dealing with the same type of fires we had 3 years ago. Nasty stuff.18:44
hazmatSpamapS, re containers, i'm pretty sure lxc-clone isn't what i want... i really just want a local deb repo... for offline ensemble env deploys, the reprepo looks spot on for it, thanks18:44
SpamapSrobbiew: be safe18:44
SpamapShazmat: ok, cool. :)18:45
* hazmat goes back to working on presentation18:45
niemeyerIt surprises me a bit that we have no required/optional flag in the config 19:20
SpamapSagreed19:21
SpamapSthere will be plenty of services with no sane default config.. they should error at deploy.19:22
SpamapSalright rev 336 uploaded to ubuntu19:23
niemeyerSpamapS: Hmm19:24
niemeyerSpamapS: That was probably the rationale for the choice19:24
robbiewfire contained....all is good19:25
niemeyerSpamapS: The service should always have a sane default19:25
robbiew:)19:25
niemeyerrobbiew: Phew19:25
SpamapSniemeyer: there's no sane default title for a blog. :)19:25
SpamapS"Abandoned Server" maybe19:25
niemeyerSpamapS: Indeed, but it's fine to define it afterwards19:25
SpamapS"TELL MY OWNER TO CHANGE ME"19:25
niemeyerSpamapS: Exactly! ;-)19:25
robbiewniemeyer: indeed...was bit dangerous for awhile, but the fire chief lives in our neighborhood...so I'm thinking that helped get resources here quickly :D19:26
SpamapSniemeyer: we run into this all the time in packaging.. some services need deep configuration before being started.19:27
niemeyerrobbiew: As some say, it's important to know everything, but just to have the phone number of someone who does ;)19:27
niemeyers/it's im/it's not im/19:27
SpamapSmysql you can really shoot yourself in the foot if you don't tune it first19:27
SpamapSpostgres too19:27
SpamapSbecause the initial database creation can change the way the thing works.19:27
niemeyerSpamapS: Sure, and that's where Ensemble gets in :)19:28
SpamapSI guess my point is, deploying it w/o configuration would result in something that can't be related to things19:28
niemeyerSpamapS: Hmm.. I don't get it..19:28
niemeyerSpamapS: Why would we build a formula that can't be related to things?19:29
SpamapSThere are defaults, but they get you in trouble.19:29
niemeyerSpamapS: Sounds like we should change them?19:29
SpamapSniemeyer: if you don't turn on binary logging in mysql.. you can't relate to slaves.19:29
SpamapSbut if you turn it on, you double your write load19:29
SpamapSwhich you may not want to do if you have chosen drbd as your HA method19:29
niemeyerSpamapS: Sounds like an easy choice to make.. enable by default.. we don't support drbd19:30
niemeyerSpamapS: and offer a flag for people to tune19:30
niemeyerSpamapS: Those who want something else, can have it19:30
SpamapSWhat if you do your replication w/ rabbitmq19:30
SpamapSmeh19:30
SpamapSIts not critical19:30
niemeyerSpamapS: :-)19:30
SpamapSI'm just saying that often times this is what annoys sysadmins about packaging.. that the packagers make too many decisions for the user.19:31
SpamapSIn fact19:31
SpamapSwhat we need isn't optional/required19:31
SpamapSbut priority19:31
SpamapSBut thats later, when we have configurators and not just config.yaml .. n/m19:32
_mup_ensemble/rename-shutdown-command r340 committed by jim.baker@canonical.com19:48
_mup_Merged trunk19:48
hazmatniemeyer, re config required/optional.. how about runtime relation declarations20:01
hazmatrobbiew, awesome, looks quite serious from a distance20:01
_mup_ensemble/rename-shutdown-command r341 committed by jim.baker@canonical.com20:02
_mup_Fixed review items20:02
hazmatwe just mod the formula state serialization directly to add the additonal rel endpoints20:02
niemeyerhazmat: How's one related to the other?20:03
hazmatgood question :-)20:04
niemeyer:)20:04
hazmatthere both relating to runtime configuration of a service20:04
niemeyerhazmat: Good try ;-)20:05
hazmatsay i have a generic wsgi/rails/node deploy formula.. pulls from vc... etc. based on configuration.. figures out what the particular service dependencies are for this app  and then updates the relation declarations20:05
hazmatatm, i have to declare the entire world of possibilties as optional relations20:06
niemeyerhazmat: Yeah, but there are some important reasons for that20:06
niemeyerhazmat: I know what you mean, but we'll have to dig deeper to see how to model that kind of relationship20:07
niemeyerhazmat: It's not sensible to simply dynamically show up with new relationships20:07
niemeyerhazmat: The admin would be surprised, the formula itself would become weird (how can a new relation be introduced if there are no files expecting for it), etc20:07
_mup_ensemble/trunk r339 committed by jim.baker@canonical.com20:08
_mup_Merged rename-shutdown-command branch [r=niemeyer,fwereade][f=838215]20:08
_mup_Renames shutdown command to destroy-environment.20:08
hazmatniemeyer, dynamic relations seem more 'sensible' to me then dynamic ports... everything is a tradeoff..  dynamic over static looses analysis,  i mean the admin is configuring  the formula to point to a particular platform app.. 20:09
niemeyerhazmat: Awesome.. we've managed to talk about dynamic ports and required/optional config settings in the context of dynamic relations :-)20:10
hazmat:-)20:10
hazmati should eat my lunch while its warm 20:10
niemeyerhazmat: Enjoy!20:10
niemeyerbcsaller: Is it the case that all of the inputs to config.validate are string=>string?20:46
niemeyerbcsaller: If you remember that from the top of your head at all.. I can look otherwise20:46
bcsallerniemeyer: I thought conversions came out of the validate process20:53
niemeyerbcsaller: Yeah, that's what I think as well.. cool20:53
_mup_ensemble/go-formulas r16 committed by gustavo@niemeyer.net21:39
_mup_Added config validation support.21:39
_mup_Bug #843299 was filed: Formula config in Go port needs validation support <Ensemble:In Progress by niemeyer> < https://launchpad.net/bugs/843299 >21:43

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!