=== varud_away is now known as varud === varud is now known as varud_away === varud_away is now known as varud === varud is now known as varud_away === varud_away is now known as varud === JoseeAntonioR is now known as jose === varud is now known as varud_away === varud_away is now known as varud [03:42] fwereade: Still there? [03:50] fwereade: I'm not sure exactly what you mean by early departure for peer relations regarding Bug #1192433, but if it stops the unit being listed in relation-list while all the -depart and -broken hooks are running that addresses the bug. [03:50] <_mup_> Bug #1192433: relation-list reporting dying units

=== varud is now known as varud_away [03:52] fwereade: Ordering of provides/requires actually sounds more interesting to me. For example, it would have addressed Bug #1187508 [03:52] <_mup_> Bug #1187508: adding a unit of a related service to postgresql is racy [03:53] fwereade: If the provider's join hook was run before the requirer's join hook, then I can guarantee that access from the require has been granted by the provider before the requirer attempts to connect. This makes the client charms much easier to write. [04:17] stub: I'd be a slight departure from the current story, not that I don't have a problem with that perse, but the idea I've gotten enstilled as to what each hook means is as follows: [04:19] joined - "handshake" hook, expect to have no data available [04:19] changed - "stuff on the wire", alwasy check values, idempotency [04:19] departed - "waving goodbye", relation values should still be available, do what you need to with these values [04:19] broken - "clean up", make any changes post-mortem [04:19] how does having the clients joined/changed hook run prior to the providers create a race condition? That just seems like non-idempotent hooks to me [04:20] * marcoceppi reads bug report [04:20] marcoceppi: Lets say you have an existing setup, with a single unit of service A related to a single unit of service B. [04:20] marcoceppi: Now add a new unit to service A (the requirer or client). [04:21] marcoceppi: It looks at the relation, picks up authentication credentials, and attempts to use them *before B has had a chance to authorize the new unit's IP address*) [04:21] right, I see. db-admin relation data is already set because that unit's done it's thing, but relation-changed needs to run on the postgresql side to complete the setup [04:22] stub: I'm inclined to say this is a bug with juju, from my understanding the data should be unique per unit <-> unit (though core may need to correct my understanding of this) [04:22] In this case, each unit should get it's own credentials, not the credentials cached from the prior units relation [04:23] and should follow the normal hooke execution as if it was a new add-relation [04:23] marcoceppi: Yes, you could make that argument. You could even make it somewhat backwards compatible if there was relation->unit and unit->unit data, with the unit->unit data overlayed over the top of the relation->unit data. [04:24] * marcoceppi nods [04:24] I don't think we would see a change like that until 2.0+ though, even if it was agreed on today. [04:25] I feel like this would resolve your issue and is what the expected behavior of juju, at least the first part of a fresh relation each time [04:25] Possibly [04:25] In the mean time your patch for postgres landed (while not the best answer to the question) [04:26] ta :) [04:27] I'd hope this could be documented and solidified either as expected behaviour or fixed in core soon to represent the actual anticipated behaviour [04:27] I'll poke #juju-dev tomorrow [04:27] "tomorrow" [04:30] marcoceppi: The documentation certainly needs to be clearer. I think everyone has troubles when they start dealing with more complex charms, and end up writing test cases to work out wtf is actually happening. [04:31] stub: yeah, even during my demo last week I was like "wtf is going on" and had to step through debug-hooks as if it was part of the training, only so I could know understand what was actually going on [04:31] marcoceppi: I don't know if the change you propose to relation storage will make things better conceptually or not. [04:31] I'd like to think so, but I may be biased [04:32] It means that to retrieve data from a specific bucket in the relation, two units need to be specified rather than one. [04:34] I like specifying the ordering of hooks. I don't think it will slow things down much in the real works, and greatly reduces the number of states hooks need to cope with. [04:34] c/in the real works/in the real world/ === varud_away is now known as varud [04:45] stub: I don't see how you'd have to specify two units? [04:46] I think relation-get always assumes the current unit, as you technically can't/shouldn't be able to spy on other units [04:46] marcoceppi: Right. But how do you retrieve the current units data? The data on its end of the relation? It has one bucket per remote unit. [04:47] stub: in the relation-* hooks $JUJU_REMOTE_UNIT is prefilled and used by the relation-* commands [04:47] it's just pulled from env [04:47] ok, so you would make it impossible to access the other buckets [04:48] peer relations certainly need to spy on other units data [04:48] stub: With the exception of peer relations, I think the idea is you shouldnt' be able to tap in to other relations that the unit isn't aware of [04:49] peers are a bit different, but still follow the same event cycle, it's just each unit is individuall connected to each other so you could relation-get -r and get data as you're technically connected to it [04:51] * marcoceppi heads to bed [04:52] Goodnight === varud is now known as varud_away [04:52] I'll continue chasing race conditions from the hoops I have to jump through because there is no defined order that my hooks get called :) [04:53] A maze of twisty hooks, all of them alike. === varud_away is now known as varud === varud is now known as varud_away === varud_away is now known as varud === tasdomas_afk is now known as tasdomas [07:41] stub, heyhey [07:41] fwereade: heya [07:42] stub, so my current plans are to cause peer units only to (appear to) depart the relation as soon as they're dying [07:43] stub, because that directly addresses the case in your bug (for which many thanks, btw) [07:43] That sounds fine to me. [07:45] stub, it is not a watertight guarantee of ordering, though -- any given unit might be busy with, say, a long-running config-changed hook while the remote unit is departing [07:46] stub, and *could* thus still end up only observing the departure after it actually happened [07:47] Yes, if other hooks are allowed to run at the same time, then anything that the hook knows may be a lie. It only has a snapshot of the environment from when it started. [07:48] I'm just avoiding that whole issue at the moment in places where it is important, such as failover. [07:48] stub, I don't think that there is much realistic hope of enforcing the sort of lockstep ordering I think you would ideally like to see across the whole system [07:48] So if you remove-unit a master, you are going to blow your foot off if you are silly enough to make other changes before it is finished. [07:49] fwereade: I agree lockstep accoss the system is probably not a good idea. However, I'm thinking the sequence of relation joined/changed/broken/departed would be helpful to write less buggy charms. [07:50] stub, however I am very keen to make it *easier* to write charms that are sane and effective [07:50] The number of possible states even a simple relationship can be in is very large, and some of those states are rare (eg. a unit is being removed before another units -joined hook has started running). [07:52] stub, are you familiar with https://juju.ubuntu.com/docs/authors-charms-in-action.html ? I wrote it quite a long time ago but I don't think it's had quite the exposure it should have (I wasn't paying enough attention to the user-facing docs, I kinda threw what I'd written over the wall and didn't follow up) [08:28] urgh... proxy seized up === defunctzombie is now known as defunctzombie_zz === defunctzombie_zz is now known as defunctzombie === defunctzombie is now known as defunctzombie_zz [09:19] stub, sorry, didn't see you come back [09:20] stub, not sure if you saw; I asked if you'd seen https://juju.ubuntu.com/docs/authors-charms-in-action.html ? [09:26] fwereade: yes, I've seen that [09:28] absorbed it, maybe not :) I see it is describing the current departed/broken/actually leaves behavior [09:32] hey guys, does ppa:juju/stable (1.12 I think) support completely offline maas+openstack deployment? [09:35] fwereade: In case it didn't get through, I said before that my highest priority issue is Bug #1200267 to make my tests reliable. The hook ordering stuff is all complex and tricky, but the charm I'm working on is complex and tricky. [09:35] <_mup_> Bug #1200267: Have juju status expose running and pending hooks [09:39] stub, ok, that's interesting... there's a bit of a problem there in that it's impractical to try to track the possible future, but it would be possible for the unit agent to report whether or not it's *currently* idle [09:40] stub, and knowing that every unit in play is currently idle would probably be sufficient -- *at the moment* -- to determine that the system's in a steady state [09:40] ehw: we may have bugs in that area (especially with charms themselves), but with some config, I think stable has enough to work disconnected from the internet. You'll need to do stuff like get tools into the local cloud first, etc. [09:41] fwereade: Yes. If no hooks are running, and no hooks are queued, we are steady. [09:41] stub, I'm just not sure how useful that guarantee is in practice -- there's always the possibility that some other user will add a unit, or change config, or something [09:41] fwereade: This is for writing tests. [09:41] jam: is there a way to get the tools in? are they stored in MAAS? [09:42] * ehw has seen the FileStorage bits in MAAS, but they don't seem connected to anything [09:42] ehw: there is a 'juju sync-tools' command. I believe in the 1.12 stable release it always downloads them from amazon, in 1.13 there is a "--source" flag so you can copy them from your local disk. [09:42] ehw: there is also juju:ppa/devel if you find 1.12 d [09:42] doesn't work [09:42] stub, and if in the future we add a juju-run mechanism (ie allowing a unit too feed info back into juju outside of a regularly scheduled hook) it's almost no guarantee at all [09:43] stub, ok, the test context makes it much saner [09:44] jam: ok, I'll see if I can try this out with 1.12 [09:44] stub, is it very important to know *which* hooks are queued/running, or is a single bit of steadiness info per unit sufficient? [09:45] fwereade: A single bit of steadiness is just fine for my use case. [09:45] ehw: so I'm pretty sure that for 1.12 the client (the place you are running 'juju *' from) needs at least some internet access. Once you're set up, I don't think the actual MaaS nodes need public internet access (ex [09:45] except for some specific charms that talk out the network, etc) [09:45] It could even just be 'juju wait', that blocks until a steady state is reached. Then I don't even have to bother with polling juju status. [09:46] jam: have a deployment next week in a secure environment; working on a methodology for this atm [09:47] stub, that feels like overreaching on juju's part, considering the possibility of juju-run, which prevents us from making that guarantee -- I would be much more comfortable with exposing current steadiness per unit, with the caveat that the information can only be acted upon if you have a lot of specific knowledge about the environment [09:48] fwereade: ok [09:51] I just need some way to tell that it is ok to proceed with making the test, rather than the current approach of sleep() and hope :) [09:55] fwereade: Since we are talking about this only being useful for tests, this might sit better in amulet if there is some way of reaching into juju and inspecting what needs to be inspected. [09:57] (the current implementation of wait there just parses the 'juju status' output, similar to my own test harness) === defunctzombie_zz is now known as defunctzombie === defunctzombie is now known as defunctzombie_zz [10:13] juju-run might fix this horrible bit of code. For a certain operation, I need to block until the database is out of backup mode. It might be in backup mode because backups are being run, or it might be in backup mode because a backup started and failed to complete. The charm currently emits details and instructions how to clear it manually to the log file every few minutes until it is unblocked. [10:14] Hmm... I can see this turning into twisted :) === dosaboy_ is now known as dosaboy_afk === tasdomas is now known as tasdomas_afk === rogpeppe1 is now known as rogpeppe [14:52] Reminder: Charm meeting in ~1 hour! [14:53] marcoceppi: marcoceppi utlemming arosales ^^^ === freeflying is now known as freeflying_away [15:01] I'm using ppa:juju/devel and looking at "juju debug-log"... [15:03] jcastro, ack and thanks for the reminer [15:03] *reminder [15:23] jcastro, what time is the charm call? [15:24] jamespage: top of the hour [15:24] jcastro, ack [15:24] ~40 minutes [15:53] arosales: pad is up to date and ready! [15:54] jcastro, thanks are you setting up the hangout? [15:54] yep [15:58] http://ubuntuonair.com/ for those want to follow along in the meeting [15:58] https://plus.google.com/hangouts/_/96e374b291c31f8310be00593580b52eb8418b02?authuser=0&hl=en if you want to participate [16:01] * m_3 having hardware issues with latest nvidia updates :-( [16:04] m_3: yaeh, those killed me last night === varud is now known as varud_away === varud_away is now known as varud [16:23] marcoceppi: maybe write that as a full deploy line [16:24] jamespage, jcastro also this bug we need to address in precise https://bugs.launchpad.net/juju-core/+bug/1203795 [16:24] <_mup_> Bug #1203795: mongodb with --ssl not available in precise

[16:25] at least figure out the story since if we can't backport mongo and lxc [16:28] jamespage: must the ceph.yaml go in the .juju directory? [16:28] kurt_, no [16:28] where should it go? [16:28] kurt_, you pass it to juju deploy with the --config flag [16:28] ah ok [16:28] juju deploy --config --config [16:29] juju deploy --config config.yaml [16:29] its unnecessary after the configuration, correct? [16:29] sorry, deployment [16:29] is openstack swift required for juju integration? | http://askubuntu.com/q/335447 [16:35] jamespage: is the ceph.yaml in scuttlemonkeys guide (http://ceph.com/dev-notes/deploying-ceph-with-juju/) up to date? I'm looking specifically at the ceph-osd source and device information and ceph-radosgw source. [16:35] jcastro, jamespage http://bazaar.launchpad.net/~openstack-charmers/+junk/openstack-ha/view/head:/python-redux.json [16:36] and I'm wondering in the source string for gui for ceph would then be changed to quantal to try it that way? [16:43] kurt_, precise is fine [16:43] re source string for gui [16:43] er default [16:46] hazmat: default in gui for source string is "cloud:precise-updates/folsom" [16:47] kurt_, oh sorry you meant for ceph within the gui, not for deploying the gui [16:47] hazmat: my questions above may seem confusing, sorry - I had separate ones for CLI and gui methods [16:49] kurt_, if your deploying the precise charm that's a good default .. its the charm that determines which distro release your getting, so you want the source string to match the charm series name. [16:49] CLI - my questions were on the ceph.yaml and for the gui my question was about the source and now looking at it, the odd-devices string. [16:51] ok, there appears to be no guide using the gui to deploy ceph. I can see there is a separate charm for ceph-radosgw, ceph-osd, and ceph [16:52] i should probably just stick to manual method for now [16:56] Is there any important data stored on ephemeral storage? Use Case: Stopping AWS bootstrap and staring it again. === BradCrittenden is now known as bac [17:04] hazmat: bah I lost the URL, deployer docs are at ... ? [17:07] http://pythonhosted.org/juju-deployer/ [17:07] ta [17:08] hazmat: so this doesn't explain how to use it though, no examples in the man page either [17:08] assuming I have example.yaml as an exported bundle how do I deploy it? [17:08] jcastro, good point. juju-deployer -v -W -c example.yaml name_of_bundle [17:14] hazmat: aha! I get an error when trying to deploy this: http://bazaar.launchpad.net/~openstack-charmers/+junk/openstack-ha/view/head:/python-redux.yaml [17:14] jcastro, which is? pastebin? on maas? [17:15] http://pastebin.ubuntu.com/6011028/ [17:16] scuttlemokey: I have some updated instructions for the monitor-key generation section for your ceph documentation [17:16] http://pastebin.ubuntu.com/6011030/ [17:18] jcastro, that reads like an error in the config file [17:18] jcastro, ie. its not valid yaml [17:19] hazmat: ah nm, I found the problem [17:19] jcastro, html instead of yaml ? [17:20] yeah so if you wget that page from lp you get the html [17:20] duh [17:20] jcastro, yeah.. just verified that the python-redux.yaml is correct [17:20] er.. is valid yaml [17:28] heya kurt_ you gotta check this out, you still have maas? [17:29] yep. [17:29] snag that yaml file [17:29] then get juju-deployer from ppa:juju/pkgs [17:29] http://pastebin.ubuntu.com/6011079/ [17:30] jcastro: will this tear up what I have running currently? [17:30] It will probably clobber the universe [17:30] jcastro: wait, is juju-deployer in juju/pkgs? [17:31] actually, I have it in saucy [17:31] I've seen it in juju/pkgs [17:31] its in the distro [17:31] in saucy i think [17:31] jcastro: that's not the right version [17:31] I need to remove that [17:31] jcastro: shall I wait then? [17:32] yeah [17:32] jamespage: it bombed out during this part: http://pastebin.ubuntu.com/6011085/ [17:33] do I need to have a bunch of locally branched stuff? [17:33] I've never heard of switf-storage-z1 [17:33] marcoceppi, 3 swift nodes [17:34] juju-deployer also in a daily ppa https://launchpad.net/~ahasenack/+archive/juju-deployer-daily with dep https://launchpad.net/~ahasenack/+archive/python-jujuclient or from pypi.. virtualenv --system-site-packages deployer && source deployer/bin/activate && pip install juju-deployer [17:35] about swift-storage, the charm is called swift-storage, but the config deploys it as swift-storage-zN [17:36] where N is 1, 2 or 3 (if we are talking about the openstack juju-deployer config file) [17:36] so I think that config is missing a "charm: swift-storage" entry under swift-storage-z1 [17:36] 2013-08-21 13:29:48 Deploying service swift-storage-z1 using local:precise/swift-storage-z1 <-- wrong [17:37] should be [17:37] 2013-08-21 13:29:48 Deploying service swift-storage-z1 using local:precise/swift-storage [17:39] evilnickveitch jcastro: I also found out there's about 25 or so extra undocumented environments.yaml config options for juju-core [17:40] okay, 25 might be an overestimate, but there's a quite a few [17:43] jamespage: in looking at your cephalopod article - do you have precise instructions instead of quantal? [17:51] ahasenack, fwiw that's fixed in deployer trunk, charm: becomes optional, it will use the name in the charm md if none is specified [17:53] ok === defunctzombie_zz is now known as defunctzombie [19:57] ahasenack hazmat btw I've got two merge proposals for deployer, once is minor and cosmetic the other is a blocker for amulet: https://code.launchpad.net/~marcoceppi/juju-deployer/local-branches/+merge/181207 if you could reveiw sometime this week that'd be great! [19:59] marcoceppi, ack i'll get those merged in this evening. [19:59] hazmat: thank you sir! === BradCrittenden is now known as bac [21:10] adam_g: did you have issues with rabbitmq-server 2.7.1? it's failing to start when deploying on precise local provider [21:11] sidnei, no, its been working fine in non-local deployments for a while [21:11] sidnei, with the exception of clustering in environments with funny DNS [21:11] i guess it might be a lxc-specific failure [21:11] sidnei, what is the failure? [21:11] there's nothing interesting in the rabbit logs, only 'failed to start' [21:11] and epmd is left running [21:12] sidnei, can you resolve your localhost name and IP? [21:12] er, local hostname [21:12] :) [21:12] uhm, hostname is set to ubuntu, it might be resolving to multiple addresses [21:13] or not [21:14] well, it's probably that [21:14] $ ping ubuntu [21:14] PING ubuntu (10.0.3.15) 56(84) bytes of data. [21:14] but local ip is 10.0.3.187 [21:14] will dig further after the break [22:16] https://code.launchpad.net/~sidnei/charms/precise/haproxy/trunk/+merge/181421 up for review again [23:08] jamespages: do you have any guides for ceph deployment with cinder and openstack? === mhall119_ is now known as mhall119