[03:34] jamespage, marcoceppi - ugg. so that merge fixed our use case, but broke all other use cases. please review @ https://code.launchpad.net/~1chb1n/charms/trusty/cinder/next.ephem-key-error/+merge/266826 === moqq_ is now known as moqq [07:47] beisner, landed === Spads_ is now known as Spads [09:17] ddellav, reviewed - couple of niggles - also amulet tests are failing - not had time to dig yet. [10:51] jamespage: Hi [10:51] apuimedo, hello [10:51] I was looking at the next version of nova-compute [10:52] * jamespage nods [10:52] did I get this right, that it will run nova-api-metadata on each machine if neutron-openvswitch sends it the shared metadata secret [10:52] ? [10:53] apuimedo, yes - that was added to support neutron dvr for ml2/ovs [10:53] infact I think that's in the stable charm as well [10:53] that was a 15.04 feature [10:54] aha [10:55] apuimedo, neutron-openvswitch will also setup and configure l3-agent and metadata-agent in that particular configuration [10:55] jamespage: and the neutron-metadata-agent still runs on neutron-gateway pointing to nova-cloud-controller, right? [10:55] oh, neutron's metadata-agent will run on each compute host? [10:55] apuimedo, neutron-metadata-agent on the gateway points to itself - it runs a nova-api-metadata as well [10:55] apuimedo, that's correct [10:55] interesting [10:55] thanks [10:55] :-) [10:56] apuimedo, it only services requests for instances located on the same hypervisor [10:56] that should help with scalability ;-) [10:56] and I guess it gets the data from the rabbitmq server [10:56] apuimedo, yah [10:57] apuimedo, nova-api-metadata <-> conductor [10:57] that's a potential bottleneck still but it is horizontally scalable [10:57] I'd been considering whether we should support 'roles' for nova-cc [10:57] yeah [10:57] roles? [10:57] so you could have a scale out conductor/scheduler pool [10:58] with a smaller public facing set of API services [10:58] apuimedo, we have something like that in cinder atm [10:58] the cinder charm can do all roles, or just some of them, allowing this type of split [10:58] cinder-api/cinder-scheduler/cinder-volume [10:58] esp import for when using iscsi volumes [10:59] you want a big volume backend, with a smaller set of schedulers and api servers [10:59] I see [11:00] jamespage: sounds like you'll end up with a similar split as Kolla has [11:00] where almost everything is a separate unit [11:00] apuimedo, not that familiar with kolla - sounds like I need to read [11:01] oh - openstack/dockers [11:01] apuimedo, well I guess the bonus with juju charms is we could do a kolla like single process approach, or you can opt todo more than one thing in the same container [11:01] * jamespage likes flexibility [11:02] ;-) === apuimedo is now known as apuimedo|lunch [11:23] o/ jamespage, gnuoy [11:23] fyi in for just a few min before preschool registration, then back again. [11:25] gnuoy, i see a pile of error instances on serverstack belonging to you and to me [11:28] jamespage, thanks for the merge === CyberJacob is now known as zz_CyberJacob === zz_CyberJacob is now known as CyberJacob [13:33] jamespage, hmm, weird that amulet tests are failing, i didn't change that much. [13:33] ddellav, that might be serverstack tbh - we're having some troubles [13:33] ah yea, ok [13:51] yea, right at the bottom of the amulet output you can see it failed due to rabbitmq: http://paste.ubuntu.com/12000161/ [14:02] jamespage - raised this before we forget about it bug 1481362 < also fyi wolsen dosaboy gnuoy coreycb [14:02] Bug #1481362: pxc server 5.6 on vivid does not create /var/lib/mysql

[14:06] urgh - I think I just put ovs in a spin on all the compute hosts - sorry folks [14:06] fixing now [14:10] jamespage, yeah, connectivity lost to bastions. it's ok. you'll make it all shiny and new i know you will. [14:11] also just raised this against pxc re: deprecation warn on > vivid: bug 1481367 [14:11] Bug #1481367: 'dataset-size' has been deprecated, please useinnodb_buffer_pool_size option instead

[14:30] jamespage: is there some minimal openstack bundle that groups some charms in the same machine like nova-cloud-controler and neutron-api? [14:31] apuimedo|lunch, yes [14:31] I only saw https://jujucharms.com/openstack-base/34 [14:31] and I don't have 17 machines around :P [14:31] apuimedo|lunch, oh - I was about to point to that === apuimedo|lunch is now known as apuimedo [14:31] apuimedo|lunch, that only needs 4 servers [14:31] (check the readme) [14:31] its 17 units, 4 physical machines [14:32] ah, I must have misread the bundle file [14:32] I don't see any lxc reference [14:32] in the bundle.yaml [14:33] oh! [14:33] jamespage: why is there a bundel.yaml and a bundle.yaml.orig? [14:33] only the latter has the lxc references [14:33] apuimedo, its an artifact of the charm store injestion [14:34] so the one that should be used is the .orig, right? [14:34] apuimedo, although I don't believe it should scrub machine placement like that [14:34] apuimedo, yes [14:34] for now [14:35] rick_h_, ^^ is that right? https://jujucharms.com/openstack-base/ - the bundle.yaml has lost the machine placement data? [14:35] jamespage: looking [14:35] rick_h_, thats def changed - it used to be fewer machines than units, but not any longer [14:36] urulama: rogpeppe1 ^ looks like lxc got lost in the transition from v3 to v4? [14:36] * rogpeppe1 reads back [14:38] rogpeppe1: basically https://api.jujucharms.com/charmstore/v4/bundle/openstack-base-34/archive/bundles.yaml.orig vs https://api.jujucharms.com/charmstore/v4/bundle/openstack-base-34/archive/bundle.yaml it lost the placement [14:38] rick_h_: yeah, i see that now. just investigating [14:39] jamespage: with the deployer release tvansteenburgh is doing today we can do true machine placement in the new bundle. We'll look into the bug, but the best way forward, once that deployer is out there, is to check out the 'machines' part in https://github.com/juju/charmstore/blob/v5-unstable/docs/bundles.md [14:39] rick_h_, jamespage: i see what has happened [14:39] jamespage: and get rid of bundles.yaml and go to bundle.yaml as the only file in the bundle. [14:39] jamespage: i wasn't aware (and i don't think it was documented) that the "to:" field in a legacy bundle could be list as well as a string [14:40] eeek! [14:40] rogpeppe1: urulama that's a bit :( as openstack is preparing new release and their bundles are kind of busted atm now [14:42] coreycb, thanks for the merge :) [14:43] i'm quite surprised that the goyaml package allowed unmarshaling of a list into a string (it seems to have just ignored it) [14:43] rick_h_: i'd say +1 on the quick fix of using bundle.yaml until we come up with a solution [14:43] jamespage: can the to: be representing as a string and repushed to get through a fix until the charmstore can be updated? [14:44] beisner, you're welcome, thanks for the updates [14:44] jamespage: hmm, looks like not with ceph/nova compute [14:45] urulama: the problem is that it's not supposed in the deployer yet until the release is out and folks get it. Kind of rock/hard place atm. /me wonders if you can do ceph, ceph, ceph for the nova-compute one...where's that doc [14:46] rogpeppe1: http://pythonhosted.org/juju-deployer/config.html#placement is the docs for that and the wordpress example has the lists. [14:49] rick_h_: it's possible i skipped over that syntax because there were no bundles around that actually used it (that I could find - there are none in the corpus used for testing migratebundle) [14:50] is it possible in bundles to define a config that applies to all the charms that share the setting [14:50] like openstack-origin ? [14:51] apuimedo: certainly. there's an overrides: key that you can use [14:51] lazyPower: at the same level as "services" ? [14:52] apuimedo: its a parent level key, same topical level as "services" [14:52] good [14:52] thanks [14:53] jamespage: in the new format, the syntax would be: [14:53] to: ["lxc:ceph/1"] [14:53] I wonder why the bundle.yaml.orig doesn't use it [14:53] apuimedo: something like this - http://paste.ubuntu.com/12000543/ [14:53] jamespage: (the unit syntax being the same as juju's usual unit syntax) [14:53] I seem to remember that it was in some previous versions [14:53] rogpeppe1: is this new syntax on stable? [14:55] apuimedo: "on stable" ? [14:55] apuimedo: the new bundle syntax is considerably stripped down from the old syntax (and somewhat more general in places too) [14:58] I mean the stable bundle deployer [14:58] when you install juju on trusty [14:59] apuimedo: it's been in trunk for a long time. tvansteenburgh is working on a release. It's not in the current trusty :( [14:59] beisner, is there a bit of dellstack I could use to refresh the charm-store bundle? [14:59] ok [15:00] rick_h_: does that mean that bundles will have to be updated? Is there some degree of backwards compatibility? [15:00] apuimedo: yes, it'll support both but we'll be working on deprecating v3 and disallowing new v3 ones to the charmstore [15:00] apuimedo: the format is only slightly different. It's mostly the same, just removes the top key from the file [15:01] most bundles will work with a delete of the first line and dedent [15:01] ok [15:02] rick_h_: does the "to" allow you to put two charms on the same container scope, or only when one of them is subordinate? [15:02] apuimedo, rick_h_: but placement is thing that has changed most [15:02] the ting [15:02] the thing [15:04] apuimedo: the format does. i'm not sure about the deployer implementation. [15:06] rick_h_, urulama, rogpeppe1: working on moving to bundle.yaml now [15:07] jamespage: Makyo can help with that [15:07] lazyPower: the bundles that have "0" in a bundle are deployed? [15:07] the one for openstack has a few charms like that [15:08] apuimedo: i'm sorry i dont follow. [15:08] rick_h_, Makyo: first cut here - lp:~james-page/charms/bundles/openstack-base/bundle [15:08] rick_h_, is there a nice way I can programatically query for the latest charm revisions? [15:09] apuimedo: can i get a link to the bundle you are looking at? [15:09] jamespage: sure thing https://api.jujucharms.com/v4/ceph/meta/any [15:09] rick_h_, ta muchly [15:09] sure [15:10] lazyPower: https://api.jujucharms.com/charmstore/v4/bundle/openstack-base-34/archive/bundles.yaml.orig [15:10] jamespage: if you need just revision, you can use this as well https://api.jujucharms.com/v4/ceph/meta/id-revision [15:11] you'll see that there's quite a few with units: 0 [15:11] apuimedo: in the case of subordinates, thats a requirement [15:11] jamespage: that first line looks potentially spurious [15:11] jamespage: did you mean "series: trusty" there? [15:11] lazyPower: ah, good :-) [15:12] however i'm not sure why neutron-openvswitch has num_units: 0 [15:12] its not a subordinate that i can see [15:13] i lied, line 2 - it sure is [15:13] so, yeah - everything in here with num_units: 0 is due to the charm being a subordinate apuimedo [15:13] it is [15:13] ;-) [15:13] thanks lazyPower [15:14] np [15:14] mbruzek: I was in your ancestral town on Sunday [15:14] took a panorama picture from the bus [15:14] apuimedo: wow! [15:14] apuimedo: Cool! [15:15] jamespage, First glance: first line should be 'series: trusty' and last line should be removed, but it looks okay otherwise. [15:15] I wish they would have stopped so I could buy pastries [15:15] Er, sorry. First line should be removed entirely, last line can stay. [15:15] jamespage, ^ [15:15] got it [15:17] Makyo, how do I actually test this? [15:18] jamespage, one sec, spinning up a GUI with latest code; dragging the yaml there will run it through the bundle validation. [15:19] jamespage, Actually, you should be able to use demo.jujucharms.com - dragging the bundle.yaml onto the canvas should validate the bundle. [15:20] jamespage, (recent updates to the GUI affect committing uncommitted bundles, which is unrelated) [15:22] Makyo, error - console? [15:22] I'm such a web brower numpty - apologies [15:25] jamespage, oops, hmm, that's partly our fault. That message should be changed for that site. Running it locally, pastebin in a sec. [15:29] apuimedo: please send me the picture if you would. [15:32] It's a bit moved [15:36] jamespage, Here's a working v4 bundle http://paste.ubuntu.com/12000801/ (v4 bundles need a machine spec, even if it's empty, to allow placement directives like "lxc:ceph/0") [15:37] jamespage, I validated that with https://github.com/juju/juju-bundlelib (git clone; make; devenv/bin/getchangeset bundle.yaml) [15:58] jamespage, The only thing that might be a problem is that I had to include juju-gui in order for one placement to work, which may not fly when deploying via the gui. [15:59] Makyo, updated again [16:00] (I wrote a small parser to query the charmstore api and update revisions) [16:12] jamespage, Here's one that passes validation: http://paste.ubuntu.com/12000995/ We have to use the placement directives from the older style of bundles to reference the bootstrap node (cc rick_h_ urulama ) [16:12] jamespage, we also no longer have a top-level YAML node ('openstack-base') in the charmstore [16:13] Makyo: oh, yea bootstrap node is a no no [16:14] rick_h_, Meaning I should take it out? [16:14] Makyo: no, meaning that I expect that to not be pretty [16:14] rick_h_, ah, alright. Yeah, if we're referencing the bootstrap node, we have to use v3-style placement directives, otherwise it gets lost looking for a machine named "0" [16:17] Makyo, ok - that's validating now [16:18] jamespage, Awesome === Guest47499 is now known as anthonyf === anthonyf is now known as Guest410 === CyberJacob is now known as zz_CyberJacob === zz_CyberJacob is now known as CyberJacob [17:51] jamespage or gnuoy - almost forgot this puppy. please see/review/land: https://code.launchpad.net/~billy-olsen/charms/trusty/rabbitmq-server/ch-sync-cli-fix/+merge/266619 [17:51] wolsen, fyi ^ [17:52] beisner, last vote is that amulet fails still for it - though it does fix the import issue [17:52] wolsen, ack. my vote is to merge as-is, on the basis of fixing an import issue. and address pre-existing failure of that 1 test separately. [17:52] beisner, but I would be +1 on removing the import - yeah [17:53] wolsen, that may or may not still happen, but as it's syncd into all of the other os-charms, that is the state of things. [17:54] wolsen, ie. we won't very well be able to t-shoot the functional test with the import error. [17:54] beisner, if jamespage, gnuoy, or dosaboy don't respond in short order I'll land it [17:54] wolsen, ack, tyvm [18:09] dosaboy, wolsen - thanks fellas === Guest410 is now known as anthonyf === anthonyf is now known as Guest26675 [19:10] coreycb, fyi, dashboard 051-basic-trusty-juno-git is failing atm http://paste.ubuntu.com/12002116/ so far, that's the only *git* test failing in today's pre-release checks. [19:13] beisner, ok I'll take a look [19:14] coreycb, ta [19:43] wolsen, ping [19:43] beisner, pong [19:43] so rmq passes locally, because the deployed instances and the bastion/host running tests are in the same ip space. it fails in uosci, because the bastion host is on a different /24 than the deployed units. [19:44] and... the undercloud (serverstack) sec group looks like this: http://paste.ubuntu.com/12002355/ [19:44] ie. packets just don't arrive [19:44] so is my theory [19:45] this hasn't been an issue, because no other os-charm tests attempt communication from the machine that's executing tests. [19:46] it's all done via charm hooks and juju run, etc., which puts traffic on the same side of the fence so to speak. [19:46] * wolsen looks at the code again [19:47] beisner, seems like a perfectly plausible scenario [19:47] what you are describing [19:53] wolsen, well, maybe not. the first 2 make it and check ok. the 3rd check, which is to send to one rmq unit, then check the other rmq unit for that message, is what fails. [19:53] idea 1 sec.. [19:56] beisner, yeah it shouldn't be the port issue since rabbitmq-server is exposed [20:01] beisner, agree with your idea- I think its likely that there's a timing thing going on [20:06] wolsen, well no dice even with a 2 min wait. [20:07] beisner, hmm may want to get some queue information to see what the hapolicy is on the queue [20:07] make sure it is what we think it is [20:08] wolsen, i think the test is fine. i think rmq or its cluster is not ok. [20:09] i've got the enviro stood up, can send msg and chk msg from same unit ok. but when i send one manually to one unit, it never arrives at the 2nd. [20:10] beisner, well actually if you look at the "cluster status", its only reporting itself in the cluster (each of them actually) [20:10] indeed wolsen [20:12] oh wait this thing is hard-coding a vip @ 192.168.77.11 [20:12] wolsen, ^ [20:13] I need haaaaalp! anyone knows a list of *all* (or at least most) ports that Juju uses? [20:14] beisner, that sounds like a bug as well, but I'm not sure its _the_ problem [20:14] beisner, as the clustering doesn't rely on a vip iirc [20:14] wolsen, right, this test isn't checking based on that, but when the tests are extended, they will need to consume the vip env var as pxc and hacluster do. [20:16] wolsen, so the earlier tests should actual fail out on this. ie. # Verify that the rabbitmq cluster status is correct. [20:16] jose, 22, 17017, and 8040 for the storage port I believe (though check your ~/.juju/environments.yaml for the storage-port) [20:17] beisner, well that would be dashing if it did fail on that [20:17] lol yep wolsen [20:19] thanks wolsen [20:23] wolsen, so this is my first dive into the rmq test. [20:23] so if i deploy 2 rmq units, do they just know to cluster togther, like bunnies? [20:27] beisner, well if 2 bunnies got together you'd have a lot more than a cluster :P [20:27] beisner, but essentially, that's the theory [20:27] iirc, there will be an exchange of erlang cookies and the configuration files updated [20:30] beisner, do you have hook logs? [20:31] yeah but destroy, redeploying ... [20:36] wolsen, i mean yah, we have a dozen or more jenkins jobs, all with full logs and etc pulls. [20:36] beisner, duh, should've looked there thx [20:38] wolsen, lol my proposal with the delay just freakin passed test 20. http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/5639/console [20:38] live view ^ [20:39] beisner, with the 2 minute delay? [20:39] wolsen, 30s [20:39] wolsen, the 2 min was me doing it manually [20:39] beisner, well most importantly - it clustered [20:40] wolsen, i really want to refactor these tests. we're doing cluster tests inside one named relation check, etc. [20:40] beisner, +1 [20:40] wolsen, and check for presence of all units in the cluster check, instead of just checking that the command succeeds. [20:41] yep [20:47] bump bummmm. 30 passes [20:54] dang. #40 failed [21:00] ping jose [21:07] jcastro: ping [21:17] wolsen, for test 40 (just 2 rmq units), the cluster-relation-changed hook is never triggered (which looks to be by design). we get an install hook, and a config-changed hook. so afaict, it should not be expected to form a cluster. [21:23] cluster-relation-joined rather [21:51] 1.24 doesn’t use differnt ports does it? i ran an upgrade from 1.23 to 1.24 and all the machines worked as expected except one, and it’s stuck on 1.23 and when i start the agent it fails to connec to the master on 17070 [21:52] ah [21:52] nevermind [21:52] * moqq facepalm [22:00] hello marcoceppi. There have been several changes to charm-helpers that I am interested in, can you make a release to pypi? [22:00] or authorize me to do such a thing [22:17] moqq are you still having high CPU issues on 1.24? [22:30] Hello. [22:30] I need to install a deb in my tarmac machine, but the tarmac charm doesn't allow for this. [22:30] should I extend the charm to get deb urls to install, or just remember to install it manually in case I have to redeploy it? [22:33] elopio: extending the charm is one way, another way, if it's just a deb taht needs to be added, is to create a subordinate charm that only has an install hook which installs that deb [22:34] that way you don't have to fork the main charm and you can pack any customizations into that subordinate charm [22:34] marcoceppi: I'll try sending my change upstream. If they don't apply it soon, the subordinate seems good. [22:42] sebas5384: yo!