_mup_juju/trunk r385 committed by kapil.thangavelu@canonical.com00:05
_mup_merge unit-info-cli [r=fwereade,niemeyer][f=863816]00:05
_mup_Allow units to obtain their public and private addresses in a provider00:05
_mup_independent fashion00:05
_mup_Bug #867991 was filed: local provider needs documentation <juju:In Progress by hazmat> < https://launchpad.net/bugs/867991 >00:09
_mup_Bug #867993 was filed: Code/module docs seem to use "j" instead of juju <juju:New> < https://launchpad.net/bugs/867993 >00:12
jimbakerhazmat, here is the master-customize.log: http://pastebin.ubuntu.com/702511/00:17
jimbakeri tried going against the latest in unit-info-cli, no changes i could see00:18
jimbakerin terms of the log or juju status'00:18
hazmatjimbaker, what's the output of > host archive.ubuntu.com192.168.122.100:19
hazmatwhoops space between those00:19
hazmatjimbaker, it looks like a network connectivity problem00:19
jimbakerhazmat, here's the output: http://pastebin.ubuntu.com/702516/00:20
hazmatjimbaker, one more.. what's the output of sudo cat /var/lib/lxc/yourusername-yourenvname-0-template/rootfs/etc/resolvconf/run/resolv.conf00:21
jimbakerhazmat, nameserver
hazmatjimbaker, odd00:23
hazmateverything sounds like its correct00:23
hazmatjimbaker, aha00:23
jimbakerhazmat, ?00:23
jimbaker(if you want i can give you a login to this box)00:24
hazmatjimbaker, so oneiric is still in beta so old packages aren't per se kept around00:24
hazmatjimbaker, you'll need to update manually your lxc cache00:24
jimbakerhazmat, ok00:24
hazmatso it has the latest oneiric packages00:24
hazmatjimbaker,  $ sudo chroot /var/cache/lxc/oneiric/rootfs-i386/00:25
hazmatjimbaker, $ apt-get update && apt-get upgrade00:25
hazmatthat should fix things, else the pkg cache was ref'ing non existant packages upstream i bet00:26
jimbakerok, doing that, but against amd64 :)00:26
* hazmat heads out for a dog walk00:26
jimbakerhazmat, makes sense, this would explained those bad refs in master-customize.log00:26
hazmatjimbaker, also you can update your juju-origin to lp:juju/trunk if you want.. its all committed now00:27
jimbakerhazmat, sounds good00:28
hazmatjimbaker, that work for you?00:56
niemeyerhazmat: Man!01:14
niemeyerhazmat: wtf is happy with unit-get!01:14
hazmatniemeyer, :-)01:14
* niemeyer dances around the chair01:14
hazmatniemeyer, cool, i'm out going to go enjoy some down time01:14
niemeyerhazmat: Enjoy it :)01:15
hazmatniemeyer, sent out some mails to the list inviting additional local provider testing01:15
niemeyerI'm going down too01:15
niemeyerhazmat: Superbsome01:15
* niemeyer => poweroff01:17
jimbakerhazmat, hmmm, the upgrade was successful of the lxc cache. looks like i have a networking issue, the agent logs (which i finally have) just report zk errors about network is unreachable01:51
jimbaker(agent logs for mysql/0, wordpress/0)01:51
jimbakerthe other detail is that the master-customize.log was not touched. is that only used in the event of an error?01:52
jcastrohazmat: oh nice, I'll try local/LXC as soon as the build is finished02:10
jcastrohazmat: I've been looking forward to this.02:10
jcastroanyone try the local provider? I'm having some problems getting the environment right02:49
jimbakerjcastro, i have tried the local provider several times, but yet to get it running02:52
jcastroI'm not even getting past the environments.yaml bits02:52
jimbakerjcastro, this is my environments.yaml for lxc: http://pastebin.ubuntu.com/702557/02:54
jimbakerjcastro, for me it's failing in the network at this point02:55
jcastro"error: internal error Network is already in use by interface virbr0"02:56
jcastrolook familiar?02:56
jimbakerjcastro, i have seen that. one thing to try (hate to tell you it) is to reboot your computer02:56
jcastrohah, awesome.02:56
jimbakerthe initial lxc networking for whatever reason didn't come up properly until i did this02:57
jimbakerjcastro, before doing that, might as well make sure you upgrade  the pkg cache02:57
jimbakersee above, approx 2.5 hours ago02:57
jcastrojimbaker: woo! status works03:00
jimbakerjcastro, cool, but the provisioning agent runs on your machine, so it's not indicative of the lxc bits03:01
jimbaker(other than using local of course)03:01
jimbakeri just got past that just now, the fragility was in the package cache03:02
jcastrothey show up in status, but their state is still null03:02
jimbakerjcastro, do something like $ ps -ef | grep juju - verify you see unit agents for mysql/0 and wordpress/0 (or whatever stack you are attempting)03:04
jcastrodoesn't look like it, just one for zookeeper itself, which I assume is the provisioning one03:05
jimbakerjcastro, yes, you will have one zookeeper instance as part of the provisioning setup03:07
jimbakerjcastro, so check for the existence of master-customize.log - this will be in the data dir, owned by root03:08
jimbakerjcastro, ideally you will have logs for each of service units there too, a container log (for lxc debugging) and the unit.log itself03:09
jcastroall the log dirs are empty03:09
jimbakerjcastro, ok, so you are seeing something else wrong (and i assume you did the reboot)03:10
jcastrothe bootstrap runs, completes, no errors03:10
jcastrojimbaker: ok heading to bed, openstack conf tomorrow, I will certainly try on the plane. :)03:12
jimbakerbootstrap really means nothing unfortunately - and status is less useful here03:12
jimbakerwell, of node 003:13
jcastroyeah it looks like it thinks it's firing off things, but they're not happening03:13
jcastroat first I was like "wow, this is fast."03:13
jcastrobut it doesn't appear to be actually doing anything, heh03:13
jimbakeri think it might actually be fast, but yeah, still waiting to get it to work03:13
jimbakerok, enjoy the conf!03:13
jcastroi feel like we're close!03:13
jimbakeri think so too, there are some basic setup doc issues, possibly ones that can be scripted, to make it work, i suspect i just have one more to go through03:14
hazmat /me catches up03:20
hazmatsurprised  rebooting is needed03:20
hazmatjcastro, its all async same as other providers so its fast to execute commands03:21
hazmatbut takes a few moments for reality03:21
hazmatjcastro, planes are probably a problem, network connectivity is needed for non cached package installs03:22
hazmatjimbaker, so the lxc cache update didn't solve the issue for you?03:24
_mup_juju/remove-sec-grp-do-not-ignore-exception r383 committed by jim.baker@canonical.com03:25
_mup_Merged trunk03:25
hazmatjimbaker, the master customize log is updated only after the complete run, u should use ps | grep lxc to verify that03:28
hazmatps uax03:28
hazmatoh... no.. gluster acquired03:28
_mup_juju/trunk r386 committed by jim.baker@canonical.com03:38
_mup_merge remove-sec-grp-do-not-ignore-exception [r=niemeyer][f=863510]03:38
_mup_Simplified logic for removing security groups to ensure any exceptions03:38
_mup_are not ignored by being consumed by the Twisted reactor.03:38
jimbakerhazmat, i got further with the lxc cache update04:10
jimbakerbut now i'm seeing this networking issue04:10
jimbakerhazmat, so in the unit.log files (in my case mysql/0, wordpress/0), i'm seeing this repeated: Network is unreachable04:14
jimbakerso something is wrong w/ the virtual network setup04:14
xerxasHi all !09:46
xerxasis juju well suited for my use case:09:46
xerxasI want to launch a complete infrastructure with 1 command, and service knowing one each other address09:46
xerxasand , also , I want to specify a list of packages to install on each nodes , and some sed commands09:47
xerxasis chef-solo better for doing this ? or cloudformation (I'm working on EC2)09:47
TeTeTxerxas: from my reading I think juju can do that for you - but keep in mind it is under development. Not sure if you want to base any production sites on it right now09:49
xerxasTeTeT:  you mean, if I want to make live my infrastructure ?09:52
xerxasIf I'm adding unit afterwards ?09:53
TeTeTxerxas: adding units should work nicely, if the charm allows for it. What is your schedule for going productive?09:53
xerxasnow ;)09:54
xerxaswhat's the simplest way de provision ec2 infrastructure (not instances, but infrastructures !)09:55
xerxascloud-init seems ok for instances, is cloud-formation the cloud-init of infrastructure ?09:55
kim0xerxas: actually I'd say juju is exactly what you need, and is the simplest way going forward09:58
kim0xerxas: it's just that it's alpha quality right now09:59
xerxaskim0: why ?09:59
kim0because it does what you described :)09:59
xerxaskim0:  I don't mind if it's alpha or not, if it boots my 3 instances, setup a rabbitmq on one , et create a config file on a second one containing my rabbitmq server ip address09:59
xerxaskim0:  ;)09:59
xerxaskim0: this is maybe why I'm asking here ;)10:00
xerxasahh , also, question , I tested juju ... but10:00
xerxasjuju client, needs to be ubuntu , as juju "controller" and , juju booted intances ?10:00
kim0xerxas: no client can be osx too10:00
xerxasby controller I mean the instance where zookeeper is installed10:00
kim0or other linux10:01
xerxaskim0:  nice, I'm running osx ;)10:01
xerxasbut the rest needs to be ubuntu10:01
kim0xerxas: I think it's available on some osx repo10:01
kim0can't remmeber what osx folks use to install cli tools :)10:01
kim0yes that's it10:01
xerxasand easy_install ;)10:01
TeTeTwasn't there fink back in the day?10:01
TeTeTwhich was a lot like apt10:01
kim0think brew it is now10:02
xerxasTeTeT:  fink is useless know10:02
TeTeTbeen 6 years since I worked with osx ...10:02
xerxasbrew is git based, easy formula (which is metadata for building packages) writing , social ...10:02
xerxasbeen 6 years I had a ubuntu based desktop ;)10:03
kim0xerxas: here is a rabbitmq charm: https://code.launchpad.net/~charmers/charm/oneiric/rabbitmq-server/trunk10:03
xerxasbut never left ubuntu on the server ;)10:03
kim0xerxas: good choice ;)10:03
xerxasand will never (you're doing good work , guys ! )10:03
xerxaskim0: thx for the url10:03
kim0xerxas: so one instance is rabbitmq, what are the other two ?10:03
xerxassome daemon I wrote in python consuming messages10:04
kim0ah cool!10:04
xerxasone will have some lxc container10:04
jamespagemorning - anyone tried using the LXC local provider yet?10:04
kim0xerxas: sounds like a cool project!10:04
xerxasanyway, my python daemon have a configuration file and in that configuration file I have an ip address, the one of the rabbitmq server10:05
xerxas(more precisely, one of rabbitmq-s server, because we might have several of them ... )10:05
kim0xerxas: I'd love to write an article about that project when you get it running .. keep me in the loop :)10:05
TeTeTjamespage: it's on my plate for later this week10:09
kim0TeTeT: is it already working fine ?10:19
TeTeTkim0: I don't know yet, this I why I'd love to test it10:33
* HarryPanda can't get local charm repositories working anymore11:14
hazmatgood morning11:15
HarryPanda`charm create test /root/charms && juju deploy --repository /root/charms/ local:test` = no dice11:16
hazmatHarryPanda, you need to use a series (like 'oneiric') as additional directory in the repository from the base to the charm..11:17
hazmatHarryPanda, ie.. so looking at the example formulas that come with juju.. its examples/oneiric/wordpress11:18
HarryPandaaah, I haven't been keeping up with the changelog11:18
hazmatHarryPanda, yeah.. that was recent11:20
* HarryPanda got confused about store.juju.ubuntu.com etc.11:20
hazmatthere was a mail out to the list about the chance11:20
hazmatHarryPanda, its basically a charm repository.. its not active yet11:20
hazmatHarryPanda, yeah.. its a bit confusing on the error message11:20
hazmatfwereade, ^11:20
hazmatjamespage, haven't gotten any new feedback yet11:21
jamespagehazmat, hey - I had a go based on your posting to juju@l.u.c11:22
hazmatjamespage, cool, how'd it go?11:22
jamespagehit a few errors - the environments.yaml entry wanted a type and admin-secret entry as well ( so I had a guess)11:23
hazmatjamespage, doh.. yeah11:23
hazmatjamespage, late night emails.. bad11:24
hazmatjamespage, and then?11:24
hazmatjamespage, fwiw admin-secret is anything you want it to be11:24
jamespageI appear to have a running environment!11:25
jamespageI've not deployed anything yet - you caught me as I was typing juju bootstrap --environment....11:25
* HarryPanda is slowly getting our full stack running on a mix of vmware+orchestra, ec2 and lxc :)11:25
hazmatjamespage, cool.. the real work happens when you first deploy a charm.. and it starts creating lxc containers..11:26
hazmatHarryPanda, fun11:26
jamespagehazmat, OK about to try that11:27
hazmatjamespage, its async to the cli, so giving progress on it is a little hard, you can ps aux | grep lxc   to watch it happen, or tail the log file in data-dir/machine-agent.log11:27
jamespagehazmat: hmm - not sure the machine agent started11:36
hazmatjamespage, hmm.. haven't seen that one11:39
hazmatjamespage, can you try bootstrapping with the verbose flag? ie. juju -v bootstrap11:39
hazmatjamespage, that should give the traceback11:40
HarryPandajamespage: I'm getting the same11:40
HarryPandalocal/agent.py:59, changed JUJU_ORIGIN=%s to JUJU_ORIGIN=ppa11:40
jamespagehazmat: http://paste.ubuntu.com/702706/11:41
hazmatjamespage, thanks11:41
jamespagenp - want me to raise a bug?11:41
hazmatjamespage, i think its a trivial, i can commit it to trunk, just double checking11:42
hazmati was verifying this with a juju-origin defined in my environments.yaml.. but i didn't test without one11:42
hazmatyeah.. niemeyer mentioned he'd changed the interface on this function, but i forget11:43
jamespagehazmat: hmm - might also be better to depend on zookeeper rather than zookeeperd as the local provider starts its own instance11:46
jamespagesaves a running java process which is always a good thing11:46
hazmatjamespage, good point.. libzookeeper-java should have what we need11:47
jamespagekeen to test this as I have seen panics running Java stuff under lxc with openstack11:49
jamespagemight disappear as a result tho :-)11:49
hazmatjamespage, we don't require any java under lxc.. but noted11:54
hazmatfwereade, http://paste.ubuntu.com/702712/11:54
hazmatcould i get a +1 on that trivial11:54
jamespagehazmat, well juju does not - but charms might...11:54
hazmatjamespage, yup11:55
hazmatno cassandra might hurt11:55
* hazmat sheds a tear over gluster's acquisition11:55
HarryPandaredhat? 0.o11:56
fwereadehazmat: sorry, I'm not quite following the paste11:59
fwereadehazmat: well, just the first bit12:01
fwereadehazmat: I guess, the packages need to change, so... you change the packages12:01
fwereadehazmat: consider my confusion withdrawn12:01
fwereadehazmat: +112:02
hazmatfwereade, yeah.. let me repaste.. the origin handling changed, but i made it to a branch copy by accident12:03
jamespagehazmat: getting there - I now have a running lxc container - however its not showing up as started yet12:06
jamespageI can log into it using SSH12:08
hazmatfwereade, updated diff.. http://paste.ubuntu.com/702715/12:08
hazmatfwereade, does a couple of things, logs juju origin when starting machine agent, fixes the handling for juju origin to reflect new return signature from get_default_origin, changes the package dep from zookeeperd to libzookeeper-java12:09
fwereadehazmat: cool; have a much-happier +1 :)12:11
HarryPandajamespage: what's the 'state' for the service unit?12:12
jamespageHarryPanda: null12:13
jamespageI can see the failsafe upstart configuration waiting for network interfaces to come up within the lxc instance12:13
_mup_juju/trunk r388 committed by kapil.thangavelu@canonical.com12:14
_mup_[trivial] require libzookeeper-java not zookeeperd, update handling of get_default_origin, log origin used [r=fwereade]12:14
hazmatjamespage, interesting i never noticed that b4 re waiting for network within the container12:17
hazmatjamespage, sadly it happens on my laptop.. sort of destroys any value to making boot faster12:18
hazmatjamespage, the unit agent gets started via upstart.. it should have a symlink to its log at $data-dir/units/$unit-name/unit.log12:19
hazmatthe link will be broken if the unit agent never started12:22
hazmatHarryPanda, where you get to with the origin=ppa change?12:24
HarryPandahazmat: enough to get it working so I could deploy stuff12:25
HarryPandaI will re-checkout after lunch12:25
hazmatHarryPanda, cool, and you see the units with state: started in juju status?12:25
hazmatHarryPanda, cheers12:25
hazmatlxc-ls shows the containers, the output is a bit odd, it will list it twice if its running, once if its defined but not running12:26
* hazmat regen's the ppa12:27
* HarryPanda still has a lot of catching up to do in the meantime, like puppet integration12:29
hazmatjamespage, can you pastebin your $data-dir/units/master-customize.log12:31
jamespagehazmat: http://paste.ubuntu.com/702724/12:36
hazmatjamespage, nice.. looks good12:37
hazmatjamespage, but your unit agents aren't starting?12:37
jamespagenot within the instance12:38
hazmatjamespage, do you have any output in $data-dir/units/$unit-name/unit.log or is it a broken link?12:39
jamespagehazmat, broken link ATM12:40
jamespagethis instance is stalled waiting for the network interface to start12:40
jamespageso I don't the right run-level has been initiated yet12:40
hazmatjamespage, if your able to ssh into the unit, you can try manually starting the agent using its upstart file as a guideline /etc/init/$unit-name-unit-agent.conf12:41
hazmatjamespage, hmm.. i haven't seen that one before re network start wait in the container, except on the host12:41
* hazmat doesn't understand why oneiric is waiting on network config12:42
jamespagehazmat: so I did a sudo start xxx on the unit agent12:42
jamespagenow showing as started12:42
hazmatjamespage, cool12:43
jamespagehazmat; adding another unit has resulted in another stalled lxc instance - bah!12:43
hazmatjamespage, had you used lxc before today?.. i'm just wondering if destroying the env, and clearing out the lxc cache would help /var/cache/lxc/oneiric/*12:46
hazmatits been at least week since i did that myself12:46
jamespagehazmat, hmm - I can try that12:46
hazmatjamespage, alternatively you can try manually chroot into the cache and upgrade it.. but if i remember the bug correctly its pretty temperamental wrt to package install/upgrade/removal12:47
* hazmat will bbiab12:58
niemeyerMorning all!13:01
rogniemeyer: hiya13:05
jamespagehazmat, looks like /etc/rcS.d/S07resolvconf is blocking - which is stopping the rest of the lxc instance from coming up OK13:12
niemeyerrog: Yo13:15
niemeyerjamespage: Hey James13:15
niemeyerjamespage: Very interesting ideas in your testing charms post13:15
jamespageniemeyer, morning13:15
niemeyerjamespage: Still digesting them13:15
jamespageniemeyer, I did a bit of refactoring yesterday based of sabdfl's comments13:16
niemeyerjamespage: We have a test suite continuously running at wtf.labix.org ATM, and it made me ponder if we could do more there13:16
jamespagethe charm-tester is now a bit more clever and less implicit13:16
jamespageniemeyer, that might be good13:16
jamespageniemeyer, whats driving the testing ATM?13:22
niemeyerjamespage: A few trivial scripts.. the whole thing is in lp:juju/ftests if you'd like to check it out13:22
niemeyerfwereade: ping13:26
fwereadeniemeyer: pong13:26
fwereadeniemeyer: it's turned out *much* easier than I expected13:26
niemeyerfwereade: Oh, that's so great to hear13:26
niemeyerfwereade: I went to sleep a bit concerned yesterday imagining if we'd have to go back13:27
fwereadeniemeyer: didn't stop me trying to do it wrong earlier, ofc, but you should have a fresh MP in a short while13:27
niemeyerfwereade: Woohay!13:27
fwereadeniemeyer: yeah, it's a great relief :)13:27
fwereadeniemeyer: and then the auto-update on top of that should be almost trivial13:27
fwereadeniemeyer: EOD or before13:28
* fwereade crosses fingers13:28
niemeyerfwereade: Superb.. I'm hopeful we can close down the client side for fixes only today still13:28
niemeyerSpamapS: ^13:28
jamespagehazmat: so it looks like plymount --ping is hanging forever; hence why the failsafe config is kicking in13:32
* jamespage scratches his head13:33
SpamapSniemeyer: if you guys can give me the revision to upload I will start some builds and tests13:52
niemeyerSpamapS: Sounds great.. we just need the stuff fwereade is punching on right now13:53
SpamapSniemeyer: here is r381's build failures btw  https://launchpadlibrarian.net/81962870/buildlog_ubuntu-oneiric-i386.juju_0.5%2Bbzr381-1juju1~oneiric1_FAILEDTOBUILD.txt.gz13:54
hazmathaven't seen those13:56
hazmatjamespage, lame.. dhcp should just work13:57
jamespagehazmat, I think it kinda is - however plymouth --ping is hanging  when it tries to configure its logging13:57
hazmatjamespage, we can prepopulate resolvconf/resolv.conf.d/base like we did before but for the container startup, straight dhcp should just work13:57
* hazmat does a man plymouth 13:58
hazmathmm.. undocumented13:58
hazmatjamespage, so plymouth --ping sounds like it just pings upstart13:59
hazmatfrom its help13:59
jamespageits the thing that multiplexes all of the output from init scripts and upstart13:59
* hazmat is out of his element13:59
jamespageI've pinged hallyn_ - he might have some ideas esp. with regards to why its borked in a lxc container14:00
hazmatjamespage, great, just sent some updated instructions to the list and noted the plymouth hangs14:01
* hazmat resets his lxc cache14:02
* hazmat really needs to find a 7mm ssd14:03
fwereadeniemeyer: about "breaking" changes14:20
fwereadeniemeyer: it's ok to just commit something that will  not work with existing agents deployed from an older branch... or is it?14:21
niemeyerfwereade: *TODAY* it is.. tomorrow, it's not. :-)14:21
fwereadeheh, good :)14:21
mdeslaurhey guys, you should get the juju mailing list added here: https://lists.ubuntu.com/14:22
mdeslaurit's kind of hard to find if you don't know what it's called14:22
hazmatmdeslaur, good point14:26
_mup_Bug #868391 was filed: Juju mailing list should be listed on lists.ubuntu.com <juju:New> < https://launchpad.net/bugs/868391 >14:27
hazmatmdeslaur, at the moment the easiest way to find the ml is from the section at the bottom of juju.ubuntu.com14:27
mdeslaurhazmat: yes, thanks14:28
niemeyerrog: Great stuff indeed in update-server-interface man, thanks!14:33
niemeyerrog: I've sent a few comments/suggestions, but pretty superficial stuff14:33
rogniemeyer: good, i'm, erm, relieved you like it :-)14:33
rogdon't you hate it when you know you've made some changes somewhere, but you can't find them anywhere? makes me wonder what else i'm also missing :-)14:37
fwereadeniemeyer: how would you like really-isolate-formula-revisions? stacked branch on isolate-formula-revisions, both set to needs review?14:40
hazmatjamespage, works for me ootb with a clean lxc cache14:40
hazmatjcastro, ping14:40
rogniemeyer: ah, i've just realised that update-server-interface is several revisions ago.14:41
niemeyerrog: Well, please do the suggested changes in the version that is up for review, and let's merge it as is14:41
niemeyerrog: Then you can push a separate branch with the new content14:41
rogniemeyer: i thought you were reviewing factor-out-service ...14:41
niemeyerrog: Nope.. reviewing this now14:41
niemeyerrog: Please do the changes in the specific revision that was actually reviewed so we can merge it14:42
rogniemeyer: will do.14:42
niemeyerfwereade: Hmmm14:42
niemeyerfwereade: Not sure I understand your question14:42
niemeyerfwereade: You have a review on the previous branch.. what is happening with it?14:43
fwereadeniemeyer: the trivials are fixed in that branch; the big one is fixed in the stacked branch14:43
fwereadeniemeyer: the actual changes are trivial but widespread14:43
fwereadeniemeyer: I'm just as happy to merge back into the original branch, or to add a new MP14:44
fwereadeniemeyer: whatever's easiest for you to review14:44
niemeyerfwereade: Please just push it all on that single branch.. I'll have to review it jointly either way since the prior one has blockers without the follow up14:44
fwereadeniemeyer: ok, cool14:44
niemeyerrog: Just added a [10] comment now that I've figured a detail in the follow up branch14:48
rogniemeyer: one thing that concerns me about doing kill(pid, 0) is that it's system-specific. but i guess we're tied to linux anyway.14:49
niemeyerrog: Yeah.. I'm not very concerned about WIndows to be honest :)14:50
niemeyerrog: The way to fix that would be to add something in the Process interface itself14:50
niemeyerrog: IsRunning or whatever14:50
SpamapSrog: server side linux is a good bet. clients may be OS X or something else.14:50
niemeyerrog: But that's not something I worry about myself.. I'll let the Windows folks worry about it14:51
rogthis is client-side too, right? i'm also slightly concerned that some users may get EPERM even though the process is actually running.14:51
rogi'm not too au fait with modern unixy capability stuff14:52
niemeyerrog: If they get EPERM, that's great.. it's not their process and they shouldn't be fiddling with it14:59
* rog nods.15:00
jamespagehazmat: I have something to try - I have plymouth in debug mode on my laptop for another issue - might be causing problems15:04
rogniemeyer: hmm, there's a problem with checking if server process is actually running: what do you do if the pid.txt file exists, but there's no running server? you'll have to remove or rewrite the pid.txt file, but then you no longer get the nice race-free create semantics that you've got currently, so two concurrent callers could start two servers at once.15:15
fwereadeniemeyer: btw, https://code.launchpad.net/~fwereade/juju/isolate-formula-revisions/+merge/78164 is ready for another look15:17
jamespagehazmat: that fixed it - for some reason having plymouth:debug as a kernel boot options breaks the in-instance plymouth15:21
niemeyerfwereade: Ok, let me see how's lunch looking like here15:33
niemeyerfwereade: I'll review it either right now or right after lunch15:33
niemeyerrog: I don't get the problem.. if the pid file exists and there's no server running, there's simply no server running?15:34
rogniemeyer: yes, but there's a problem if two processes are both trying to start a server at the same time in the same directory15:34
rogniemeyer: currently that situation is dealt with fine15:34
niemeyerrog: One of them will fail because the port number won't be available15:35
rogniemeyer: but the wrong pid might end up in the pid file15:35
hazmatjamespage, awesome!15:36
* niemeyer thinks15:36
rogniemeyer: my current thought is that we should still return an error, but that the error should reflect our knowledge of whether the server is or isn't currently running15:36
hazmatkill(pid,0) should work on osx i would think15:37
roghazmat: yeah, it should.15:37
roghazmat: i was more thinking of windows.15:38
hazmatah, i don't think anyone cares about that in practice for juju ;-)15:38
niemeyerrog: Please ignore the concurrency issues for now and just proceed as we discussed15:38
roge.g. "server is currently running" vs "server was terminated abnormally. Remove /.../pid.txt to force a start"15:38
niemeyerrog: We can use actual file locking to sort the problem out, but that's really not for this branch15:39
rogniemeyer: oh yeah, another thought: does anyone use shared filesystems these days?15:39
rogniemeyer: 'cos that'll break it too.15:39
niemeyerrog: I don't know, but that's not a problem to worry about now either15:40
hazmatniemeyer, latest tip should work with the go branches in review?15:41
niemeyerhazmat: It should15:44
* hazmat compiles tip and dives in for a review15:44
niemeyerfwereade: Ok, I'll have lunch before finishing your review.. will be back ASAP and dive into it directly15:52
fwereadeniemeyer: cool, the auto-upgrade itself is simple, and works AFAICT, but testing it is a bit tedious15:52
hazmatfwereade, should be much faster with local provider ;-)15:53
fwereadehazmat: it's just *writing* the tests that's bugging me :)15:53
hazmatfwereade, ah15:54
jimbakerhazmat, i tried the latest (trunk r388), still seeing the same networking issues15:55
jimbakeri'm going to just reboot this box and see if that helps15:55
hazmatjimbaker, yeah.. bcsaller is as well15:55
hazmatjimbaker, rebooting won't do anything for this15:55
bcsallerhazmat: last I heard that was about virbr0 errors for jimbaker, thats not what I'm getting15:56
jimbakerhazmat, ok, it was the ultimate solution for the first time, but definitely welcome any ideas15:56
hazmati'm clearing out my apt-cacher-ng cache, and trying again, already cleared out the lxc cache and it worked fine.. so afaics i shouldn't have anything cached localy15:56
hazmatjimbaker, your not getting virbr0 errors anymore are you?15:56
jimbakerhazmat, no15:56
hazmatbcsaller, so i cleared out lxc cache, apt-cacher-ng cache, and it works for me..15:57
jimbakeri'm just getting network connectivity issues between the lxc instances (mysql/0, wordpress/0) and zk15:57
SpamapShazmat: we may want to add a juju-local-dev metapackage with all those extra deps15:57
hazmatSpamapS, perhaps.. bootstrap will fail and tell you whats missing though15:57
bcsallerhazmat: yeah, thats expected, right. Its stale cached data that makes the issue15:57
SpamapShazmat: think upgrades15:58
SpamapShazmat: if we add more of those, or don't need more of those, we can help users add/remove them automatically :)15:59
hazmatSpamapS, yeah.. but that puts the extra dep everywhere15:59
hazmatSpamapS, including machines deploying units15:59
hazmatSpamapS, ie require java ;-)15:59
hazmati'd rather avoid that16:00
SpamapShazmat: which is why we put it as juju-local-dev, which is only a Suggests of juju16:00
hazmatSpamapS, ah. true that, yeah.. that sounds fine to me.. i think niemeyer objected last time around on that, but if your up for doing it.. sounds good16:01
SpamapShazmat: its a distro choice really. :)16:01
hazmatSpamapS, well then i leave it to some anonymous distro person ;-)16:02
jamespageslightly frustratingly my local lxc instances are not picking up any dns resolvers...16:12
bcsallerhazmat: ^^16:12
hazmatbcsaller, why would dnsmasq need to be specified explicitly in the container if its dhcp based16:17
bcsallerhazmat: maybe if the resolvconf package isn't installed, I don't know the details, it might be a matter of competing systems there16:18
hazmatbcsaller, so what do you have in resolv.conf of the master template?16:18
koolhead17SpamapS: hey16:21
jimbakerhazmat, finally the lxc containers came up after cleaning the caches (i assume /var/cache/lxc, /var/cache/apt-cacher-ng), but i'm still getting Network is unreachable in the unit logs: http://pastebin.ubuntu.com/702840/16:24
niemeyerfwereade: Diving in again16:25
hazmatjimbaker, do you have zookeeper running in the host?16:27
hazmatsounds like it on a different port then what the unit agents expect16:28
jimbakerhazmat, it was started by the local provider16:29
jimbakerwith this setting: clientPort=5369916:30
hazmatjimbaker, what ip address does the lxc instance have?16:34
hazmatlxc-ls .. grab container name.. and ping dnsmasq... host container-name
hazmatjamespage, what do you end up with in resolv.conf of the container? is it empty?16:40
jamespagehazmat, it had the resolvconf header but no configuration16:40
hazmatbcsaller, yeah.. perhaps we need both16:41
hazmati find it extremely odd that we don't pick up the dns server from dhcp16:41
jimbakerhazmat, not clear what you mean here...16:41
hazmatjimbaker, the container has an ip address, the easiest way to discover it (if the unit agents aren't connecting) is to query the dnsmasq with the container name16:42
jimbakeri have consulted the man page/other howtos on dnsmasq. in any event, i can see the containers with lxc-ls16:43
jimbakerhazmat, ok, i was getting confused about things ;) just use standard tools16:44
jimbakerhazmat, do we actually setup dnsmasq with the local provider?16:45
hazmatjimbaker, no its from libvirt16:45
hazmati need to grab some lunch16:45
hazmatbcsaller, i guess the way to verify re dns is to add it to both run/resolv.conf and resolv.conf.d/base16:46
jimbakerhazmat, i do have dnsmasq running. in any event, we can debug more when you get back16:46
niemeyerhazmat: Please ping me when you're bakc16:49
niemeyerSpamapS: ping16:49
niemeyerfwereade: ping16:59
fwereadeniemeyer: pong16:59
niemeyerfwereade: Looks fantastic17:00
niemeyerfwereade: I'm pretty surprised as well17:00
fwereadeniemeyer: yeah, totally unexpected17:00
niemeyerfwereade: We should try to get a +1 from hazmat as well given this is such a core concept17:01
niemeyerfwereade: I have just a few trivials for you meanwhile17:02
fwereadeniemeyer: sounds good; sounds good17:02
niemeyerfwereade: Sent17:03
fwereadeniemeyer: cheers17:03
niemeyerfwereade: Is there anything else in the pipe I can help you with while you take a look at these?17:03
fwereadeniemeyer: try previewing lp:~fwereade/juju/always-upgrade-local17:04
fwereadeniemeyer: just pushed it17:04
niemeyerfwereade: I'm on it17:04
fwereadeniemeyer: pretty sure it works, was just setting up a local provider to try to verify a bit faster17:04
niemeyerfwereade: Awesome17:04
fwereadeniemeyer: re: caching in a private attribute, the idea was that CharmDirectory instances should still report the correct revision even if they're up-revisioned mid-flow17:09
fwereadeniemeyer: on balance, that might not actually be necessary17:09
fwereadeniemeyer: because that *should* only happen via set_revision (or at __init__ time)17:09
niemeyerfwereade: Yeah, I think that's the exception, so I'd rather delay the behavior until necessary17:10
niemeyerfwereade: Opening the file all the time feels bad given we don't depend on this behavior today17:10
fwereadeniemeyer: in which case... yep, I'll just call it once, will simplify everything I think17:11
fwereadeniemeyer: cheers17:11
fwereadeniemeyer: others look good, ta17:11
niemeyerfwereade: Thanks!17:11
niemeyerfwereade: http://paste.ubuntu.com/702861/17:18
fwereadeniemeyer: [1] yep, clearer17:19
fwereadeniemeyer: [2] ambivalent; I'm not sure local bundles are common enough to be a major worry17:20
fwereadeniemeyer: it feels like the same class of issue as "what if the local repo is not writable"17:21
niemeyerfwereade: They're not.. but if we don't take care of this case, it'll explode and render upgade-charm non-useful with bundles17:21
fwereadeniemeyer: hm, true17:21
niemeyerfwereade: Different case, I think17:21
fwereadeniemeyer: fair enough17:22
fwereadeniemeyer: offhand, is there a non-isinstance way to tell the difference?17:22
niemeyerfwereade: Nope, I think isinstance is the way to go17:27
kim0hey folks .. is this plymouth thing blocking local deployment, do can I play already17:29
kim0lxc deployment I mean17:29
hazmat /me is back17:44
hazmatniemeyer, ping17:44
hazmatfwereade, so the charms have revision directly on them now.. the callback into the metadata stuff was wierd..17:44
* hazmat checks out the branch latest17:45
hazmatkim0, try it out17:48
kim0cool will17:48
hazmatkim0, it was a issue to jamespage setup he had plymouth in debug mode17:49
fwereadehazmat: yeah, the idea was to avoid changing the charm state format17:53
fwereadehazmat: turns out, revision was never actually used from charm state17:53
fwereadehazmat: (well, it was, but it came from charm_url, and we only touched metadata.revision to assert they were the same)17:53
hazmatfwereade, cool17:54
hazmatfwereade, so the branch looks good to me, the only thing is the file contents should be stripped before attempting to int()17:54
hazmatfwereade, if people hand edit it, editors are want to do wonky things17:55
fwereadehazmat: >>> int("  \t27\n  \r\n  ")17:55
fwereadehazmat: but still, no actual dsagreement17:56
fwereadehazmat: explicit has occasionally been postulated to be superior to implicit, after all ;)17:56
hazmati definitely remember that not working at some point17:56
hazmatoh well, relic of the past17:57
fwereadeso, hazmat, can I count that as an approve?18:04
hazmatfwereade, yeah.. i'm just wondering about the extra zipfile construction in get revision on the bundle, i guess its fine since its cached, but we have the zip file already in init18:04
hazmatfwereade, +118:05
fwereadehazmat: and we always call get_revision in __init__, too18:05
fwereadehazmat: I'll just fix that and merge then18:05
fwereadehazmat: cheers18:05
hazmatfwereade, cool18:06
rogsee y'all tomorrow18:15
hazmatrog have a good one18:17
niemeyerrog: Cheers!18:19
niemeyerfwereade: I'm writing an email to the list warning about this change18:21
fwereadeniemeyer: tyvm18:25
jimbakerhazmat, so i just wanted to continue with you on the local provider debugging18:30
jimbakerhazmat, whenever it's a good time, just ping me18:30
niemeyerfwereade: Hey!18:31
niemeyerfwereade: One thought just occurred to me18:31
fwereadeniemeyer: go on18:31
niemeyerfwereade: While writing the email18:31
niemeyerfwereade: Rather than erroring out with a missing revision file, we should create one with revision 1 automatically18:31
niemeyerfwereade: In a charm directory18:32
fwereadeniemeyer: good plan18:32
fwereadeniemeyer: 0 maybe?18:32
niemeyerfwereade: Sounds good.. I'm a fan of zero indexing as well ;-)18:32
fwereadeniemeyer: or is that just too look-at-me-I'm-a-*programmer*?18:32
niemeyerfwereade: LOL18:32
niemeyerfwereade: Maybe.. but memory initialized to zero by default is such a great idea!  I'm sure we can't go wrong in this case either. ;-D18:33
fwereadeniemeyer: anyway, sounds good, I'll pastbin you a trivial shortly, just fixing up the always-upgrade-local branch18:33
hazmati thought it was backwards compatible already?18:40
hazmatonly on invalid content of the metadata file would it error18:41
hazmatjimbaker, ping18:44
fwereadeniemeyer: http://paste.ubuntu.com/702915/18:46
niemeyerhazmat: It's backwards compatible, but it's a significant change in structure that deserves a ntoe18:46
fwereadehazmat: it is backwards compatible; this is a distinct case, where you create a charm and never even consider the possibility of needing a revision18:47
hazmatsounds good18:47
niemeyerhazmat: Otherwise people will get a crazy warning that makes no sense after an unknown point in time18:47
jimbakerhazmat, hi18:47
niemeyerfwereade: Hmmm.. that's not quite it I think18:48
niemeyerfwereade: There's an important difference between a revision file not being present and it containing unexpected content18:48
fwereadeniemeyer: ...very good point18:48
* fwereade looks shifty18:49
hazmatits more of a preamble to the method if not os.path.exists(revision_path)18:49
niemeyerfwereade: Don't worry.. you can look shifty for the next few months without any damage to your image given how late it is there and how much you've been pushing :)18:49
hazmatjimbaker, so to continue the epic, our heroes where last trying to figure out why they couldn't get to the zoo18:49
hazmatfwereade, and it also has to play nice with the backwards compatible  stuff, ie. what might already be in the metadata18:50
* niemeyer puts that in the quotes page18:50
hazmatfwereade, bringing shifty back into style ;-)18:51
hazmatjimbaker, where you able to get the container address?18:51
jimbakerhazmat, not certain what the procedure for that should be18:52
hazmatjimbaker, query the dnsmasq server18:52
hazmatjimbaker, is it running?18:52
hazmatbcsaller, where you able to try out adding the dnsmasq server to base in addition to run/resolv.conf?18:52
jimbakerhazmat, dnsmasq is running18:52
jimbakerhazmat, with this range reported from ps, --dhcp-range,
jimbakerso that looks as expected18:53
hazmatjimbaker, lxc-ls -> list of containers.. pick container name of unit... query dnsmasq on its listen address typically
jimbakerhazmat, hmm, i missing something in my knowledge of how something like this is setup; would the query be like this: dig -b jbaker-desktop-wordpress-018:55
jimbaker(yes, its listen address is, according to ps)18:55
hazmatdig @ jbaker-desktop-wordpress-018:56
hazmator less verbosely18:56
hazmathost jbaker-desktop-wordpress-0
jimbakerhazmat, now makes perfect sense, but dnsmasq doesn't have the address: http://pastebin.ubuntu.com/702926/18:59
fwereadehazmat, niemeyer: http://paste.ubuntu.com/702927/18:59
fwereadeslightly less trivial now but just barely qualifies IMO19:00
hazmatfwereade, that seems sensible.. basically only in the absence of revision set it19:01
hazmati was wondering about transparently doing migrations for folks, but that's probably icky, they'll see the warning19:02
hazmatfwereade, +119:02
niemeyerfwereade: Put this within the try:  if result >= 0:\n return result19:03
niemeyerfwereade: as a follow up you'll notice you can unify the branches19:03
hazmatjimbaker, odd indeed. what's the output of > brctrl show and virsh net-list --all19:03
hazmatjimbaker, its like you have a different bridge setup19:03
niemeyerfwereade: It all looks good though, +119:04
hazmatjimbaker, for any active on virsh net-list.. can you paste virsh net-dumpxml network-name19:04
jimbakerhazmat, i don't have brctrl installed. is that a problem?19:05
fwereadeniemeyer: re try: in get_revision?19:05
niemeyerfwereade: yeah19:05
jimbakerin terms of not having some useful bridging utils installed?19:05
fwereadeniemeyer: that won't error on revisions < 019:05
hazmatjimbaker, no its just a bridge management tool..19:06
hazmatalthough i thought libvirt used it19:06
jimbakerhazmat, just checking ;)19:06
niemeyerfwereade: It will if you tweak the branches below19:06
jimbakerhazmat, definitely could install bridge-utils shortly19:06
niemeyerfwereade: You'll need a single raise and won't have to define the spurious message upfront19:06
fwereadeniemeyer: ...oh, *branches*, not branches19:06
hazmatjimbaker, do you have libvirt-bin installed?19:06
niemeyerfwereade: LOL19:06
* fwereade looks shifty again19:06
niemeyerfwereade: Yeah, sorry :-)19:06
hazmatjimbaker, bridge-utils is a dep of it19:06
jimbakerhazmat, i do, but apparently an upgrade is available19:07
jimbakerhazmat, enjoying the oneiric edge for sure19:07
hazmatjimbaker, right.. but you should have brctrl.. its a dependency for libvirt-bin19:07
jimbakerhazmat, certainly strange19:07
jimbakerhazmat, the upgrade of libvirt-bin did not install bridge-utils/brctrl, so don't know why the dependency is not being honored (or different here)19:09
jimbakerhazmat, sorry, it was just suggesting bridge-utils be installed for missing brctrl, but i do have it19:10
jimbaker(the bridge-utils package)19:10
hazmatjimbaker, okay... so output of brctrl show would be helpful... also do you have other virtualization packages (vmware, virtualbox, etc) on the machine?19:11
jimbakerhazmat, no other virtualization installed on this box19:11
hazmatjimbaker, cool19:11
jimbakerhazmat, still looking for a package with brctrl19:11
hazmatjimbaker, so the brctrl output and the virsh net-dumpxml default output are next19:11
hazmatjimbaker, sorry its brctl19:12
jimbakerhazmat, virsh net-dumpxml default: http://pastebin.ubuntu.com/702935/19:12
jimbakerhazmat, brctl show: http://pastebin.ubuntu.com/702936/19:13
hazmatjimbaker, so all that looks good.. time to check the container config19:14
jimbakerhazmat, sounds good19:15
fwereadeniemeyer: and at last: https://code.launchpad.net/~fwereade/juju/always-upgrade-local/+merge/7830619:15
hazmatjimbaker, can you pastebin  sudo cat /var/lib/lxc/jbaker-desktop-wordpress-0/config19:15
fwereadeniemeyer: already addressed the points you brought up before19:15
niemeyerfwereade: Cool, looking!19:16
jimbakerhazmat, here is is: http://pastebin.ubuntu.com/702937/19:16
niemeyerfwereade: http://wtf.labix.org/19:19
niemeyerfwereade: Looks like it never came up.. not sure if it's a hiccup or if it's actually broken for unknown reasons19:20
niemeyerI'll kick it and run it again19:20
fwereadeniemeyer: blech, I'll verify here19:21
niemeyerfwereade: +1 on the change19:23
niemeyerfwereade: 390 is already on the pipeline running19:30
niemeyerfwereade: Let's see..19:30
fwereadeniemeyer, something is screwed up, the cloud-init scripts fall over19:33
fwereadeniemeyer, but I don't *think* this was me...19:33
fwereade2011-10-05 19:24:02,144 - cc_apt_update_upgrade.py[WARNING]: Failed to install packages: ['bzr', 'byobu', 'tmux', 'pyt19:34
fwereadehon-setuptools', 'python-twisted', 'python-argparse', 'python-txaws', 'python-zookeeper', 'default-jre-headless', 'zoo19:34
fwereadekeeper', 'zookeeperd']19:34
fwereadeniemeyer: I'm seeing a lot of 403s trying to talk to http://us-east-1.ec2.archive.ubuntu.com19:35
hazmat jimbaker that's just wacky19:36
jimbakerhazmat, how so? ;)19:36
hazmatjimbaker, everything looks good19:36
jimbakerhazmat, ahh. maybe i should try the reboot. just this once19:36
fwereadeniemeyer: does that sound familiar in any way?19:36
hazmatreboot is for windows and kernel upgrades (/me shake fist at oracle for buying ksplice)19:37
jimbakerhazmat, i know, i know19:37
niemeyerfwereade: I've heard something about a name change yeah19:37
niemeyerfwereade: What's strange is that the wtf broke in that specific revision19:37
fwereadeniemeyer: theories regarding what I did received gratefully... :/19:38
jimbakerit's an act of desperation. but is everyone else who's tried the local provider seen it work?19:38
hazmatjimbaker, not afaik, i think jamespage and bcsaller had it fail on a different network issue (dns wasn't working)19:39
niemeyerfwereade: The previous revision looks suspicious19:39
niemeyerfwereade: 38819:39
niemeyerfwereade: But that'd mean something in wtf got the revision wrong somehow19:39
niemeyerfwereade: Which isn't impossible19:39
bcsallerhazmat: I'm testing a small diff to juju-create that I think fixed everything here19:39
hazmatbcsaller, writing both base and run/resolv.conf ?19:40
fwereadeniemeyer: hm, that went in a good while ago, and all it hits is providers.local... I thought unused providers weren't even imported?19:40
bcsallerhazmat: yeah, http://pastebin.ubuntu.com/702946/ w/ the apt-get update after the cache is in place19:41
niemeyerfwereade: True..19:41
niemeyerfwereade: It could be something in Oneiric itself too19:42
niemeyerfwereade: I'll rollback wtf to 388, and make it test everything again from there19:42
fwereadeniemeyer: cheers19:42
hazmatbcsaller, that's slightly dangerous.. if the environment is long running, it will hit the same problem during a beta cycle.. but ok19:42
niemeyerfwereade: If 388 changes to broken, it might not be your revision19:42
* fwereade feels slightly hopeful19:42
niemeyerfwereade: Your merge log message are not following the convention, btw19:43
hazmatbcsaller, its bewildering though to me, its like dhcp isn't being respected, we shouldn't have to manuall setup dns for a dhcp setup19:43
niemeyerfwereade: Do a "bzr log" on trunk to see the difference19:44
bcsallerhazmat: I haven't verified it resolvconf is messing that up, /etc/dhcp/dhclient-enter-hooks.d/resolvconf makes me think it should be working as well19:44
SpamapSniemeyer: pong!19:45
fwereadeniemeyer: damn, I'm sorry, I thought I'd retrained myself right19:46
bcsallerhazmat: another issue, the way wp is coming up now its only addressable by IP, I'll look at adding an alias with the assigned name and see if that fixes it19:46
hazmatbcsaller, that's not really a problem its only resolvable by the browser by ip19:47
hazmatbcsaller, that's irrelevant imo19:47
niemeyerfwereade: No worries, it's just important because it's hard to tell what was merged from e.g. http://bazaar.launchpad.net/~juju/juju/trunk/revision/38919:47
bcsallernot a deal breaker, no19:47
niemeyerSpamapS: Hey man19:47
niemeyerSpamapS: Was going to ask you for some reviewing, but we already pushed it forward19:47
niemeyerfwereade: 388 is churning19:48
fwereadeniemeyer: the "mfrom: (348.8.15 isolate-formula-revisions)" on that page gives us as much information as does the summary line, surely?19:49
fwereadeniemeyer: but I'll try to force myself to keep to the standard anyway19:50
fwereadeniemeyer: that's sort of good news and sort of bad news19:50
_mup_juju/ftests r12 committed by gustavo@niemeyer.net19:50
_mup_- Fixed butler so it doesn't jump revisions.19:50
_mup_- Compute the waterfall after every revision.19:50
niemeyerfwereade: Indeed19:53
niemeyerfwereade: Hadn't noticed it to be honest19:53
fwereadehazmat: do you have a moment for a review of https://code.launchpad.net/~fwereade/juju/always-upgrade-local/+merge/78306?19:55
hazmatfwereade, sure19:55
SpamapSniemeyer: ahh ok. :)20:02
jimbakerhazmat, so one useful aspect of doing the reboot is looking at this scenario: what happens to the lxc containers? bootstrap reports it's still bootstrapped, but status fails20:03
hazmatjimbaker, their down20:05
hazmatthey don't autostart20:05
hazmatphone bbiam20:05
jimbakerhazmat, exactly. is there a way to restart them?20:05
jimbakerhazmat, later20:05
hazmatjimbaker, lxc-start -n name-of-container20:07
jimbakerhazmat, sounds good. so we can support that later in some sort of automatic way i guess20:07
hazmatjimbaker, i believe there are mechanisms to autostart them as well20:07
hazmatjimbaker, but the zk and machine agent are down as well20:08
jimbakerhazmat, in any event, the reboot did nothing (as expected)20:08
hazmatjimbaker, so on reboot you'll need to destroy and bootstrap20:08
hazmatthe env20:08
hazmatfor now20:08
jimbakerhazmat, correct, that's why juju status failed so badly20:08
jimbakerhazmat, which is fine for now20:08
hazmatjimbaker, absolutely, we're not touching the host this release with local provider20:09
hazmatoutside of the minimum needed to work (lxc containers)20:10
jimbakerhazmat, unless you have any more ideas, i'm going to put off getting local provider running. hopefully jamespage, bcsaller, and others have better luck20:10
hazmatjimbaker, well, we havent done anything since its been rebooted20:10
bcsallerjimbaker: with the small patch I posted a short while ago its working fine for me again20:10
jimbakerbcsaller, what was that?20:10
hazmatjimbaker, destroy-environment and bootstrap & deploy20:10
hazmatjimbaker, the issue your seeing is different than the patch which fixes some dns resolver issues20:11
jimbakerhazmat, yes, i did that, and units were started, but the network issue is still there20:11
niemeyerUh oh20:11
hazmatjimbaker, so you still have units unable to connect to zk?20:11
jimbakerhazmat, correct, exact same issue in the unit.log (s)20:12
hazmatjimbaker, i'm not entirely out of ideas yet ;-)20:12
niemeyerhazmat, SpamapS: AMBER CODE20:12
jimbakerhazmat, ok, then let's try them out20:12
hazmatniemeyer, ready and waiting20:12
niemeyerEC2 deployment is broken20:12
niemeyerand it doesn't look like something we did20:12
niemeyerSomething probably changed in Oneiric in the last few hours20:13
hazmatjimbaker, confirming (netstat -al) that zk is listening on a localhost with port known to the agents20:13
niemeyer388 was passing moments ago, and is now broken20:13
hazmatjimbaker, i'm going to swtich off to the ec2 issue20:13
hazmatniemeyer, debugging20:13
niemeyerfwereade: Looks unrelated to your changed indeed20:13
hazmatniemeyer, could be a bad image20:13
niemeyerfwereade: Do you have more details that could help us pinpoint the bug20:13
niemeyerfwereade: (from your test run)20:14
jimbakerhazmat, definitely is priority20:14
hazmatjimbaker, but if you could confirm that info it would be helpful20:14
hazmatjimbaker, did you mention early i could get a login to that machine?20:14
jimbakerhazmat, yes, i will create one for you20:15
fwereadeniemeyer: sorry, not really: the only other thing I verified was that even bzr wasn't installed20:15
niemeyerfwereade, hazmat: Maybe one of the packages is missing (or has been renamed)20:15
hazmatso if the ec2 repos are changing names20:15
hazmatniemeyer, i'm firing up an ec2 environment more info in a few20:15
niemeyerhazmat: From your log:   [trivial] require libzookeeper-java not zookeeperd20:16
niemeyerhazmat: Shouldn't that be done to the EC2 provider as well?20:17
hazmatniemeyer, no.. they don't install this stuff.. that's for running it on the host20:17
* niemeyer checks the list of packages20:18
niemeyerLooks ok20:22
niemeyersmoser: Hey!20:23
niemeyersmoser: Were the AMIs updated today by any chance?20:23
smoserniemeyer, utlemming updated them. https://lists.ubuntu.com/archives/ubuntu-cloud/2011-October/thread.html20:29
niemeyersmoser: What should I be looking for there?20:30
hazmatniemeyer, so i get the same traceback that fwereade got20:31
niemeyerhazmat: What's the traceback?20:31
smoserRefreshed UEC Images of 10.04 LTS (Lucid Lynx) [20110930]20:31
hazmatsmoser, niemeyer http://paste.ubuntu.com/702970/20:31
smoserRefreshed Cloud Images of 10.10 (Maverick Meerkat) [20111001]20:31
smoserRefreshed UEC Images of 11.04 LTS (Natty Narwhal) [20111003]20:31
niemeyersmoser: We're using the Oneiric images20:32
smoserthe dailies?20:32
smoserthey're updated every day20:32
niemeyersmoser: I'll ping utl* about it20:32
niemeyersmoser: Oh, ok20:32
niemeyerWhat the heck.. what's a Pangolin?20:32
smoserhazmat, you have the cloud-init-output.log now20:33
smoserpastebin that20:33
hazmatsmoser, sure20:33
smoser/var/log/cloud-init-output.log (i think that is the path we used)20:33
smoserapt just failed, either archive issue or package failure20:33
hazmatsmoser, what's the cli for pastebin called?20:33
hazmatgot it20:34
hazmatsmoser, http://paste.ubuntu.com/702972/20:34
hazmatsmoser, and http://paste.ubuntu.com/702973/20:34
hazmatwoah.. that second one is fubar20:34
hazmatforbidden on the repo?20:34
smoseri'm asking in #is20:36
niemeyerUgh.. that explains a ton20:36
niemeyerI'll stop the waterfall for the moment20:39
niemeyerfwereade: Thanks for your help on this issue, and sorry for the false alarm20:39
niemeyerfwereade: Is that last branch in?20:40
_mup_juju/ftests r13 committed by gustavo@niemeyer.net20:41
_mup_Fixed authorized_keys handling in ec2-setup.sh when moving ~/.ssh away20:41
_mup_so that the user can still log in.20:41
niemeyerhazmat, smoser: Thanks for debugging this as well folks20:41
niemeyerI'll step down for a cup of coffee.. biab20:41
hazmatniemeyer, np20:41
hazmatthat sounds like a good idea20:41
fwereadeniemeyer, sorry, got pulled away for a while20:48
fwereadehazmat: I presume the weirdness has prevented you from taking a look at that branch?20:49
hazmatfwereade, it has, also trying to debug local provider on jim's machine20:50
hazmatfwereade, many beautiful butterflies.. i'll stop and have a look now20:50
fwereadehazmat, only if you don't know  who *isn't* horribly busy atm, who I could hit up instead?20:51
niemeyerfwereade: Man, I feel bad that we're holding you back still.. that must have been a long day for you20:56
fwereadeniemeyer, don't worry about it20:56
statikheya hazmat, whats the path to the daily ppa that I should use to try out the local provider awesomeness you posted about?20:57
hazmatstatik, ppa:juju/pkgs20:57
statikah ok, I didn't realize that was daily20:58
hazmatbcsaller, +1 on that patch btw20:58
hazmatjimbaker, i'm going to pause on debugging this and switch back to a review for a bit20:59
fwereadehm, niemeyer, just noticed a minor lurking bug in the deploy/upgrade changes: the error when --repository is needed but not included is entirely unhelpful21:02
fwereadeniemeyer: worth a minor to raise FileNotFound from resolve(), or possibly LocalCharmRepository.__init__()?21:03
niemeyerfwereade: How does it look like?21:03
fwereadeniemeyer: NoneType has no attribute .endswith IIRC21:03
niemeyerfwereade: Ouch :-)21:03
fwereadeniemeyer: hm, it's not quite FileNotFound though21:03
niemeyerfwereade: It's CharmNotFound I guess21:04
fwereadeniemeyer: not even quite that, if anything it's RepositoryNotFound ;)21:04
niemeyerfwereade: Hmm21:05
fwereadeniemeyer: or, really, RepsitoryNotSpecified21:05
niemeyerfwereade: Yeah, but only if the charm is local:, and if it's not yet in the env21:06
hazmatfwereade, what's the leapfrog do?21:06
hazmatrev 0 to rev 2?21:06
niemeyerfwereade: and the problem is similar if the repository is provided, but the specific charm isn't there21:06
fwereadehazmat: bumps to deployed+1, which is a jump of >1 from the one in the repo21:07
hazmatfwereade, ah cool21:07
fwereadehazmat: if yu can think of a beeter name I'm all ears :)21:07
* fwereade peers suspiciously at his keyboard21:07
fwereadeit's the tools, I tell you!21:07
fwereadeniemeyer: I feel like the arg parser is really the right place to catch it21:08
fwereadeniemeyer: at the moment anyway21:09
niemeyerfwereade: I can't see how the arg parser is related to this21:09
niemeyerfwereade: This is highly contextual..21:09
fwereadeniemeyer: that might change once we actually can specify a repo in the environment21:09
niemeyerfwereade: "if not repository_path and charm is local and charm not in environment: raise foo"21:09
fwereadeniemeyer: maybe, but it's connected: the reason the bug exists is because the "repository" arg is no longer required21:10
niemeyerfwereade: Sure.. that's quite expected given that we just removed it21:10
niemeyerfwereade: We just need to consider the case where it's not there and raise an error21:10
niemeyerfwereade: We can do that tomorrow, though, in a bug fix21:11
niemeyerfwereade: with you relaxed ;-)21:11
hazmatfwereade, review in one minor, but very awesome21:11
fwereadehazmat: cool, thanks21:11
fwereadeniemeyer: sounds like a plan21:12
niemeyerfwereade: Please just file a bug about this so you can sort it out later21:12
fwereadeniemeyer: will do, I meant "tomorrow" sounds like a plan :)21:12
niemeyerfwereade: Awesome, thank you!21:12
hazmatSpamapS, m_3 .. fwereade has answered the common desire of all formula authors.. auto incrementing versions on upgrade21:16
hazmatnegronjl, ^21:16
m_3ah nice21:16
SpamapS\o/ !!!21:17
SpamapS 21:17
_mup_Bug #868729 was filed: deploy and upgrade-charm give unhelpful errors when repository not specified <juju:New> < https://launchpad.net/bugs/868729 >21:25
fwereade...and that's it for me, nn all21:26
niemeyerfwereade: Woohay!21:28
niemeyerfwereade: Have a restful night! :)21:28
hazmatniemeyer, bcsaller, jimbaker just a heads up, i'm moving over not in progress bugs to the next milestone22:45
bcsallerhazmat: k22:46
niemeyerhazmat: Awesome.. if you find anything interesting on the way that could deserve some attention, please raise a flag22:46
hazmatjimbaker, this one is a duplicate afaics https://bugs.launchpad.net/juju/+bug/84605522:48
_mup_Bug #846055: Occasional error when shutting down a machine from security group removal <juju:New> < https://launchpad.net/bugs/846055 >22:48
hazmatperhaps not22:48
jimbakerhazmat, that's a txaws parsing issue, when we separate out the related bug 86351022:57
_mup_Bug #863510: destory-environment errors and hangs forever <juju:Fix Released by jimbaker> < https://launchpad.net/bugs/863510 >22:57
hazmatjimbaker, this one also seems closed afaics https://bugs.launchpad.net/juju/+bug/82422222:57
_mup_Bug #824222: juju bootstrap should be more robust <juju:Triaged by jimbaker> < https://launchpad.net/bugs/824222 >22:57
hazmatjimbaker, i only see tracebacks when in verbose mode22:57
jimbakerhazmat, yes, that's correct22:58
hazmatjimbaker, and this one is fixed? https://bugs.launchpad.net/juju/+bug/80267822:58
jimbakerhazmat, so let's definitely close that one. i think we could consider expected errors vs not, but that can be in the next milestone22:58
_mup_Bug #802678: Update watch_ports <juju:Confirmed for jimbaker> < https://launchpad.net/bugs/802678 >22:58
hazmatbcsaller, i assume statusd isn't getting merged in the next day or so?23:00
bcsallerhazmat: no23:00
jimbakerhazmat, the test has gone away23:00
jimbakerhazmat, let me see where it lives now, if at all23:00
jimbakerhazmat, let's mark that as fix released, the current test is very close to what is the attached patch as would fail when repeated23:05
jimbakerit doesn't now23:05
hazmatwell its invalid at this point23:05
jimbakersure, that works too23:06
jimbakerhazmat, are we planning on doing jenkins for ec2 functional testing? i thought wtf covered it23:06
hazmatjimbaker, fair enough.. feel free to mark it wont fix23:07
jimbakerhazmat, ok23:07
hazmatkim0, ping.. just curious about the orchestra docs23:08
jimbakerhazmat, what do you think of bug 697093 - this purely seems to be an artifact of our testing setup23:08
_mup_Bug #697093: Ensemble command should return nonzero status code for errors <cli> <juju:New> < https://launchpad.net/bugs/697093 >23:08
jimbaker(at the very least i need to update the description/summary accordingly)23:09
hazmatjimbaker, oh.. perhaps23:11
hazmatjimbaker, yeah... ideally we could test for status codes, but if i'm reading that right, we're really just testing the mocks not the actual exit codes23:11
* hazmat calls it a night23:11
jimbakerhazmat, yeah, it would be nice to get it right, but tricky for sure23:12
jimbakerworthwhile for florence however23:12
jimbakerhazmat, enjoy!23:12
hazmatniemeyer, as far as interesting bugs we might want to still consider for eureka.. https://bugs.launchpad.net/juju/+bug/814987 and https://bugs.launchpad.net/juju/+bug/828326 the latter is a new feature though23:13
_mup_Bug #814987: config-changed hook is retried on 'resolved' even when --retry is not passed <juju:New> < https://launchpad.net/bugs/814987 >23:13
_mup_Bug #828326: need to be able to retrieve a service config or schema from the cli <juju:In Progress by hazmat> < https://launchpad.net/bugs/828326 >23:13
hazmatre the first one.. actually the issue i'm interested isn't really the about config-hooks, but that install error retries are broken23:14
hazmatfor getting to a working state23:14
niemeyerhazmat: Why are they broken?23:14
hazmatniemeyer, because the automatic transition from install to start doesn't happen23:14
hazmatafter the error recovery23:15
hazmatthe unit just stays in the installed state23:15
niemeyerhazmat: Ugh23:16
niemeyerhazmat: Ok23:16
niemeyerhazmat: Sounds like something that would be nice to get done indeed, if we find some spare time23:17
niemeyerI'll kick off the wtf23:55
niemeyerHopefully the repositories are back already23:55
niemeyerUh oh.. Steve Jobs is no more.. :(23:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!