skylerbergmarcoceppi: charm == 'tintri-cinder', self.test_charm == 'deployer'00:02
marcoceppiskylerberg: that's so very wrong00:02
marcoceppiskylerberg: how are you running the tests?00:03
marcoceppiskylerberg: charm_name (and test_charm) are derived from either os.getcwd() or if JUJU_TEST_CHARM is in the environment00:04
skylerbergI am running them with python, but I was in the wrong directory I think. I just realized deployer was the directory I was in. However, it doesn't work in tests either.00:05
marcoceppito get around this you can export JUJU_TEST_CHARM as cinder-tintri, but this should work if you're either running bundletester, juju test, or the test file directly (ala tests/test-file-name)00:05
marcoceppiskylerberg: all the test runners execute from the charm directory root, not the tests directory00:05
marcoceppipython tests/name-of-test should work from the root00:06
skylerbergOkay, I will give that a try00:06
skylerbergThanks, that looks like it solved that problem.00:07
beisnerhi coreycb, looks like ceph/next needs the c-h sync00:22
beisnercoreycb, also fyi, that bit i mentioned earlier buggified:  T-L deploys are blocked on bug 148629300:38
mupBug #1486293: trusty-liberty - multiple packages cannot be authenticated <amulet> <openstack> <uosci> <keystone (Juju Charms Collection):New> <https://launchpad.net/bugs/1486293>00:38
coreycbbeisner, ceph's updated now and I'll take a look at the liberty issue00:53
stubbcsaller: https://bugs.launchpad.net/charms/+source/postgresql/+bug/1486257 could be a real life use case for composer and relation stubs.07:21
mupBug #1486257: make port configurable and send port+protocol on the syslog relation <postgresql (Juju Charms Collection):New> <rsyslog (Juju Charms Collection):New> <https://launchpad.net/bugs/1486257>07:21
magicaltrouthello chaps11:53
magicaltroutquick question11:53
magicaltroutif I want to require tomcat, but specifically tomcat 711:53
magicaltroutcan I configure that in metadata.yaml?11:53
plarswget: unable to resolve host address ''ubuntu-14.04-server-cloudimg-amd64-root.tar.gz'13:25
plarsgetting this when trying to deploy things to lxc under maas13:25
plarshere's a more complete log, any ideas? http://paste.ubuntu.com/12121232/13:25
plarsmarcoceppi maybe? or any idea who to ask? The only significant difference I can see in this environment vs. the one I have at home (that works) is the working one has juju 1.24.4 and the non-working one has 1.24.5, but I haven't updated the one at home to see if it breaks13:27
beisnergnuoy, coreycb - along the lines of liberty uca enablement in the charms, i've selectively synced the fetch bits into mongodb as a merge proposal for review @ https://code.launchpad.net/~1chb1n/charms/trusty/mongodb/sync-fetch-helpers-liberty/+merge/26841313:54
gnuoybeisner, why a selective sync? Does a full sync break mongo>13:56
beisnergnuoy, coreycb - i went minimal since it's not in the next cadence, and didn't want to introduce other potential issues with a full sync.  but i will if you think that'd be best.13:56
beisnergnuoy, i'll re-sync all of it now and let tests run.13:58
marcoceppiplars: is there any proxy in place in this environment?14:37
plarsmarcoceppi: no14:37
marcoceppiplars: it's trying to download the trusty template from the bootstrap node, but it doesn't appear to be working14:38
plarsmarcoceppi: yeah, it seems to complain about the certificate?14:39
marcoceppiit doesn't have a server name for the wget line14:39
plarsmarcoceppi: then later it's just... yeah, bad url14:39
marcoceppilooks like some weird logic tree, possibly an error. what base environment are you using? MAAS?14:39
plarsmarcoceppi: trusty+ppa, maas version is 1.8.0+bzr4001-0ubuntu2~trusty114:40
plarsmarcoceppi: maas version is the same as the one I have at home, and works fine there14:41
marcoceppiplars: if possible I'd try downgrading to 1.24.4 - this might be a regression14:41
plarsmarcoceppi: that was going to be the next thing I tried, would I just need juju and juju-core?14:42
plarsmarcoceppi: and do you have a link to the old version somewhere?14:42
marcoceppiplars: just juju-core, juju is a meta package14:43
marcoceppiif you're on x86 I could upload a file, I don't think they're just floating around14:43
marcoceppiplars: http://ppa.launchpad.net/juju/stable/ubuntu/pool/main/j/juju-core/14:44
plarsmarcoceppi: thanks, downgrading it now, then I'll redeploy everything14:50
g3narohow do i juju deploy local and set the bridge device  to use?15:10
g3narowithouth changing /etc/lxc/default.conff ?15:10
plarsmarcoceppi: it hasn't given up yet, but I ran juju status in another session and it still seems to have the same problem after downgrading: http://paste.ubuntu.com/12125553/15:15
lazyPowerg3naro: to clarify, you want to use a juju local provider, on a different networking bridge, but not edit the default lxc network bridge?15:16
puzzolostrange problems with juju. My lxc machines dont get ips, while maas offers them15:20
g3narowhat is the configuration in /etc/lxc/default.conf ?15:25
puzzoloUSE_LXC_BRIDGE="false"  # overridden in lxc-net15:26
puzzolo[ -f /etc/default/lxc-net ] && . /etc/default/lxc-net15:26
puzzoloeven configuring lxc statically wont do the trick15:28
marcoceppiplars: this seems like a misconfiguration somehwere, can you confirm that the agent-version for node 0 is 1.24.4?15:29
puzzololxc-net, which overrides lxc.conf uses lxcbr015:30
plarsmarcoceppi: hmm, no it seems to be 1.24.5, but I downgraded... does it cache somewhere?15:30
g3narohmm do you have the lxcbr0 made?15:30
plarsmarcoceppi: and I did re-bootstrap after downgrading15:30
marcoceppiplars: agent stream will always look for latest tools, you may need to bootstrap with an explicit version15:31
plarsmarcoceppi: of course, I just downgraded juju-core, not juju15:31
plarsmarcoceppi: how do I do that?15:31
marcoceppiplars: I'm not entirely certain now15:32
puzzolog3naro: machine was provisioned by juju. lxbr0 is there. Still in containers config i get: lxc.container.link = juju-br015:32
puzzoloi changed config to not using lxc bridge. lxcbr0 disappeared15:40
puzzoloproblem is still there15:41
g3narotry this15:41
g3naroedit your ~/.juju/environments.yaml configuration15:41
g3naroyou have an option to specify the bridge device there15:42
puzzoloi am no expert, as this is my first deployment with lxc containers. And i did not quit get which layer of configuration comes first. For which container I get a config file with points to juju-br0, which would expose all containers to maas dhcp. And that should be ok, making the services reacheable. lxcbr0 instead is a natted network for lxc continaers. This seems like a bug to me.. still I cant get to understand why it fails.15:49
puzzolojuju-br0 is the right bridge for lxc-containers or should services be natted?15:50
puzzolostrange is that with default deployment, containers do point to juju-br0, do ask dhcp .. and maas does offer them. ufw disabled anywhere. I must use tcpdump at least to see where the udp/tcp breaks15:55
asanjarkwmonroe: updated and tested hbase with jujubigdata416:00
kwmonroeexcellent asanjar!16:02
plarsmarcoceppi: yeah, sshing to node 0 I see that it had the old version, but also seems to have downloaded 1.24.516:02
puzzolomu problem seems to be related to juju-br0 bridged to eth0, which goes to maas internal network. juju-br0 does receive dhcp offers from maas for all containers, still wont "redirect" them to veth interfaces16:38
lazyPowerpuzzolo: is this a LXC networking reachability issue you're encountering? eg: Juju deploy --to lxc:# and then attempting to relate/route to those containers fails?16:48
jogarret6204hi all.   I have juju 1.24.3 upgrade to 1.24.5 stuck.  anyone tell me how to "undo" or kik it along to finish?16:53
plarsmarcoceppi: is there some way to force it to downgrade? or to debug where things might be going wrong and why it's getting those errors in 1.24.5?16:53
lazyPowerjogarret6204: any further details than the upgrade is stuck? has the bootstrap node completed the upgrade cycle and its now stuck pushing out to the agent nodes?16:54
lazyPowerjogarret6204: also do you have any debug log output during the upgrade that could help point us to a root cause? juju debug-log can assist here, but you might have to specify a legnth and scroll back through the election spam if its been a long while since you initiated the upgrade   juju debug-log -n <number>16:55
puzzololazyPower: indeed. I tryed deploying a juju charm, landscape-maas-dense. It installes apache2 on phys0, inquiring maas for a new node. On that machine it builds 5 lxc containers. Everything ok, till containers attached on juju-br0 need an ip. They'll ask for it. Maas will provide them, but ack all stop at juju-br0.16:56
jogarret6204lazyPower: ERROR WatchDebugLog not supported16:58
jogarret6204dont think bootstrap node is upgraded either16:58
puzzoloit is truly frustating.. as i cant get this continaers to get an ip, or understand where i am wrong.16:58
jogarret6204machine 0: agent-version:
lazyPowerdimitern: ping16:59
lazyPowerjogarret6204: which provider are you using?17:00
jogarret6204I'm in no hurry this is a lab.17:00
lazyPowerhmm, its not giving you a debug log, thats a weird bug17:00
jogarret6204I have machine 0 as a VM.  I can cat the machine-0 log there...17:02
lazyPowerpuzzolo: we had some networking changes land in 1.24 that should have addressed that17:03
lazyPowerbut it appears we've missed the TZ window to contact the developer, let me put out some feelers and see what i can turn up about this17:04
lazyPoweri've run into this with other substrates however, where LXC networking isn't adding the forwarding rule to the machines and therefore the containers are unreachable outside of the host17:05
dimiternlazyPower, pong17:07
lazyPowerdimitern: heyo sorry for the late ping :)17:07
dimiternlazyPower, no worries :) what's up?17:07
puzzololazyPower: it felt like a bug. Anyway to check this is my issue?17:07
lazyPowerdimitern: i've had a couple questions over the networking in 1.24 wrt lxc containers. Correct me if i'm wrong but on certain substrates the forwarding should "just work" and cross host container communication w/ lxc should be enabled?17:07
lazyPowerpuzzolo: this conversation above is related to your scenario :)17:09
puzzoloi guessed that guys :*17:09
lazyPowerjogarret6204: if you can grab the all-machines log that would be helpful17:09
lazyPowerjogarret6204: that machine-0 log might have the details of whats happening with the upgrade as well, so thats probably a good place to start17:10
jogarret6204relevant log message seems to be this one17:10
jogarret6204sorry - was trying it.. all of it17:10
jogarret6204failed to fetch tools from "": bad HTTP response: 400 Bad Request17:10
jogarret6204I see my lab proxy changed when I try that...  let me go fix that17:12
=== lukasa is now known as lukasa_away
=== lukasa_away is now known as lukasa
puzzololazyPower: can this "forwarding" thingie be handput, or should i just wait for devs to do the dev's things?17:29
lazyPowerpuzzolo: i'm  not certain what is actually put on the host in terms of forwarding, so i'm pending a response from dimitern17:29
=== tvansteenburgh1 is now known as tvansteenburgh
puzzololazyPower: we'll wait together then. Strange is that those containers do reach maas, but get nothing back.17:30
puzzoloreq ok. offer ok. never "acks"...17:31
puzzolobut sure is forwarding issue, as static ips wont do either17:31
lazyPoweryeah, i think its just an iptables rulechain or route thats added17:32
lazyPoweri'm not sure which17:32
lazyPowerits been a while since i've dealt with this by hand17:32
jogarret_6204lazyPower: fixed proxy.  logs still showing upgrade in progress in error logs of juju state VM.  but nothing is upgrade.  can I back out and redo upgrade?17:34
jogarret_6204On puzzolo issue - you guys check ebtables and ufw too?17:34
lazyPowermarcoceppi: do you know if there is a way to stop an in progress upgrade?17:34
dimiternlazyPower, hmm in 1.24 we have this only in a few places17:34
lazyPowerdimitern: AWS, and MAAS correct?17:34
puzzololazyPower: sure this thing affects quite a bunch of charms. I tried landscape just to get the feeling of what will be openstack deployment.17:34
dimiternlazyPower, a few more - AWS and MAAS with address-allocation feature flag17:35
lazyPowerahh so thats behind a feature flag during deployment?17:35
dimiternlazyPower, by otherwise by default only on MAAS17:35
lazyPowerpuzzolo: you're using maas correct?17:35
dimiternlazyPower, yeah, it's possible to lift it for 1.25, but it's not decided yet17:35
=== lukasa is now known as lukasa_away
=== lukasa_away is now known as lukasa
firlyay, I will be able to come to the juju charm summit17:54
lazyPowerfirl: awesome!17:55
firlyeah, I work remote in texas but the office I Work with is 2 miles from the summit so it’s perfect17:56
=== natefinch is now known as natefinch-afk
jcastromarcoceppi: which would you say is a good hello world for explaining actions to people18:06
lazyPowerjcastro: a great one would be to get troubleshooting information from a system18:06
lazyPowerdump the versions of software, configuration data, and relevant things from the service18:07
jcastrosorry I meant, hello world example charm18:07
jcastrolike, one I can point to people and then explain18:07
lazyPoweri got meta there fo ra second, sorry18:07
jcastrono worries18:07
lazyPoweretcd has a system health action18:08
lazyPowervery simple to parse18:08
jcastrocool, any others?18:08
lazyPoweri have some actions for DroneCI18:09
jcastroperfect, anyone else have some decent actions they want to share?18:09
puzzololazyPower: maas18:11
puzzolojogarret_6204: i checked ufw only. defaulting it to accept all, or even disabling it. From maas down to phys to even lxc containers18:13
puzzolojogarret_6204: didnt try ebtables.. as it never touched me in these years to custom firewall  bridge prots18:14
puzzolo[    8.062781] IPv6: ADDRCONF(NETDEV_UP): vethQ3EQVL: link is not ready18:18
puzzolo[    8.062787] juju-br0: port 2(vethQ3EQVL) entered forwarding state18:18
puzzolo[    8.062790] juju-br0: port 2(vethQ3EQVL) entered forwarding state18:18
puzzolo[    8.065762] juju-br0: port 2(vethQ3EQVL) entered disabled state18:18
jogarret_6204puzzolo:  Those are just some things I have tried.  I'm not experienced with containers.18:18
plarsmarcoceppi: it's worth noting that if I use the --no-check-certificate flag to wget, I can get it to download from the url in the error, but I don't think I can inject that anywhere18:21
plarsmarcoceppi: It downloads ok, but still gives: WARNING: certificate common name ‘*’ doesn't match requested host name ‘’18:21
puzzoloon maas, everything seems to be working just fine18:23
puzzoloAug 19 20:23:01 maas-1 dhcpd: DHCPDISCOVER from 00:16:3e:43:ce:3f (juju-machine-11-lxc-0) via eth118:23
puzzoloAug 19 20:23:01 maas-1 dhcpd: DHCPOFFER on to 00:16:3e:43:ce:3f (juju-machine-11-lxc-0) via eth118:23
puzzolotell me if i'm flooding too much guys. Bridge configuration, seems to be outstanding18:27
lazyPowerpuzzolo: I'm not positive on where to go from here. I need to talk with dimiter more tomorrow in the AM to get a better view of whats implemented and how to consume it18:39
lazyPowerpuzzolo: once i've had that conversation i should be able to lend a better helping hand. You're the third user this week thats had container networking issues and we've got work landed that should make any manual intevention moot18:40
lazyPowerjuju should be able to do what is right for that networking component in the stack, and if we do anything by hand its not as reproduceable as juju doing it for you :) so i hesitate to help in the creation of a snowflake18:40
puzzoloa small bit different in my setup, is that i am using a kvm machine with libvirt bridged networking.18:41
lazyPowerpuzzolo: i can circle back tomorrow if you're going to be here, i'll also be pinging the list with my findings to help the broader user audience in general.18:41
plarsmarcoceppi: I tried to fake it out by resetting the symlink under /var/lib/juju/machine-0, but it replace it. Also tried just replacing the 1.24.5 binary with the 1.24.4 one, but it just gets stuck in an endless update loop if I do that, and doesn't let me deploy :(18:42
puzzolothank you for your time. I'll check what those users had on lxc netwroking, to get a better picture myself18:42
lazyPowerpuzzolo: sorry i didn't have better info today. this is very much new stuff that we've been working on for the last cycle18:44
lazyPowerso its a known problem, but we've got some tools out there to help, i jsut need to re-up on that info and we should have you sorted in short order18:44
puzzolono problem. One thing did catch my eye now, on lds-1, the first service unit phys machine, where containers are deployed for the other services in the charm bundle:18:47
puzzolo2015-08-19 18:39:28 INFO juju.networker networker.go:163 networker is disabled - not starting on machine "machine-11"18:47
plarsah, hang on, it was way easier. I didn't see that upgrade-juju takes a version parameter, and it doesn't seem to try to circumvent what you specify there :)18:47
plarsmarcoceppi: ok, I'm getting farther... once it's completely done, I'd like to file a bug on this. Is lp the best place for bug reports? github?18:57
plarsmarcoceppi: I've just about convinced myself it's a regression in 1.24.518:57
=== scuttle|afk is now known as scuttlemonkey
puzzololazyPower: mumble mumble, can it actually be a problem regarding my managed switch?19:42
lazyPoweri wouldn't think so, but its possible19:57
beisnergnuoy, coreycb - mongodb c-h sync for liberty uca - mp ready for review @ https://code.launchpad.net/~1chb1n/charms/trusty/mongodb/sync-fetch-helpers-liberty/+merge/26841319:58
coreycbbeisner, I'll have to defer to gnuoy.  I can't land to mongodb19:58
beisnercoreycb, ok np, thanks.   fyi, mbruzek is on the mysql sync review as soon as tests complete.19:59
beisnero/ mbruzek :-)   the queue was thick today, but that mysql amulet test is next up.20:00
puzzoloupgrading to 1.24.5 over 1.24.420:00
mbruzekin meeting beisner I will get to that after that20:00
beisnermbruzek, yep np, the test will be +~1hr i imagine.  thanks a ton.20:00
=== natefinch-afk is now known as natefinch
puzzololazyPower: http://askubuntu.com/questions/615433/landscapes-openstack-installation-fails-due-to-containers-unable-to-obtain-an-i after a couple of hours of diggin' around, this guys in the link seem to have my same issue. I am not using esxi, but libvirt alone, with kvm as phys0 where lxc get created. I will try to see if my problem is related to libvirt other then juju.20:18
puzzoloI am starting to think that problem is related to macvlan on libvirt20:30
puzzolowhich is a sort of  cannibilized bridge, working splendidly for kvm networking... but is not aware of lxc containers mac addresses inside kvm.20:31
lazyPowerpuzzolo: that is very possible20:32
lazyPowersorry i've been unresponsive, in a community hangout with some users of our k8's bundle20:32
lazyPowersplit brain today :)20:32
skylerbergI need to change the configuration in nova-compute (I need to pass in some nfs protocol settings). However, there doesn't seem to be an applicable interface for me to connect to from my charm. What is the way forward? Add an interface to nova-compute? Hijack an interface meant for something else?20:37
=== blr_ is now known as blr
lazyPowerskylerberg: is this just for adding NFS support? or is this to extend into a new region of NFS goodness?20:54
lazyPowerskylerberg: meaning the NFS charm relation20:54
skylerberglazyPower: I just need to edit a couple of settings in nova.conf (nfs_mount_options and maybe cpu_mode).20:57
lazyPowerskylerberg: its a bad idea to have something else editing nova.conf outside of the nova charm20:57
lazyPowerthat smells of config race conditions should nova ever receive a hook event firing that will re-write the template20:57
=== scuttlemonkey is now known as scuttle|afk
=== scuttle|afk is now known as scuttlemonkey
lazyPoweriirc there is a config manager lib, that you can send data to over the relation. we have a similar construct in teh cinder charm20:58
lazyPowerbeisner: does nova have the same config manager that cinder does?20:58
lazyPowerskylerberg: let me loop in our test master of openstack, if thats the case i should be able to get you a recommended path forward that has a high liklihood of getting accepted as a PR for the charms20:58
skylerberglazyPower: Right, so I am using the cinder storage-backend relation and that is working great. I just don't see an equivalent in nova.20:58
skylerberglazyPower: It might be nice to have a generic interface for editing nova's config. Right now it looks like all the interfaces are very specific (ceph, rabbit, etc.)21:00
lazyPowerskylerberg: thats by design as its p2p orchestration in that instance :)21:00
lazyPowerwhich makes it ery clear to anyone thats relating into nova, what data is being sent/received21:00
lazyPowerand allows nova to respond in kind based on whats incoming, eg: installing any storage drivers, et-al21:01
beisnerhi lazyPower - they both use the contexts approach with regard to collecting config data and rendering the conf template, which keeps things safe-ish.21:01
lazyPowerbeisner: so is it safe to link to the cinder-vnx charm to illustrate how that hsould be used?21:01
lazyPoweror are they different context managers?21:01
puzzolothis problem is killing me21:04
beisnernot sure i'm fully following.  i think the cinder-vnx + cinder charms are a good example of a subordinate affecting a principle's config if that's what you mean.21:05
lazyPowerbeisner: well skylerberg is wanting to add NFS storage backend to nova21:05
lazyPowerand i'm recommending using the context approach to add that config update, vs a racey interface approach that edits the config outside of the context aware template approach21:06
beisneroh neat, for instance image storage?21:06
beisnerif so, we already have nova-compute plumbed for ceph-backed instance storage, and it may be worth looking at that code21:07
beisnercaveat there of course is:  all instance disk i/o then traverses the wire, so the network needs to be ROCKING.21:08
lazyPowerskylerberg: seems like you've got some template code you can consume, and i would recommend adding an interface unless you're going to recycle the same data coming from nfs, then use the NFS provided interface :)21:08
lazyPowersorry that wsa a bit round about, i just wanted to make sure you were setup for success vs running into a racing config scenario. I've had my fair share of those and have lost sleep over it.21:09
beisner+1000 for race avoidance21:09
skylerberglazyPower, beisner: So the way forward is to add an interface to nova-compute that uses the same type of mechanism as the storage-backend interface in cinder to update the config?21:12
beisnerlazyPower, skylerberg - so re: design logic in adding new features, i'd prefer to pull in one of the folks like coreycb, gnuoy, wolsen or dosaboy who are primary authors.  i become familiar with the code paths through testing, but i don't generally pave new paths in these charms as such.21:22
coreycbskylerberg, sorry, catching up21:23
beisnerthat, and inspecting how we've already done the ceph-backed instance storage in nova-compute.21:23
coreycbskylerberg, there's a config-flags optoin in nova.conf to edit general config options, but be careful21:25
coreycbskylerberg, in nova-compute that is21:25
coreycbskylerberg, nevermind me, saw you had a question about general nova.conf settings earlier21:28
skylerbergcoreycb: It seems like I would either need to still use an interface so that users could connect my charm to nova and then my charm would set the config-flags or I would need to have instructions with my charm saying that they need to set this option on nova-compute, which seems less than ideal.21:28
skylerbergcoreycb: What I need to do with nova-compute is pretty simple, it just needs to set a couple config options and make sure the service is restarted so that the config is loaded. It should be a lot like how cinder-vnx connects to cinder.21:29
wolsenskylerberg, this configuration setting needs to be applied after a subsequent charm is related to it?21:32
skylerbergwolsen: That is what I am imagining. Someone deploys cinder and nova-compute, then they connect my charm to both to alter their configurations to use the proper storage backend.21:33
wolsenskylerberg, yeah unfortunately thare's not a generic interface that exists to relate nova and cinder backends, its more specific (e.g. the ceph-client interface which ceph can be related with nova-compute)21:35
wolsenskylerberg, though there's the possibility of having a shared-storage relation or something similar that may be more generic21:36
skylerbergwolsen: I think my use case is even more generic than relating storage backends, it is just editing the config settings. Could we add a generic config setting interface?21:37
wolsenskylerberg, sorry my son & I are both sick today - can I get back to you?21:37
skylerbergwolsen: Yeah, no problem. Take it easy.21:38
beisnerthanks skylerberg, lazyPower - i've also got to run, eod.  i think most of the openstack charmers are also end-of-day.  it may be worth starting a thread on the juju mailing list to state the end goal, gather input, ideas, etc.21:40
beisner (thanks too, coreycb wolsen  )21:40
wolsenskylerberg, one of my concerns is that adding a generic config interface for the charms will now compete with the charm config itself21:41
wolsenskylerberg, so it becomes ambiguous if both the charm itself and a related charm specify "create this config option with this value" - which one is the right one?21:42
skylerbergwolsen: I see. Yeah, I will think about what makes sense for my use case and get back to you.21:44
puzzololazyPower: nothing. I tried everything i could. from kernel flags on bridge till promisc and "allmulti" options on all layers  of network interfaces. Still nothing. Hope to hear from you and dmnitry tomorrow21:56
lazyPowerpuzzolo: i've got you at the top of my list to have that discussion before i switch feet back to k8's :)21:56
puzzolothank you dude :*22:03

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!