/srv/irclogs.ubuntu.com/2016/07/12/#juju-dev.txt

katcoaxw: hey thanks for the review; i responded00:05
mgzwallyworld: nvm by the way, saw after you'd put up a pr against the split out repo that removed the rfc text as well as other things00:08
wallyworldmgz: yeah, sorry, the pr did a couple of things, including removing the bad file00:12
axwkatco: thanks/sorry for misunderstanding00:55
mupBug #1602053 opened: juju deployed lxd containers are missing a default gateway when configured with multiple interfaces <juju-core:New> <https://launchpad.net/bugs/1602053>02:04
mupBug #1602054 opened: juju deployed lxd containers are missing a default gateway when configured with multiple interfaces <juju-core:New> <https://launchpad.net/bugs/1602054>02:04
mupBug #1602053 changed: juju deployed lxd containers are missing a default gateway when configured with multiple interfaces <juju-core:New> <https://launchpad.net/bugs/1602053>02:17
axwwallyworld: are you busy? could you please take a look at http://reviews.vapour.ws/r/5221/?02:32
wallyworldsure, give me a few minutes02:33
axwnp02:33
mupBug #1602067 opened: [2.0b11] Repeated "log-forwarder" errors in debug-log with Azure provider <juju-core:New> <https://launchpad.net/bugs/1602067>02:53
axwwallyworld: PTAL03:15
wallyworldok03:15
wallyworldaxw: tv, lgtm03:17
wallyworldty03:17
axwwallyworld: thanks03:18
axwwallyworld: I'll do a bootstrap && deploy just to be paranoid...03:18
wallyworldsgtm03:18
veebersaxw: Hi, seems I can reproduce https://bugs.launchpad.net/juju-core/+bug/1599779 everytime using log-forwarder and setting debug level to INFO (as per convo from yesterdas). It useful for you if I file a new bug or just comment on that existing one?03:36
mupBug #1599779: Very high CPU usage from 'creating cloud image metadata storage'  emitted in debug log every second <juju-core:New> <https://launchpad.net/bugs/1599779>03:36
axwwallyworld: ^^ are you looking at this?03:37
wallyworldaxw: not yet, hacking around in unit tests, will have a look next03:38
axwveebers: I *think* we need a separate one, really depends on the root cause. I guess a separate one until we analyse it, and then we may mark as dup03:38
axwveebers: there's an issue to do with how we start/stop workers in juju which is (I think) the cause of the linked bug, but I think there's more to the log-forwarding issue03:39
veebersaxw: ack, I'll file a bug for it, it can always be marked as dupe or whatever if needed.03:40
axwveebers: thanks03:41
veebersnw03:44
mupBug #1602084 opened: log-forwarding with level set to INFO starts endless logging loop. <juju-core:New> <https://launchpad.net/bugs/1602084>03:56
mupBug #1602084 changed: log-forwarding with level set to INFO starts endless logging loop. <juju-core:New> <https://launchpad.net/bugs/1602084>04:02
mupBug #1602084 opened: log-forwarding with level set to INFO starts endless logging loop. <juju-core:New> <https://launchpad.net/bugs/1602084>04:08
mupBug #1580497 changed: juju2 on maas permanent " connection is shut down" msg and loss of connection <cpe-sa> <orangebox> <juju-core:Expired> <https://launchpad.net/bugs/1580497>04:26
wallyworldaxw: when you get a chance, here's some logfwd fixes http://reviews.vapour.ws/r/5223/04:49
wallyworldveebers: are you able to share your notes on how to manually set up a test environment for the log forward - ie what charm you used, the steps to generate certs, the charm set up etc?04:52
veeberswallyworld: I'm in the process of writing a ci test for it (which would be easy to reproduce it), but I can also share the steps that I took to manually setup and test it04:56
veeberswallyworld: I'm just about to EOD though and head off to a conference meeting. I could throw something together for you after that but not sure when that would be for you04:57
wallyworldveebers: tis good, i'll see what i can get done and we can sync up tomorrow if needed04:57
wallyworldveebers: i will hopefully have landed by then code to allow the log fwd to be toggled on/off via a syslog-enabled config flag04:58
veeberswallyworld: nice! :-)04:59
wallyworldso you have have it set up and turn on / off as needed04:59
wallyworldof start without and set the config later04:59
wallyworldor04:59
veeberssweet, I'll ensure to have a test do both (at bootstrap and later on)04:59
wallyworldawesome, ty04:59
wallyworldi'll look at this cpu issue05:00
veeberswallyworld: if it'll help I can provide rough and ready details now and refine later on? (i.e. I have a script that generates a cert + key given an IP address for the server + the config used on the rysync sink side)05:01
wallyworldveebers: great ok, if it's easy to send through, otherwise i'll hack something up05:02
axwwallyworld: looking05:04
veeberswallyworld: heh, it's not pretty but it's the steps I used to bootstrap my testing env: https://pastebin.canonical.com/160714/05:08
=== urulama|__ is now known as urulama
wallyworldty05:08
veeberswallyworld: oh, I may not have mentioned that once you expose the rsyslog chamr, the public ip address there needs to be plugged into the cert generator script as that needs to be embedded into the client cert. There is a comment in the script where that happens (like I said, rough and ready)05:10
wallyworldveebers: no problem, the SAN list is a pita for sure05:11
veebersI have the command line details if you want to generate certs that way too. I know now a lot more about create certs then I did a couple of days ago05:15
lazyPowerwallyworld - bruzer and i hacked on the tls layer quite a bit this cycle. we're backending with EasyRSA but lib/tls.py has some helpers you may be able to pull out for helping with SANS generation/listing/manipulation05:41
lazyPowerhah just kidding, i mean we put it in the reactive bits - https://github.com/juju-solutions/layer-tls/blob/master/reactive/tls.py#L29105:42
wallyworldlazyPower: thanks! will take a look if needed. i hacked up a cert last week and did the SAN dance by hand. at this stage, i just need the one set of certs05:43
lazyPowerack, just trying to help if there's anywhere we can share/collab i'm game to do so. My go-fu however, is non-existant.05:43
wallyworldthak you, appreciate it05:48
axwwallyworld: reviewed05:55
wallyworldta05:56
wallyworldaxw: i can't reproduce the CPU issue - the log forward worker uses a http endpoint to stream out the log entries so that doesn't create a loop. i quickly did an initial test by hacking the worker to write forwarded logs to a file rather than a log sink. but indications are that it's not related to any log forward issue (I did test on the latest code though that hasn't landed yet)05:56
axwwallyworld: it does make API calls to update the last-sent log though doesn't it?05:57
wallyworldnot that i saw, but i didn't dig into that part so much05:57
wallyworldi am tailing a file with forwarded data and it seems to behave as expected05:58
axwwallyworld: okey dokey, I guess it's just the same issue as previously reported then06:10
wallyworldaxw: could be. i did also start another model and it still behaved itself06:11
urulamawallyworld: model inheritance ... looks to be working. bootstrapping with test-mode config did go into all models07:04
urulamanot sure if that was supposed to happen or not yet :)07:04
wallyworldurulama: you put that in cliuds.yaml?07:06
wallyworldclouds.yaml07:07
wallyworldthat bit works07:07
wallyworldaxw: off to soccer, will push final changes when i get back, hopefully can land tonight so chris can pick up tomorrow, we'll see how i go07:07
axwwallyworld: okey dokey. I have another change to push shortly also.07:08
axwwallyworld: finally, http://reviews.vapour.ws/r/5224/ -- sorry it's a little on the large side. nett negative though if that's consolation07:36
dooferladjam, voidspace - hangout time09:02
rogpeppe1anyone know the best way of manually destroying a juju lxd controller and all its resources (juju kill-controller isn't doing the job - i bootstrapped with a different, incompatible, juju version) ?09:48
babbageclunkrogpeppe1: I think it's just `juju unregister <controller>` and then `lxc delete ` the machines.10:02
rogpeppe1babbageclunk: what about all the jujuds ?10:02
babbageclunkrogpeppe1: Well, I'd be surprised if they stayed after you killed the containers.10:03
babbageclunkrogpeppe1: (I might be misunderstanding your question.)10:03
rogpeppe1babbageclunk: lxc doesn't seem to have a delete subcommand10:04
rogpeppe1babbageclunk: hmm, it does actually10:05
babbageclunkrogpeppe1: phew10:05
babbageclunkrogpeppe1: was getting confused there.10:05
rogpeppe1babbageclunk: it gave me a usage message but i must've typed the wrong command10:07
rogpeppe1babbageclunk: jujuds are gone now, ta!10:07
babbageclunkrogpeppe1: :)10:07
urulamababbageclunk: hey ... did you see my last comment here https://bugs.launchpad.net/juju-core/+bug/1593828 ?10:09
mupBug #1593828: cannot assign unit E11000 duplicate key error collection: juju.txns.stash <ci> <conjure> <deploy> <intermittent-failure> <oil> <juju-core:In Progress by 2-xtian> <https://launchpad.net/bugs/1593828>10:09
urulamababbageclunk: could you verify with a simple test that it actually work or not?10:09
babbageclunkurulama: yes! I started replying to it and then I started making a bug10:09
urulamababbageclunk: you know what it could be?10:09
babbageclunkurulama: I see the same behaviour - I don't think it's anything to do with that bug.10:10
urulamababbageclunk: +1 on that10:10
babbageclunkurulama: Well, not really - the failed machines have cloud-init logs that stop before installing all the juju stuff.10:10
babbageclunkurulama: but I don't know why that is.10:10
urulamababbageclunk: oh, ok, i'll pay more attention to cloud-init logs, thanks10:11
babbageclunkurulama: I'll just finish grabbing logs and put them on the bug, then I'll put a link to it into the other comments.10:11
urulamababbageclunk: awesome, ty10:11
babbageclunkurulama: here it is: https://bugs.launchpad.net/juju-core/+bug/160219210:16
mupBug #1602192: deploy 30 nodes on lxd, machines never leave pending <juju-core:New> <https://launchpad.net/bugs/1602192>10:16
urulamababbageclunk: thanks, bumping importance, and preferred milestone10:19
mupBug #1602192 opened: deploy 30 nodes on lxd, machines never leave pending <juju-core:New> <https://launchpad.net/bugs/1602192>10:21
babbageclunkurulama: Thanks - I don't know the protocol on those yet, never sure whether I should be setting them.10:24
jambabbageclunk: https://bugs.launchpad.net/juju-core/+bug/160219211:03
mupBug #1602192: deploy 30 nodes on lxd, machines never leave pending <juju-core:New> <https://launchpad.net/bugs/1602192>11:03
axwwallyworld: sorry, I could've sworn the sink name was set to the sensible thing :/11:03
jamI certainly believe that this is reproducible for you, but we need the logs from machine-0 which is doing the provisioning11:03
jamvs the individual machines that fail.11:03
axwwallyworld: do you agree that it should be what I described?11:03
babbageclunkjam: ok, I'll add those.11:04
wallyworldaxw: tis ok, it should be changed i think yeah11:04
axwwallyworld: I'm thinking we'd have something like storage pools, and in this case the sink name is equivalent to storage pool name11:04
axwsorta11:04
axwso that remains fixed for the lifetime of the sink, but the config may change11:05
wallyworldyep11:05
jambabbageclunk: so a cloud-init-output.log that stops fairly early... sounds a bit like just slow cloud-init due to your machine trying to run 30 containers at the same time. not sure11:05
jambabbageclunk: the other thing that might be useful is 'cloud-init.log' vs 'cloud-init-output.log' if cloud-init is failing to complete.11:05
babbageclunkjam: certainly my machine wasn't that happy when they were starting up.11:05
jammachine-0.log would be if Juju provisioning is failing to request enough containers from LXD11:06
jamcloud-init.log is for when cloud-init itself breaks11:06
jamcloud-init-output.log is for when cloud-init thinks it's happy, but the initialization we request of it is wrong.11:06
jam(its been rare that cloud-init.log is the problem, which is why it isn't my first go to)11:06
babbageclunkjam: makes sense - adding the controller machine-0.log and cloud-init.log from a failed machine11:10
mupBug #1602084 changed: log-forwarding with level set to INFO starts endless logging loop. <juju-core:New> <https://launchpad.net/bugs/1602084>11:21
urulamabut the LXD start 90s apart, which was enough time that they were up and running before another one hit11:29
urulamababbageclunk, jam ^11:29
babbageclunkurulama: I realised I hadn't rebuilt with my mgo patch, which didn't seem to be the problem but definitely muddied the water. So I'm redoing it now.11:40
jamurulama: just a dup of bug #159977911:49
mupBug #1599779: Very high CPU usage from 'creating cloud image metadata storage'  emitted in debug log every second <juju-core:New> <https://launchpad.net/bugs/1599779>11:49
jamnothing to do with INFO level11:49
jamand I don't think it is actually a 'loop' in the general sense, just something that we're doing poorly.11:50
urulamajam: so you assume when CPU issue is resolved, then provisioning will work as well? I doubt that, my CPU was not using all cores 100%12:02
jamurulama: no, I think provisioning is separate from bug #159977912:10
mupBug #1599779: Very high CPU usage from 'creating cloud image metadata storage'  emitted in debug log every second <juju-core:New> <https://launchpad.net/bugs/1599779>12:10
jamit *might* be related if the provisioning failure is because there is too much CPU load causing the provisioned machines to fail to download the tools they need12:10
urulamakk12:10
mupBug #1602231 opened: juju status should use natural sort for units <status> <juju-core:New> <https://launchpad.net/bugs/1602231>12:27
=== akhavr1 is now known as akhavr
mupBug #1602237 opened: log forwarding config watcher needs to be implemented <juju-core:Triaged> <https://launchpad.net/bugs/1602237>12:48
axwwallyworld: reviewed your branch12:53
wallyworldgreat ty12:54
wallyworldi'm adding to the feature test12:54
wallyworldaxw: i replied to one of the comments, see what you think13:00
axwwallyworld: yeah, that's what I was suggesting. log but otherwise ignore the invalid config. carry on with the existing sink13:01
axwin which case "enabled" shouldn't be touched I think?13:02
wallyworldyep, am fixing that. but i still think the check for cfg.Enabled goes first before validating13:02
axwwallyworld: I'm heading off, need anything from me before I go?13:10
wallyworldaxw: all good, ty, will run live test now. just added feature test13:10
axwwallyworld: thanks. good night13:10
wallyworldttyl13:10
natefinchwallyworld: I'll start looking at interactive bootstrap ASAP.14:12
wallyworldok, ty14:12
alexisb__frobware, I am going to be late to our 1x1, will ping you14:20
frobwarealexisb__: ack14:20
mupBug #1602067 changed: [2.0b11] Repeated "log-forwarder" errors in debug-log with Azure provider <juju-core:New> <https://launchpad.net/bugs/1602067>14:33
babbageclunkrogpeppe1: Thanks for the review!14:39
rogpeppe1babbageclunk: np. we'll keep fingers crossed.14:40
babbageclunkrogpeppe1: Also, I didn't understand this morning what you were saying about jujud processes hanging around - I didn't know that processes in lxd containers were visible on the host!14:41
rogpeppe1babbageclunk: they showed up with ps alxw | grep jujud14:41
rogpeppe1babbageclunk: maybe i should run my desktop in a container...14:42
babbageclunkrogpeppe1: And with pgrep too. Maybe I should be less gung-ho about killing "stray" mongo processes on my machine in that case.14:43
rogpeppe1babbageclunk: it's ok, they start again immediately if you kill 'em14:43
babbageclunkrogpeppe1: yay!14:43
alexisb__frobware, ok, I am free will join the HO14:46
frobwarealexisb__: omw14:49
babbageclunkis that really alexisb_? Or is it someone pretending to be alexisb_?14:52
alexisb__:)14:53
alexisb__it is alexisb with a bad connection14:53
natefinchhmm... that's exactly what someone pretending to be alexisb_ would say....14:54
natefinchkatco: I guess no standup?15:14
katconatefinch: oh sorry, didn't know you were here15:14
jammgz: ping15:14
natefinchkatco: I'm here :)  just you and I, I think, though15:15
katconatefinch: yeah. let15:15
katconatefinch: let's defer until we have at least 315:15
katconatefinch: (i.e. thurs.)15:15
natefinchkatco: I'm just doing interactive bootstrap.15:15
katconatefinch: i'm still working on audit15:15
natefinchkatco: cool. standup done! ;)15:16
mgzjam: yo15:16
katconatefinch: lol yep15:16
katconatefinch: wb btw15:16
natefinchkatco: thanks.  Sorry about not updating the calendar that I'd be out yesterday.  Swear I did it, but it was like two weeks ago, so who knows.15:16
katconatefinch: no worries15:16
rogpeppe1anyone know how you're supposed to switch accounts from the command line?15:18
=== rogpeppe1 is now known as rogpeppe
rogpeppejuju switch doesn't seem to support account switching15:18
rogpeppewell, in β11 anyway15:19
cheryljbabbageclunk: have you pinged smoser about that cloud init issue for bug 1602192?15:19
mupBug #1602192: deploy 30 nodes on lxd, machines never leave pending <lxd> <juju-core:Triaged> <https://launchpad.net/bugs/1602192>15:19
babbageclunkcherylj: no - <looks up smoser>15:20
cheryljbabbageclunk: yeah, I'd ping him before spending a whole bunch of time debugging cloud init15:21
babbageclunkcherylj: ok, will do - thanks15:21
cheryljnp :)15:21
mgzhe's around today, I have spoken to him... but I have some doubts it's actually cloud-init related15:21
cheryljyeah, but he could give some hints about where to look if it's not15:21
jammgz: hi. I wanted to check if mgo is one of the -dev packages that we pull from the archive instead of bundling our own.15:22
jamas we have an important fix for the next release, but does that mean we need to get a mgo release, and into xenial backports/updates before we can build juju with it?15:22
mgzjam: it is not15:23
jammgz: good to hear15:24
mgzhm, odd, it was on the list initially15:25
babbageclunkmgz: do you have a theory about that bug?15:27
perrito666hi all btw15:27
perrito666frankban|afk: please ping me when you are back15:30
mgzbabbageclunk: not really I'm afraid15:30
natefinchperrito666: howdy15:30
mgzballoons: ^do you remember why we don't use the mgo package in xenial for juju?15:30
babbageclunkmgz: darn. Why don't you think it's cloud-init related?15:31
balloonsmgz, golang-gopkg-mgo.v2-dev is on the list. We couldn't use it for xenial because it's not in main15:32
balloonsthere's a MIR bug that's not finished for it15:33
mgzI'd expect useful stuff in cloud-init.log if there was really a problem starting the init step, given init-local succeeds for both15:33
balloonshttps://bugs.launchpad.net/ubuntu/+source/golang-gopkg-mgo.v2/+bug/156816215:33
mupBug #1568162: [MIR] golang-gopkg-mgo.v2 <golang-gopkg-mgo.v2 (Ubuntu):Incomplete> <https://launchpad.net/bugs/1568162>15:33
mgzjam: ^so, that's the answer, package not in main. your problem is fine for now then, though it may be polite to file an sru bug for the mgo update if it's likely to affect other users.... if there are other users.15:34
frobwarealexisb__: one thing I meant to mention is that the MAAS folks are having a sprint in bluefin in august - it might be worth tagging alone for a couple of days to talk about IPv615:34
natefinchjam, mgz: it shouldn't matter what's in main for most people, should it?  Since simple streams uses our own built version.  The only people using what's actually in ubuntu would be people who use upload-tools.15:40
mgznatefinch: in this case it's just a distro policy thing you really don't need to care about15:41
natefinchmgz: right, I just wanted to make sure I was on the right page, in that, for the most part, we (thankfully) don't need to care about what ubuntu does with their wacky go packaging15:42
mgzand yeah, we already have some bugs that are not fixed if people use --upload-tools15:42
babbageclunkmgz, natefinch - I'm not following this. What does it mean for mgo to be deployed from a package? When we package juju do we build it so that it's not statically linked?15:46
mgzbabbageclunk: normally when you bootstrap juju, it goes and fetches the juju binary from streams.canonical.com15:46
mgzwe build those with exactly what's in dependencies.tsv15:46
mgznot what's in the archive15:47
babbageclunkok15:47
jambabbageclunk: but the binaries you get from "apt-get install juju" (which gives you a jujud) uses ubuntu '-dev' packages.15:47
jamso we get the worst of all worlds as near as I can tell.15:47
natefinchbabbageclunk: what we deploy to streams is a different binary with different behavior than what they package in ubuntu.  We use code that matches the versions in dependencies.tsv.  They use.... some other versions.15:47
babbageclunkjam - so the apt-gotten jujud dynamically loads mgo?15:49
alexisb__frobware, agreed15:49
natefinchbabbageclunk: no, it's just built statically with a different version15:49
natefinchbabbageclunk: ubuntu wants all go applications delivered with ubuntu to use the same version of mgo.  So they pick one and compile everything with that one.15:49
babbageclunknatefinch: ah. Ok, that was the bit I missed.15:50
babbageclunknatefinch: that seems quite restrictive.15:51
jamnatefinch: babbageclunk: so according to mgz the 'mgo' library itself is not one of those, but there are about 10 dependencies that do work that way15:54
jamgocrypto being one of them, I believe.15:54
cheryljredir: sorry to add more churn in your delete^H^H^H^H^H^H remove-user command ;)15:57
babbageclunkjam, natefinch: So this only affects people using --upload-tools? Other bootstraps get a good jujud from the stream and the weird jujud on their local machine doesn't really matter?15:58
mgzbabbageclunk: yes. ideally we don't include jujud in the archive at all.15:59
jambabbageclunk: so, I'm pretty uncomfortable having a jujud out there that isn't the real jujud and both of them claim exactly the same version, but AIUI that is true15:59
mgzbut I got yelled at when I tried to sneak it out :)16:00
cheryljheh16:00
babbageclunkjam: oh, I can see why it's gross, just wanted to make sure I understood the implicatons.16:01
babbageclunkmgz :)16:01
babbageclunkthanks for the explanations everyone!16:02
perrito666rogpeppe: oh your mail brightened my day... not :p16:06
rogpeppeperrito666: sorry :)16:06
perrito666rogpeppe: I assumed some things might break that where working a bit accidentally because of the previous implementation of users :) I am a bit saddened that no test at all blew with this16:07
perrito666nah, that is not true, I am actually quite pissed :)16:08
rogpeppeperrito666: the problem was that there were no juju command tests for this16:08
perrito666rogpeppe: exactly16:08
rogpeppeperrito666: which was actually slightly deliberate, unfortunately16:08
perrito666rogpeppe: care to elaborate a bit on that?16:09
dooferladcherylj: I tracked down the cause of bug #1599779 - every log forwarding call opens the API server, which starts some workers, then logs, the workers are stopped and the process repeats. The log sender needs to keep the API open on the client side to avoid this.16:23
mupBug #1599779: Very high CPU usage from 'creating cloud image metadata storage'  emitted in debug log every second <juju-core:Triaged> <https://launchpad.net/bugs/1599779>16:23
dooferladcherylj: don't know who to assign it to, but I know I need to go cook dinner soon :-|16:23
dooferladcherylj: happy to take it myself if nobody is free who worked on it16:24
dooferladok, so the server side needs to keep things open. Oh, I need a fresh brain. Will make more notes in the bug16:26
katcodooferlad: i think wallyworld was working on this16:30
mupBug #1600302 changed: security groups could not be destroyed when an Openstack provider bootstrap was interrupted with ctrl-c <cdo-qa> <juju-core:New> <https://launchpad.net/bugs/1600302>16:39
cheryljdooferlad: thanks for digging into it.  I'll bring it up in the release call and see if we can get someone on it16:44
aisraelI've seen mongod/jujud use a ton of cpu when I only have a single controller and one model with nothing deployed. `juju debug-log` shows nothing. How can I capture info out of mongo/juju in order to debug/provide feedback?17:02
natefinchaisrael: you probably need to up the log level to show more... the default is pretty light on logging.  juju set-model-config logging-config="<root>=DEBUG" will help show more of what's going on.17:10
natefinchaisrael: you will probably want to do that both for the default model and the controller model17:11
aisraelnatefinch: Excellent, thanks. I'll work on repeating the behavior and try to get something useful out of it.17:12
natefinchaisrael: cool.  you can also do --config  logging-config="<root>=DEBUG" at bootstrap time to set the log level immediately (I'm not 100% sure how that interacts with multiple models, though)17:13
aisraelnatefinch: good to know!17:13
aisraelnatefinch: when I do that, where will I find that log output?17:16
aisraelnatefinch: nevermind, I found it. Switch to controller model and look in /var/log/juju17:16
natefinchaisrael: the usual debug-log... however most likely anything interesting will be in the debug-log of the controller model (debug-log is per-model now)17:16
natefinchaisrael: yep17:17
aisraeldebug-log wasn't showing me anything in the controller model but I have a ton of data now. I'll let it log over lunch and poke the logs when I'm back.17:17
natefinchaisrael: cool, good luck.  I hope you can figure it out.17:18
cmarshow do i pprof a controller?17:22
cmarsi have a beta11 controller that's going nuts17:22
natefinchcmars, aisrael: I think there's a bug that is causing this - https://launchpad.net/bugs/159977917:23
mupBug #1599779: Very high CPU usage from 'creating cloud image metadata storage'  emitted in debug log every second <juju-core:Triaged> <https://launchpad.net/bugs/1599779>17:23
cmarsnatefinch, thanks, checking my log17:24
cmarsnatefinch, its a bunch of 'log forwarding not supported' messages17:25
cmarsnatefinch, same or different issue, should I open a bug?17:26
natefinchcmars: I think it's worth making it a new bug even if we eventually decide it's a duplicate, just to make it easier for others to find17:27
katcocmars: there is already a bug open for that17:27
katcocmars: i believe it's this one: bug 159811817:28
mupBug #1598118: log-forwarder worker bounces endlessly when forwarding is not configured <2.0> <debug-log> <log-forwarding> <logging> <juju-core:Fix Committed by ericsnowcurrently> <https://launchpad.net/bugs/1598118>17:28
cmarsnatefinch, thanks, i feel less guilty for opening dups :)17:28
cmarskatco, thanks, that's the one17:29
natefinchcmars: a duplicate bug is far better than a missing bug17:29
natefinchafk a while17:32
=== natefinch is now known as natefinch-afk
cheryljkatco: could you review that deployer fix we chatted about last week?  http://reviews.vapour.ws/r/5227/18:39
katcocherylj: sure18:39
cheryljty!18:39
katcocherylj: you have a review18:55
cheryljthanks!18:55
cheryljkatco: for the testing comment - I did it asynchronously because if the deployer worker doesn't fail to deploy the unit, Wait() won't return19:03
cherylj(since the loop would happily keep waiting for the next event)19:03
katcocherylj: is there any way to set up the test to always fail?19:04
cheryljkatco: yeah,  since I've embedded the interface and the next thing the deployer would do is call DeployUnit, it will panic if I don't have that defined (which I don't)19:04
katcocherylj: so, you can remove the goroutine?19:05
cheryljI could.19:05
katcocherylj: yay? do you disagree that removing it is good?19:06
cheryljIt just felt nicer19:06
cheryljto me at least19:06
katcocherylj: it has a race condition though19:06
cheryljfair point19:06
=== natefinch-afk is now known as natefinch
cheryljkatco: updated!  http://reviews.vapour.ws/r/5227/19:13
katcocherylj: k tal19:13
cheryljblergh, it didn't save my updates to the description19:14
katcocherylj: doh... also the checklist calls for testing methodology19:14
cheryljkatco: I ran it through CI19:16
cheryljCan't recreate manually19:16
katcocherylj: ah that's right19:16
katcocherylj: lgtm19:16
perrito666could anyone make a quick review to http://reviews.vapour.ws/r/5228/ ?19:16
cheryljthanks!19:16
katcocherylj: ty19:16
natefinchperrito666: I can look19:17
perrito666natefinch: its literally 2 lines19:17
perrito666just growing a table, ish19:17
natefinchperrito666: ship it19:19
perrito666tx a lot19:19
natefinchug, juju bootstrap --clouds is a terrible idea19:52
natefinchit makes juju bootstrap do two totally different things19:53
perrito666natefinch: what does it do?19:57
natefinchperrito666: it's basically juju list-clouds combined with juju list-credentials19:57
natefinchperrito666: I don't really understand why it exists, and especially not why it is a flag on bootstrap19:58
natefinchperrito666: http://pastebin.ubuntu.com/19206808/20:01
perrito666natefinch: sounds like a very specific --help20:02
natefinchperrito666: yeah20:05
mupBug #1602416 opened: Failure to perform action due to tcp i/o timeout <juju-core:New> <https://launchpad.net/bugs/1602416>20:55
mupBug #1602416 changed: Failure to perform action due to tcp i/o timeout <juju-core:New> <https://launchpad.net/bugs/1602416>21:04
mupBug #1602416 opened: Failure to perform action due to tcp i/o timeout <juju-core:New> <https://launchpad.net/bugs/1602416>21:10
wallyworldniedbalski_: we're running late, still in another meeting, be there soon22:00
thumperalexisb: is there the sts cross team?22:13
alexisbthumper, yes22:13
perrito666k leaving for a moment, wallyworld ill be back for standup22:19
wallyworldok22:20
niedbalski_cherylj, https://bugs.launchpad.net/juju-core/+bug/160205422:34
mupBug #1602054: juju deployed lxd containers are missing a default gateway when configured with multiple interfaces <juju-core:New> <https://launchpad.net/bugs/1602054>22:34
alexisbthumper, did you want to take a coupld of minites and catch up before my eod??22:49
thumperalexisb: sure22:49
alexisbhttps://hangouts.google.com/hangouts/_/canonical.com/alexis-tim22:50
perrito666redir: alexisb https://en.wikipedia.org/wiki/Teletext23:37
wallyworldmenn0: would love a quick chat at some point today, when you have 5 minutes free, can be any time23:48
wallyworldveebers: that latest log forward branch finally got landed. the new config attribute is logforward-enabled (false by default)23:49
menn0wallyworld: sure. i'm close to proposing a fix for this debug-log issue. let's talk once i've done that.23:50
wallyworldno hurry at all23:51
veeberswallyworld: sweet23:51
wallyworldveebers: i am going to look at that CPU issue again - i couldn't repro it, but there's something amiss it seems. apart from that, everything should be functional23:52
veeberswallyworld: cool. I'll see if I can repro it again with a latest build23:54
veeberswallyworld: did those instructions came in handy yesterday?23:54
wallyworldveebers: i ended up hacking something else together for expediency (used a file sink not a syslog sink) - that was all i needed to test the bits i was interested in23:55
wallyworldbut i will probs do the full set up at some point23:55
veebersah nice, wasn't aware of a file sink :-\ That may have been easier23:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!