/srv/irclogs.ubuntu.com/2017/03/17/#juju.txt

ybaumyplease make remove-unit really work05:44
ybaumyalso does set-model-constraints tags=test work for all machines und unit deployments?05:49
ybaumythe problem im having is that constraints do not work when add-unit is used05:50
ybaumyso to scale out there is no way to use a specific machine set or is there?05:51
ybaumythis really a showstopper if i cant scale out to machine sets who share the same tags05:55
pranav_Hi Folks. I am facing issue in publishing our charm for review06:51
pranav_charm list resources shows06:52
pranav_[Service] RESOURCE REVISION install  -106:52
pranav_but running "charm release cs:~vtas-hyperscale-ci/hyperscale-0 --resource install-1 --channel edge" Gives error06:52
pranav_ERROR cannot release charm or bundle: bad request: charm published with incorrect resources: cs:~vtas-hyperscale-ci/hyperscale-0 resource "install/1" not found06:53
kjackalGood morning Juju world!08:25
=== saber is now known as Guest50267
Guest50267Hey all. Any one has idea on charm push-term?08:53
Guest50267getting error ERROR unrecognized command: charm push-term08:53
Guest50267charm version is 2.2.0-0ubuntu1~ubuntu16.08:54
tejaswihi team, I am trying to install all OpenStack services using Juju. However, due to this bug - https://bugs.launchpad.net/charms/+bug/1417407, the haproxy cfg files are getting overridden by new services.09:00
mupBug #1417407: haproxy enabled by default for keystone, cinder, openstack-dashboard, nova-cloud-controller and glance breaks deploying multiple light services to the same node. <Juju Charms Collection:New> <https://launchpad.net/bugs/1417407>09:00
tejaswiHow to fix this issue ?09:01
tejaswiIs there a way to install OpenStack without haproxy ?09:01
ybaumyhmm09:10
ybaumy  '3':09:10
ybaumy    series: xenial09:10
ybaumy    contraints: tags=devcloud,node09:10
ybaumyah forget it09:10
ybaumyi should learn how to write constraints09:11
cnfybaumy: what is wrong with it?09:25
ybaumyi wrote constraints instead of constraints09:26
ybaumy-s09:26
ybaumyin the first09:26
cnfah :P09:26
cnfdem morning typos09:26
ybaumyim up since 5 so its forgiven09:27
ybaumycnf have you ever had the problem to add a unit with constraints?09:28
cnfuhm, i learned of constraints yesterday09:28
cnfi have yet to successfully use them09:28
ybaumyi wonder if i have to add-machine first then add-unit --to09:29
cnfask me again in 20 minutes, i'll tell you if my srtuff came up :P09:29
kjackaltejaswi: You could ask at #openstack-charms10:14
kjackalybaumy: you do not need to first add a machine and then deploy the app. You just do a juju deploy .... --constraints "mem=4G ...."10:15
ybaumykjackal: no its about scale out with add-unit10:16
ybaumyhow do i make constraints inherited10:17
ybaumyif that is the correct term10:17
tejaswikjackal: ok10:21
magicaltroutneed a hack, how do i manually unset a state?14:23
magicaltroutlazyPower: ping14:32
lazyPowermagicaltrout: charms.reactive remove_state14:35
magicaltroutI actually had another question, but that will hopefully help resolve it14:35
magicaltroutever seen this14:35
magicaltrout2017-03-17 14:30:26 INFO install Generating RSA private key, 2048 bit long modulus14:36
magicaltrout2017-03-17 14:30:27 INFO install unable to write 'random state'14:36
magicaltrout2017-03-17 14:30:27 INFO install e is 65537 (0x10001)14:36
lazyPowerO_o14:36
lazyPowerunable to write random state?14:36
magicaltroutthe weird thing is if i actually run the command it runs fine14:36
lazyPowerthats a new one on me14:36
lazyPowermarcoceppi: ^ have you seen this?14:36
lazyPoweryou've seen like every nit noid thing we've come up with14:36
magicaltroutits in your code, thats why i'm asking :P14:37
* lazyPower crosses fingers14:37
lazyPowermagicaltrout: well lets not point fingers14:37
lazyPowerwhat code?14:37
magicaltrouthehe, k8s master14:37
magicaltroutbut this is on my weird openstack deployment platform, so i'm not blaming you, its probably entropy rubbish14:37
lazyPoweryeah thats what i was about to say is that it sounds like the unit hasn't generated enough entropy14:38
magicaltroutbut my openstack cluster and canonical k8s really hate each other :)14:38
lazyPowerand we host entropy.ubuntu.com14:38
lazyPowerwhich should have given you enough of a random seed for entropy14:38
magicaltroutI'm gonna remove authentication.setup14:38
magicaltroutin the hope a rerun will kick it enough14:38
magicaltroutcause it just stops, but weirdly, i wouldn't have thought it got set14:39
magicaltroutso I don't know why it isn't run a second time14:39
magicaltroutbombed again14:41
magicaltroutwhat the14:41
magicaltroutweird and interesting14:42
marcoceppimagicaltrout: not enough /dev/random bits?14:43
magicaltrouthttp://pastebin.com/5WUhAr7Y14:44
magicaltroutwell14:44
magicaltroutif I run the hook14:44
magicaltroutit fails14:44
magicaltroutif I run the command via juju run14:44
magicaltroutit fails14:44
magicaltroutif I ssh into the box and run it manually14:44
magicaltroutit runs14:44
* magicaltrout gets out the big # based crowbar14:45
marcoceppimagicaltrout: is there a .rnd file in /root ?14:47
magicaltroutyup14:47
magicaltroutsorry lazyPower not trying to grumble, just trying to figure out juju/k8s weirdness on a random openstack cluster I have zero control over but have been asked to deploy kubernetes to14:56
magicaltrout:)14:56
magicaltroutanyway, i've commented it out, we'll see how we go14:56
lazyPowermagicaltrout: oh not even upset man, just trying to keep my head above water atm. I'm sick, heavily medicated, and in/out of meetings for the last hour.15:10
magicaltroutjoin the club, my nose was streaming on the way back from NYC15:11
magicaltrouti'm not sure my fellow passengers were overly impressed15:11
lazyPoweryeah, must be going around.15:12
* lazyPower reads more of the backscroll15:12
lazyPowermagicaltrout: this does seem like an issue with entropy15:12
lazyPoweri'm not sure why its working when you manually run it vs script execution though.15:13
magicaltroutyeah its a bit odd15:13
lazyPowerperhaps the fact you've logged in and issued the command gave it the bump it needed in terms of entropy15:13
lazyPowerthis is outside my scope of knowledge in crypto15:13
magicaltroutbut also fails in juj run as well15:13
magicaltroutso its happy when you have a shell open15:13
magicaltroutoh well15:13
lazyPoweri would probably be poking dustin about this and asking him to teach me to fish, or links so i can read about fishing.15:13
magicaltroutthis is possibly the slowest openstack cluster the US Govt could possibly own15:14
lazyPowerpfft15:14
lazyPoweryou had to challenge them didnt you?15:14
lazyPoweryour next stack will be a bunch of minnowboards15:14
magicaltroutwell i got more ram15:14
magicaltroutbut apt update takes ~25 minutes per server :)15:14
magicaltroutI think i might be a bit IO bound15:14
jrwrenmagicaltrout: welcome to my slow-server case. There are some apt tunings you can do.15:15
Budgie^Smoreo/ juju world!15:15
Budgie^Smorehope your feeling better today lazyPower15:15
jrwrenmagicaltrout: I've always felt that charms should be optimized to run apt-update minimally, but instead they run it rather often. It really makes for slow charm deploys when things are on slow IO :(15:15
lazyPowerBudgie^Smore: i'm mostly dead inside because of all the medication, but i'm present and accounted for.15:15
magicaltrouttell me about it jrwren, slow or completely time out15:16
* rick_h ships lazyPower some OJ15:16
lazyPowerBudgie^Smore: how are things giong for you and your maas k8s deploy?15:16
magicaltrouti spin up units manually and run apt update befoe dumping juju workloads on them15:16
lazyPowerrick_h: <315:16
lazyPowermagicaltrout: are you using the daily image stream?15:16
Budgie^SmorelazyPower oh I keep hitting hurdle after hurdle there and all of them of my own making probably15:17
lazyPowermagicaltrout: if you're using daily images your deltas should be pretty minimal15:17
magicaltroutthere aren't that many lazyPower, but it takes forever installing a new kernel15:17
lazyPowermagicaltrout: i know if you're using the "stable" image stream you could be somewhere near 3 months of package updates, depending on patch release and which image.15:17
Budgie^SmorelazyPower the latest one is having me rethink using VBox in favor of kvm due to not be able to change the power types in MaaS 2.115:18
jrwrenmagicaltrout: do you find that dumping juju workloads on them is also slow because the juju workload apt update / upgrade as well?15:18
magicaltroutna thats not as bad jrwren cause the delta is 0 by this time15:19
Budgie^SmorelazyPower still need to figure out how to solve the cert issue on my clound k8s cluster too15:19
jrwrenmagicaltrout: ah, well that is good.15:19
lazyPowerBudgie^Smore: - cert issue? i think i've missed context here... sounds like you've told me about this one and i forgo about it....15:19
lazyPowerBudgie^Smore: do you mind re-iterating and i can try to help?15:19
Budgie^SmorelazyPower but that one is getting some larger instances as dev created a tool that will swallow memory like it is going out of fashion15:20
lazyPowerhah15:20
magicaltroutI spent about 6 hours on tuesday trying to get the k8s routing working and it kept failing, i suspect because the update times were so rubbish juju failed or hadn't updated some services15:20
lazyPoweris it java based? :D15:20
magicaltroutI have more ram now, just slow disks, so hopefully I'll get further :)15:20
lazyPowermagicaltrout: k8s routing... meaning ingress?15:21
magicaltroutyeah, and dashboard15:21
magicaltroutthey all failed to route out15:21
Budgie^SmorelazyPower sure, give me a few mins to pull the acctual error but basically it looks like one of the tools is complaining about an unknown CA which is causing it not to display logs either in the dash board or from kubectl logs15:21
lazyPowerok, you do know that we dont enable external dashboard access outside of proxy by default right?15:21
lazyPoweryou can easily default that, but we dont make it easy because its a huge security hole15:21
magicaltroutyeah i run the kubctrl proxy15:21
magicaltroutjust get 504's15:21
lazyPowerBudgie^Smore: ok, that sounds like 1 of 2 things. I"ll wait for the error before i diagnose15:22
magicaltroutanyway, i'm not there yet today, when I am, we'll see :)15:22
lazyPowermagicaltrout: ok, when you do get there, make sure you have my attention and i'll help15:22
lazyPowerwe have a bug open from another user that ran into unexpected behavior15:22
lazyPowerstuff that i haven't been able to reproduce15:22
magicaltrouti mean i've only been spinning up openstack nodes for it for like 3 hours! :P15:22
lazyPowerso i'm highly interested in findign out wtf is going on15:22
magicaltroutyeah i saw that ticket15:22
magicaltroutit looked similar15:22
lazyPoweryeah man, if its the same, lets find it and nail it to the wall15:22
lazyPoweri dont like bugs i cant reproduce15:22
Budgie^SmoreI have one of those with the MaaS team right now lazyPower ;-)15:26
Budgie^Smoreoh I just noticed that I am using IPv6 at home!15:32
lazyPowerBudgie^Smore: you mean a non-reproduceable bug by the maintainers?15:38
lazyPowerBudgie^Smore: kill it with fire15:38
Budgie^SmorelazyPower yeah, install related issue where the rack controller doesn't register to the region controller15:39
lazyPower:S15:39
lazyPowerfun times, that one15:39
lazyPowerBudgie^Smore: i'm guessing nothing in the logs of the region controller? Nothing indicative that the network is not the culprit?15:40
Budgie^SmorelazyPower I saved a copy of the virtual disk one of the times it happened but haven't heard from the maintainers if they would like it... there is an error in there but didn't point to the culprit, I suspect network latency issues though15:41
stormmoreok so lazyPower, here is the output from kubectl --v=8 logs - http://paste.ubuntu.com/24195846/15:47
lazyPowerok, i see the x509 errors in here. This is consistent with a couple problems that might have arisen...15:49
stormmorelazyPower, it is worth noting that I only started getting this error once I upgrade my kubectl client version to 1.5.3, with 1.5.1 it was an annoying generic catch all failure15:50
lazyPower1 sec, let me finish gettign this multi-cloud test run of the etcd3 stuff going. Can you get me the output of juju run-action kubernetes-master/0 debug15:50
lazyPowerpost that somewhere secure, it'll have logs + config files that you dont want to leak15:50
lazyPowerstormmore: that sounds like some of the extra debugging that landed in 1.5.2, so it makes sense that was expanded when you upgraded.15:51
lazyPowerbut i'll want the output of the debug action on your master so i can verify the tls components were installed correctly15:51
stormmoreI am wondering if this is an affect from upgrading the cluster15:52
stormmore(and messing it up when I just did a juju deploy over the top of the current one)15:53
stormmorelazyPower, OK to DDC it to you?15:54
arosaleshatch: I added a comment to https://github.com/juju/docs/issues/165115:56
arosaleshatch: I also opened up https://github.com/juju/docs/issues/172315:56
hatchkewl thanks15:56
lazyPowerstormmore: hang on and i'll see about deploying a secure upload15:58
lazyPoweri'm on znc + weechat and haven't used DCC since the 90's15:58
lazyPowerthat combination means i'm probably going to have tears at the end of it :P15:59
stormmoreright! yeah it has been awhile for me too... not even sure how well Hexchat handles it15:59
magicaltroutDCC! jesus, i'd forgotten that was a thing16:02
magicaltroutthose were the days16:02
magicaltroutit didn't work then16:02
magicaltroutit won't work now :)16:02
psarwateHey folks. I am unable to get juju terms to work with my charm.16:05
psarwateEventhough the charm is not agreed upon, juju still doesn't prompt me16:06
psarwatethe terms is not agreed upon*16:06
stormmoreOK I have the debug log :) reminds me of sosreport back in the day16:13
magicaltroutI figured out my dashboard faux pas lazyPower , no UDP routes between the nodes16:22
magicaltroutoops :)16:22
arosaleshatch: one more https://github.com/juju/docs/issues/172416:30
lazyPowermagicaltrout: ah interesting16:30
arosaleshmm seems I missed psarwate16:31
arosalesI wonder if Juju was actually deploying his charm though . .  .16:31
hatchthanks arosales16:33
arosalesalso if psarwate was deploying locally then terms will be bypassed16:34
lazyPowercs:~containers/etcd-25  (edge channel) was just released, introducing deb=>snap migration path and roll forwards to etcd 3.1.3  (juju config etcd channel=3.1/stable)  -- you'll need to follow the migration path in the readme however. All early feedback appreciated17:39
lazyPowerplus sesnding a notice to the list with a bit more details than this blurb ^17:42
rick_hlazyPower: congrats!17:43
lazyPowerrick_h: its pretty cool to see the capability of rolling forward and rolling backwords using a single config option. (potential data mismatch if you're actively reciving data during the forward jump, rollback will restore to the state just prior to the upgrade). but its nice to know we support this now in the event of troubled clusters.17:44
rick_hlazyPower: yea, cool how things are moving17:44
=== hml_ is now known as hml
=== hml_ is now known as hml
petevgcory_fu: you did some work to making rebuilding the python-libjuju _client easier, right?20:19
stormmorebtw lazyPower I am looking at using SDN in this cluster20:26
cory_fupetevg: Yes.  Are you planning to rebuild it?20:47
petevgcory_fu: yes. Though we may have to tackle the multiple versions thing sooner rather than later.20:48
petevgcory_fu: filing a bug and pushing a repro right now ...20:48
petevgcory_fu: issue at https://github.com/juju/python-libjuju/issues/90, and test reproducing it at https://github.com/juju/python-libjuju/pull/9120:50
petevgcory_fu: do the docs on building live anywhere?20:50
cory_fupetevg: Yes, but my Makefile changes seem to have disappeared20:51
petevgcory_fu: that's why I couldn't find them :-/20:51
cory_fuwth20:53
petevgcory_fu: git ate your homework!20:56
cory_fupetevg: Well, I can't for the life of me figure out what happened to the Makefile changes I made, but they're not strictly necessary and were relatively minor.  The important bit is at https://github.com/juju/schemagen20:59
cory_fupetevg: Build and run that, directing the output to juju/client/schemas.json, and then run the "client" target: https://github.com/juju/python-libjuju/blob/master/Makefile#L1321:00
petevgcory_fu: cool. thx.21:00
petevgcory_fu: since the alpha has breaking changes, this is also the time to worry about different versions of the _client.py in python-libjuju, right? As in: I shouldn't just slam these changes in ...21:01
cory_fupetevg: Well, it's just one API, and we could work around it in the wrapper around FullStatus that we definitely should have.  But we do need to support facade versions eventually so if you can come up with a good way to handle that, I'm down21:03
magicaltroutso lazyPower if I run the microbot service and then want to check it21:04
petevgcory_fu: cool. That sounds more fun than downgrading juju and having to deal with halfway destroyed models cluttering my env :-)21:04
magicaltroutdo I have to munge my hosts file to make it resolve?21:04
cory_fupetevg: It would be good to have _client.py split up anyway, and splitting it by facade version shouldn't be much harder.21:04
cory_fupetevg: The hard bit would be figuring out which facade versions to use based on the controller.  I don't know if there's a way to query that21:05
cory_fupetevg: Also, I would imagine that the object layer code would need to know how to work with the different facade versions as well.21:05
petevgAyup.21:05
cory_fupetevg: Aren't the facades supposed to keep us from having version-specific logic in the code?  I'm a little unclear on how they're supposed to help21:07
* magicaltrout is getting 502's and nothing else \o/21:08
petevgcory_fu: My expectation would be that if you do nothing, you are operating against stable, and if you pass in a flag somewhere, you operate against some other version.21:08
petevgcory_fu: it would be cool to do some magic with imports, but I don't think that import time is the right time.21:08
petevgconnection time probably isn't right, either.21:08
cory_fupetevg: Maybe tvansteenburgh has some insight for us here?21:10
* tvansteenburgh wakes up21:12
* tvansteenburgh reads scrollback21:13
tvansteenburghpetevg: cory_fu: https://github.com/juju/python-libjuju/issues/49. we can dynamically use the right facade version pretty easily. the trick will be handling changing api signatures in the object layer21:15
tvansteenburghhappy to discuss that at some point, but that point is not today :)21:16
magicaltroutokay... lazyPower my microbot containers are all paused... no idea why or what that is, answers on a postcard when you get bored21:16
petevgtvansteenburgh: Cool. I'm going to head out to dinner soon, anyway. Thank you for the linky :-)21:16
tvansteenburghsure thing21:16
lazyPowermagicaltrout Budgie^Smore - i think i foudn the issue with your x509 validations - https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/23821:29
lazyPowermagicaltrout Budgie^Smore - if you could check your /etc/default/kubelet file's for the presence of the tls flags + disabling anonymous auth on kubelet, that would be choice to validate its the root cause of the validation issues post 1.5.3 update21:31
magicaltrouti don't have cert issues :)21:33
magicaltroutnot yet anyway21:33
magicaltrouthttp://pastebin.com/raw/gSWGEk1E21:34
magicaltroutany idea lazyPower \o/ :)21:34
magicaltrouteven docker run -d dontrebootme/microbot:v121:34
magicaltroutis failing on me21:34
magicaltroutwhich is weird cause I have other containers running fine21:34
lazyPowerwhats this about ipv621:35
lazyPowerand it cant seem to find any of the image layers21:36
lazyPoweri'm not sure how you got into this state but this is certainly fun magicaltrout21:36
magicaltroutlol21:36
magicaltroutthis is just post install21:36
lazyPowernot without some effort it isnt21:36
lazyPowerhow do you get docker to know about the meta of the image but have none of the data to back it up?21:37
magicaltrouti swear gov'ner over than installing the charms i've not touched it from base xenial21:37
lazyPoweryeah i'm not sure what to recommend here magicaltrout21:40
lazyPoweri've not seen this before. I suspect i'm missing pieces to the puzzle21:40
lazyPowermagicaltrout best i can suggest at this juncture would be to capture the model with a juju-crashdump report and post that for post analysis21:40
magicaltroutreboot managed to make things worse \o/21:44
magicaltroutho ho, nothing better on a friday night21:44
stormmoreoh nice lazyPower! let me look, sorry I have taken today as a chance to write some documentation21:45
stormmoreyou still there lazyPower? http://paste.ubuntu.com/24197670/ is the /etc/default/kubelet from one of the workers21:48
lazyPowerstormmore yeah looks like the tls flags + auth flags didn't make it in the upgrade21:50
lazyPowerCynerva not sure if you're still here, its pretty late in the day. ^21:50
lazyPoweri filed https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/238 in reference to this.21:50
lazyPowerstormmore thanks for validating, we should have a patch or working fix for this soon21:51
lazyPowerthe manual fix is pretty simple too, but i'd rather the charms take care of themselves and update that defaults file.21:51
stormmoreso the "workaround" would be to add those flags?21:51
lazyPoweryeah21:51
stormmoreyeah no problem, I think I have a bought a little bit of time on that anyway. need to add 3 larger instances and remove 321:52
magicaltroutlazyPower: the ipv6 stuff is cause flannel has an ipv6 address and dishes them out to the container eth adapters21:57
magicaltrouti removes /var/lib/docker which seems to have resolved the layer issues21:58
magicaltroutwhat is amazing though is21:58
magicaltroutnginx-ingress-controller runs fine on the same machine?!21:59
* magicaltrout needs hard liquor21:59
petevgcory_fu: my first (probably naive) attempt at "make client" failed with this error: https://pastebin.canonical.com/182956/  (This is using a schema.json that I generated with schemagen).22:02
petevgcory_fu: it's dinner time, though, so I'm going to do that, and then worry about things more on Monday.22:02
petevgHave a happy weekend!22:02
magicaltroutoooh fsck22:02
magicaltroutyou can't docker exec into microbot22:02
magicaltroutthat explains that then22:03
cory_fupetevg: Damnit!  That was the error I fixed in the same code that I modified the Makefile target22:03
cory_fupetevg: I really don't want to have to re-do that.  It was a bit of a PITA22:03
petevgcory_fu: darn!  I will try doing some git Magic on Monday. Maybe I can fin your change ...22:04
petevg*find22:04
magicaltroutoooh blimey microbot is running22:06
magicaltrouti take it all back lazyPower22:06
magicaltroutits an amazing platform22:06
magicaltroutnever questioned it!22:06
magicaltroutlazyPower: other question, trying to understand ingress stuff.  Is the microbot supposed to route through the loadbalancer?22:10
magicaltroutor mbruzek you're on the hook as well :P22:11
magicaltroutooh the loadbalancer is only for the masters?22:24
mbruzekmagicaltrout: no microbot is not going through the kubeapi-load-balancer, it goes through a kubernetes load balancer22:42
mbruzekwhen you create ingress rules22:42
cory_fupetevg: ah ha!  It's in the branch bug/77-deploy-resources!  You can easily see the changes I made in this commit: https://github.com/juju/python-libjuju/commit/ef59035b2596a1998615cb0ad3f73fe53953189822:49
mupBug #77: Maybe drop you in the 'edit bug' page after adding a bug? <lp-bugs> <Launchpad itself:Invalid by bjornt> <https://launchpad.net/bugs/77>22:49
petevgcory_fu: yay! Thx.22:49
cory_fupetevg: If you want, I could submit that commit by itself as a PR22:50
cory_fuProbably worth doing22:50
petevgcory_fu: sure. Thx.22:51
cory_fupetevg: https://github.com/juju/python-libjuju/pull/9222:53
stormmorelazyPower, you don't have any toys to help devs assess the resource usage - cpu, mem - of their containers do you?23:02

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!