/srv/irclogs.ubuntu.com/2019/10/08/#juju.txt

bdxyeah, check out the osm bundle it uses `bundle: kubernetes` and doesn't get a kubernetes tag in the charmstore https://jaas.ai/osm00:49
bdxand here, my bundle uses it incorrectly (non-deployable bundle) and get the kubernetes tag https://jaas.ai/u/omnivector/slurm-core-k8s/bundle/300:50
bdxhttps://github.com/juju/charmstore/issues/88700:55
babbageclunkwallyworld / hpidcock / kelvinliu could I get a review of this? https://github.com/juju/juju/pull/10692 (model-defaults test fix)01:09
babbageclunkalso this one is some assess tweaks https://github.com/juju/juju/pull/1069301:10
wallyworldok01:11
wallyworldbabbageclunk: lgtm, thanks for the fix01:14
babbageclunkthanks!01:15
babbageclunkdamn - meant to put them against 2.601:15
babbageclunkhang on, cancelling the merge on the test fix so I can retarget01:16
wallyworldbabbageclunk: we ain't gonna do any more 2.6 (touch wood) to don't feel like it's a requirement to land there firxt01:18
babbageclunkI just figured may as well fix it in both vaguely current branches01:23
thumperthread logger through apicaller worker: https://github.com/juju/juju/pull/1069402:15
thumperhpidcock: did you want to talk about bug 1847084?02:19
mupBug #1847084: Juju k8s controller is not getting configuration parameters correctly <juju:Incomplete> <https://launchpad.net/bugs/1847084>02:19
thumperthere is history there02:19
thumperhpidcock: nm, replied to the bug02:22
thumperhpidcock: the issue is --config vs. --model-defaults, they were setting proxies in the config, adding a model then wondering why the proxies weren't there02:22
hpidcockI already talked with wallyworld about it. I just wanted to gather a bit more information about what we should test it on with a fix. Since there is like 20 different CNI plugins for kubernetes that could affect how this works02:22
thumperthis isn't a network issue02:23
thumperthis is a model config issue02:23
thumperhmm...02:23
* thumper is rereading the bug...02:23
hpidcockthe model config is fine, it's not passing the proxy env vars to the controller pod02:24
hpidcockbut I wanted to understand the environment they are expecting this to work on so I can test it. Because the proxy configuration especially the no-proxy ranges could affect the in cluster/container networking02:25
thumperok02:26
thumperI think perhaps there is an issue with them setting no-proxy not juju-no-proxy02:26
thumperhpidcock: I guess I'm not sure where they expect the proxy to be set02:27
thumpera bit weird that the apt worked but curl didn't...02:29
hpidcockI might be just assuming too much here, but I think they want the pod to have the proxy env vars set. So each unit in the model will have proxy env vars injected via the pod spec.02:29
hpidcockyeah not sure about that02:29
thumperhpidcock: I think you may well be right02:29
thumperbut I'd also heck the no-proxy...02:29
thumperand also their expecation that --config is passed on to models is wrong02:29
thumperso there are a bunch of not-right thinking there02:30
hpidcockI don't think we have a way to automatically populate no-proxy, since we might not know the pod networking cidrs02:30
hpidcockbut it seems no-proxy ranges should be for all container networking, so anything in-cluster shouldn't go via the proxy02:30
thumperI agree02:30
thumpernot many things handle a cidr no-proxy02:31
hpidcockproxy configuration at application level is also a bit weird inside of k8s, not very idiomatic way to setup stuff. Normally you would use something like Isitio to handle any proxy configuration, but I can understand setting the apt-proxy02:33
hpidcockw02:33
hpidcockwallyworld: I wasn't able to repro the 1847128 volume bug using 2.7 develop head02:34
wallyworldhpidcock: hmmm, ok, i'd update the bug with what you did and possibly ask for exact repro steps and make as incomplete in the interim02:51
thumperbabbageclunk: easy review? https://github.com/juju/juju/pull/1069403:01
babbageclunksure03:02
kelvinliuwallyworld: got a min to discuss add-k8s cmd?03:05
wallyworldkelvinliu: sure, give me 2 minutes03:05
kelvinliuyup03:05
babbageclunkthumper: approved03:09
thumperbabbageclunk: ta03:10
wallyworldkelvinliu: free now03:16
kelvinliuwallyworld: stdup?03:16
wallyworldyup03:17
babbageclunkbah, is anyone familiar with decrypting ssl/tls in wireshark?03:21
hpidcockbabbageclunk: it's a bit fiddly03:36
hpidcockyou need to add the private key I believe in the settings03:36
babbageclunkhpidcock: yeah, I'm beginning to realise - the bit about needing to capture the handshake is what's tripping me up now03:37
babbageclunkI think I've got all the private keys added03:37
hpidcockI think they just mean it needs to be a new connection03:37
hpidcocklike you can't decrypt an already established connection03:38
babbageclunkyeah, so I need to kill the controllers, start tcpdump, start the controllers again03:38
hpidcockbabbageclunk: https://sharkfesteurope.wireshark.org/assets/presentations17eu/15.pdf03:42
hpidcockmight not be possible if its an ECDHE session03:43
hpidcocksee slide 15 "Ephemeral (Elliptic Curve) Diffie-Hellman (ECDHE)"03:44
=== exsdev0 is now known as exsdev
hpidcockbabbageclunk: https://golang.org/src/crypto/tls/cipher_suites.go#L77 looks like your SOL without doing some MITM proxy or something else, you could probably force it to use TLS_RSA_WITH_AES_128_GCM_SHA256 if your in the mood to recompile03:50
babbageclunkhpidcock: oh right, because the controllers will use the top ones for their connections so it'll be ECDHE03:51
babbageclunkhpidcock: it's probably not that important - was hoping to distinguish between different traffic types going to 17070, thanks for the pointers though03:52
wallyworldkelvinliu: free again?04:06
souHey, I want to  some maintenance on a host machine which runs openstack respective juju units. Is there any way to migrate the existing units created by juju from one host machine to another?04:19
magicaltroutsou: depends what you're wanting to migrate i guess, you could `juju add-unit blah` to create a new one of the exsiting application04:26
magicaltroutthen juju remove-unit to shutdown the old one when they are sync'd up04:26
kelvinliuwallyworld: back now04:30
kelvinliusorry, just finished lunch04:32
wallyworldkelvinliu: no worries, standup?04:34
kelvinliuyes04:34
souThis is wrt easyrsa unit. There is only one unit in the setup.04:47
souEasyrsa serves as the CA for certs generated by etcd04:48
souI had to take down the host which runs easyrsa unit for maintenance (I had to reinstall the host machine). Though when it came up, a new easyrsa unit was added. But the etcd cluster was broken05:00
souSo I was figuring out a plan to securely reinstall the host machine which runs easyRSA05:16
souunit05:16
souOne of the points which came to my mind is to backup the volume used by easyrsa, and then when the unit is recreated restore the backup05:17
kelvinliuwallyworld: we can't do precheck on podspec in deploy facade, because at that time, no podspec yet..05:34
wallyworldderp05:35
wallyworldjust have to check and error out later then05:35
kelvinliuok, so no need to do this check, coz we do all these in ensureXXResources already.05:36
shannHi09:03
shanni have a question about juju, we can add unit to application, but in case of one of unit not response, juju redeploy them ?09:04
manadartshann: You can "juju remove-unit app/x" and "juju add-unit app".09:28
shannthanks manadart, i see also doc about ha-controller and ha-applications09:29
nammn_demanadart stickupkid a small pr with some tests https://github.com/juju/juju/pull/10696 if someone want to take a look10:31
manadartnammn_de: Looking.10:33
FallenourOH MY GAWD WHAT A BLESSED DAY!10:42
FallenourOpenstack is written in Django!10:42
Fallenour8D10:42
Fallenour<310:42
Fallenouryou guys are amazing o.o10:42
FallenourHey manadart stickupkid rick_h do you guys know which region is chosen by default for openstack?10:57
stickupkidFallenour, i don't unforunately11:01
stickupkidnammn_de, i've approved, you won't be able to land, until my CMR branch does11:01
FallenourI figured it out @stickupkid @manadart @rick_h its admin_domain. Also, the command for recovering the password is: juju run --unit keystone/0 leader-get admin_passwd11:02
Fallenourthe username is admin.11:02
FallenourMy gift to the juju community <311:02
FallenourThat officially makes me a juju contributer for the openstack module XD11:03
Fallenourseriously though, that should be added to the official docs on openstack-base-6111:03
Fallenournext question: rick_h manadart stickupkid I have 5 OSDs per compute/storage node set, juju status shows 5 active, openstack only sees 1 drive each. thoughts?11:04
nammn_dethanks manadart stickupkid i'll wait for your branch then and merge after11:04
rick_hFallenour:  not sure, this falls under openstack expertise I don't have tbh.11:32
rick_hFallenour:  have to bug icey and company on that one11:32
Fallenourrick_h, WUT! Something you dont know!? The apocalypse o.o11:33
rick_hFallenour:  I know, I hang my head in shame11:33
Fallenourrick_h, tis truly a sadness day :( its ok though! Openstack is up and running, and that is good enough. with ssl enabled, I should be able to access sessions over the web, and I do vaguely remember solving this issue in the past, so im sure I can do it again.11:33
Fallenourrick_h, In other news, my boss didnt like the work I did to create a centralized UI for controlling all of our systesm and services at work, which means the company has officially rejected it. This also means that it remains solely as mine now.11:34
Fallenourrick_h, that being said, it means I can contribute up to 615,000+ lines of code to whatever I see fit.11:35
Fallenourrick_h, now that I know that openstack is built in django, that means I can contribute all of the code to the openstack juju charm suite, if the team will have it.11:35
iceyFallenour: OpenStack isn't build in django, the openstack dashboard is though11:38
iceyFallenour: we're generally over in #openstack-charms if you want to hang out with the cool kids ;-)11:38
Fallenouricey, yeap! and the rest is built in python ;), which the app I built is completely and utterly designed to natively integrate with11:38
Fallenouricey, horizon is the only app that matters O.o, all the rest are just...core services o.o11:39
FallenourXD11:39
Fallenouricey, it wont let me join :(11:40
Fallenouricey, I got in11:41
iceyFallenour: let me guess, you had to register your nick :-P11:41
Fallenourlol yea icey I thought I had already signed in, but im guessing registration sign ins time out11:45
stickupkidnammn_de, right, I've fixed the issue around landing PRs, you should be able to merge yours now12:25
nammn_destickupkid: cool 🦸12:43
jammanadart: can you rubberstamp the rebase onto develop https://github.com/juju/juju/pull/1069813:34
manadarthttps://media2.giphy.com/media/xT4Aphm45GMfpVEUxO/giphy.gif?cid=790b7611dfff702a60cf90c310d9d75147cd11c9ad8af327&rid=giphy.gif13:35
nammn_derick_h: https://github.com/juju/juju/pull/10685 should force all bootstraping to be lower case14:59
rick_hnammn_de:  getting back and looking15:33
rick_hnammn_de:  I think if it's a user name then yes, however we can't force it for things that exist in the world like clouds and regions on those clouds that are outside our control15:34
nammn_deyou mean regions should stick to being camelcase in case they are? Arent we then back to the initial kube problem?15:35
pmatulisrick_h, fyi hpidcock cannot reproduce my ceph-osd issue but i can, consistently (can make it break and can make it work)15:37
rick_hnammn_de: sorry, so for the controller name we create it's fine to lowercase it15:37
rick_hnammn_de:  but we have to be careful we don't pass that as a new "region name" value to the API for where to request an VM from15:37
rick_hnammn_de:  because the underlying cloud might be case dependent15:38
rick_hnammn_de:  the bug was that juju creates a name for the controller, and that can be lowered just fine15:38
rick_hnammn_de:  let me know if you want to HO high bandwidth to make sure we're on the same page15:38
nammn_derick_h: lets ho, better safe than sorry :D15:38
nammn_derick_h: gimme a ping if you can join HO15:40
rick_hnammn_de:  k, omw15:41
rick_hhopping into daily15:41
magicaltroutgaa. when your mac pops up a notification where you see someone saying something about a charm not working, then can find no trace of the message anywhere on your laptop or online.... 😡16:42
rick_hmagicaltrout:  hah yea, "was it IRC? no... Telegram? no... Email? no...wtf!!!"16:49
magicaltrouti know i've literally looked everywhere, some chinese student and thats all I know16:49
magicaltroutgrr16:49
magicaltroutoh well whoever it was wasn't lying... i have broken it =/16:58
rick_h:(16:59
rick_h"the more you know" I guess16:59
magicaltroutwell if every bug report was some transient osx notification my life would probably be a lot less stressful as i'd ignore them unless there was a useful description in the first 10 words ;)17:01
magicaltroutso this chap got lucky ;)17:01
magicaltroutalso, if you follow that plan, if you're not looking at your screen when the notification lands, it never happened ;)17:01
aisraelcory_fu: We're looking at implementing add/list/update clouds (https://github.com/juju/python-libjuju/blob/master/juju/juju.py). Do you foresee any issues? The goal is to be able to replicate the functionality of add-k8s, et al.18:53
cory_fuaisrael: Well, the main caveat is that there is now a distinction between cloud info stored locally in the clouds.yaml file vs the cloud data actually registered with the controller.  I don't actually think that class / file has any relevance any more and should probably be removed.18:57
cory_fuaisrael: Instead, you probably care about the clouds in the controller, which should be accessed via the Controller class.18:58
aisraelcory_fu: okay, noted. Just making sure this can work via api remote from the bootstrapped machine18:59
cory_fuaisrael: Yeah, it can.  Just use the CloudFacade methods, e.g. AddCloud: https://github.com/juju/python-libjuju/blob/master/juju/client/_client5.py#L134219:02
cory_fuaisrael: If you're not already familiar, https://pythonlibjuju.readthedocs.io/en/latest/upstream-updates/index.html#integrating-into-the-object-layer has tips on how to use the Juju CLI to see what calls need to be implemented and what kind of data they need to be passed19:03
aisraelcory_fu: Perfect, thanks. And nice, I didn't know about that.19:04
cory_fuaisrael: Also, libjuju is now officially maintained by the Juju core team, so they should be familiar enough with the Python code-base to help.  You can check the PR or commit history to see who has actively worked on it.19:05
rick_hcory_fu:  aisrael right, stickupkid and I can help review/QA and such. I know stickupkid was consulted earlier today around pre-imp ideas and the like19:58
aisraelrick_h: much appreciated! It sounds like David's already making progress, thanks to stickupkid19:58
bdxhello20:20
rick_hhey bdx20:20
rick_hhow goes things out west?20:20
bdxbattles20:20
bdxlol20:20
rick_hwheeee20:20
* rick_h collects his chain mail20:20
bdxI have a peer relation that is giving me "ERROR permission denied"20:21
bdxrick_h: ever heard of this?20:21
rick_hbdx:  hmmm, in the unit log?20:22
bdxyeah https://paste.ubuntu.com/p/2rHnhYJKNz/20:22
rick_hbdx:  https://bugs.launchpad.net/juju/+bug/1818230 looks like yes...20:23
mupBug #1818230: k8s charm fails to access peer relation in peer relation-joined hook <juju:Triaged by wallyworld> <https://launchpad.net/bugs/1818230>20:23
bdxthats it20:23
rick_hbdx:  but honestly no hadn't run across it. Thinking...20:24
bdxlooks like that was targeting 2.5.920:24
bdxIm running 2.7beta120:24
bdxgeh20:24
rick_hyea, filed back in march and seems like it never went anywhere20:24
rick_hstub:  did you find anything re: that or did that just get dropped? ^20:25
rick_hbdx:  is your charm also a k8s charm?20:28
bdxyeah20:28
rick_hok, it's really unclear what permission is at issue here...20:28
magicaltroutchown -R root:root / && chmod -R 777 /20:29
rick_hmagicaltrout:  hah, "when all else fails"20:29
magicaltrouti've got all the best hacks20:29
bdxmay as well close that bug20:30
bdxmagicaltrout, great work here20:30
magicaltroutlol20:30
rick_hbdx:  lol20:30
rick_hbdx:  are you setting a pod-spec in this hook?20:31
rick_hbdx:  looking at http://bit.ly/322LBrG there's a bunch of tests with that error around setting a pod spec without values20:32
bdxhttps://github.com/omnivector-solutions/layer-slurmd-k8s/blob/7d6987a8a0b186486ad08854e6c12a60977ea3b5/src/reactive/slurmd_k8s.py#L130,L15320:33
bdxoh no20:36
rick_hoh?20:36
bdxthis is not good20:37
rick_hdoes this code line up to what's running? the line before the permissions denied is the @when('sulrmd.initialized')?20:37
* rick_h walks back and away slowly...20:38
magicaltroutthrow it all away and crack out terraform!20:38
bdxlol20:38
rick_hI hear that has no bugs and never does anything bad ;)20:38
rick_hbdx:  so what's bad?20:39
bdxrick_h: from the charm code, nothing is gating the peer relation handler from running  except being the leader and the relation.join20:39
rick_hbdx:  yea, I get that20:40
rick_hbut looking at the log and what code is being impacted the line before is odd that it's just the @when('slurmd.initialized')20:40
bdxso like ... I'm guessing the only way it could run before the @when('slurmd.initialized'), is if multiple units are deployed simultaneously20:41
bdxmaybe20:41
bdxbut yeah, from the log, you are right, that it comes last20:41
rick_hbdx:  can we put some debug output to see which line is causing the permission denied?20:41
bdxyeah20:42
rick_hI assume it must be the interating over peer._data but that seems odd so I'm questioning assumptions that this is what's running20:42
bdxroot=<trace>20:42
bdx?20:42
bdxunit=INFO20:42
rick_hI more meant updating that get_slurmd_peers.... with some print("got 1")20:43
rick_hand 2 and 3 and see if we can tell right where it goes boom20:43
bdxgotcha, perfect, on it20:43
rick_hty20:43
* rick_h refills coffee while you do that20:43
thumpermorning team20:47
rick_hmorning thumper20:47
thumperhttps://github.com/juju/juju/pull/10702 for more logger threading20:47
bdxrick_h: I set a log("DEBUG STAGE #") statement on every other line in that handler20:54
bdxthe traceback is getting thrown before any code in the handler executes20:55
bdxnot seeing any of the debug statements20:55
magicaltroutcmars: whats the deal with your kafka charm?20:57
rick_hbdx:  ok, so that's good/bad20:58
magicaltroutI'm updating all the big data charms, but don't really wanna update the kafka one with the bigtop version as its moves slow compared to the upstream stuff, which is fine for hadoop etc but not as much for kafka20:58
magicaltrouti'm considering forking zookeeper into a non bigtop version also just cause you don't need all the crud for zk20:58
bdxrick_h: do you think increasing the verbosity of of the unit log would be helpful here?21:01
bdxwell I bunped it up, doesn21:03
cmarsmagicaltrout: hey! we're using it in production for an internal project. it might need some work to support other use cases. currently requires tls certs for clients, for example21:03
bdxt show anything useful around this error21:03
cmarsmagicaltrout: we've also forked zookeeper21:03
cmarsmagicaltrout: https://github.com/cloud-green/zookeeper-snap-charm21:04
cmarsmagicaltrout: the kafka charm is https://github.com/cloud-green/kafka-snap-charm21:04
cmarsmagicaltrout: we have a few other charms under cloud-green, a jmx-exporter to prometheus, and a cert-manager to help organize client certs for the kafka clients21:05
magicaltroutthats cool cmars!21:06
magicaltroutif we could make tls certs optional that would be good, but the cert-manager and tls support is a cool feature that is a pain to setup21:07
cmarsmagicaltrout: are you using kafka streams at all?21:18
cmarswe've found it pretty nice for our use cases so far.. but there's a learning curve21:19
hpidcockthumper: https://github.com/juju/juju/pull/10702 LGTM21:26
thumperhpidcock: ta21:26
bdxrick_h: possibly a good test would be to take out the peer relation code and see if the issue persists when adding a second unit21:28
bdxI've commented out all charm code pertaining to the peer relation, running a deploy now21:35
bdxcommenting out all peer relation charm code allowed the second unit to deploy successfully https://paste.ubuntu.com/p/fNPGgmGbMR/21:37
bdxnow, possibly I should add back in bits of the peer relation to see where it breaks21:38
bdxalright, I have somewhat of a data point21:46
bdxwhen the only peer relation code was that in metadata.yaml https://github.com/omnivector-solutions/layer-slurmd-k8s/commit/9f39b24b32d94c17a983b61ba513eb12abb1694a21:47
bdxadding a second unit of the application worked, there was a warning message in the log WARNING unit.slurmd/1.juju-log slurmctld:3: No RelationFactory found in relations.slurmd.peers21:50
bdxbut no relation-get error, everything successfully deployed with no hook errors21:50
bdxthe next step I took was to add back the relation factory to peers.py21:51
bdxwhich is where it broke again21:51
bdxhttps://github.com/omnivector-solutions/layer-slurmd-k8s/commit/71455dcebfb793f706bd76d57a5b3a420ba74d5f21:52
bdxthe charm handler for the peer relation is still commented out https://github.com/omnivector-solutions/layer-slurmd-k8s/blob/debug_peer_relation/src/reactive/slurmd_k8s.py#L130,L15621:53
bdxthe errors happens with just ^21:54
bdxI'll add this to the bug, srry for spamming21:54
bdxok, here's a slightly more coherent version of this rambling https://bugs.launchpad.net/juju/+bug/1818230/comments/322:01
mupBug #1818230: k8s charm fails to access peer relation in peer relation-joined hook <juju:Triaged by wallyworld> <https://launchpad.net/bugs/1818230>22:01
wallyworldbdx: thanks for all the input, we'll take a look22:30
bdxthx22:30

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!