/srv/irclogs.ubuntu.com/2017/02/06/#juju.txt

=== frankban|afk is now known as frankban
aisraelAny conjure-up folks around? I installed the snap from the beta channel and `conjure-up` fails with: ImportError: No module named 'conjureup'10:45
aisraelhttps://github.com/conjure-up/conjure-up/issues/65210:53
=== jamespag` is now known as jamespage
Odd_Blokestokachu: ^12:11
aisraelstokachu, for context: I'm at the Juju Summit, and wanted to test conjure-up before my talk. We're pushing conjure-up pretty hard, so I'm hoping my failure is just a one-off12:17
stokachuaisrael, you are running the snap one from /snap/bin/conjure-up?13:12
aisraelstokachu, yep13:12
stokachuaisrael, one sec13:13
stokachuaisrael, hmm, do you have any pyenv environment enabled?13:13
stokachuon a fresh 16.04 install i dont see this problem13:14
aisraelstokachu, nope, nothing special wrt pyenv13:14
stokachuaisrael, hmm..13:16
stokachuaisrael, whats the output of `env`?13:19
aisraelhttp://pastebin.ubuntu.com/23940964/13:21
stokachustarting up a fresh system again to make sure13:21
stokachuaisrael, xenial?13:23
aisraelstokachu, Yep, 16.04.1, up to date as of this morning13:23
stokachuok13:23
stokachuaisrael, hmm im having trouble reproducing, i just did a fresh maas deploy of 16.04.113:26
stokachuaisrael, maybe create a fresh user account and try running it with that user13:27
aisraelstokachu, ok. I'll fire up an lxd container and see if I can reproduce it locally. It might just be something on my system, and if so, I'll try to debug and provide steps to reproduce.13:27
stokachuaisrael, thanks, the python plugin for snap is pretty specific when it comes to the interpreter it runs13:28
stokachuso it may be something with that im missing13:28
aisraelI can also try building the snap locally and see if that changes anything13:31
stokachuaisrael, thanks, it uses python 3.6 within the snap, and unsquashfs -l conjure-up*snap should show you the module path13:33
stokachuaisrael, http://paste.ubuntu.com/23941001/13:34
stokachuso the module is definitely there13:34
aisraelI only have python3.5 installed. Does the snap include the 3.6 runtime?13:41
stokachuyea it does13:51
stokachuaisrael, any luck?14:00
stokachumup, ping14:26
mupstokachu: I apologize, but I'm pretty strict about only responding to known commands.14:26
admcleodhi - im trying to use a bundle (juju 2.1, juju deploy) with manually provisioned machines. the machines are provisioned and numbered 0,1,2,3,4, the bundle refers to them as such also, and the placement of the charms in the bundle refers to them either as that number (e.g. charm ceph-mon, to: 1, or to: - lxd:1, etc) - but when i try to deploy, it tells me "cannot create machine for holding"14:44
bdxadmcleod: sup, the machines specified in the bundle don't correlate to the machine numbers in the juju environment14:48
admcleodbdx: thats what i was thinking14:49
bdxthe machine placement directives in the bundle are context sensitive to that bundle, and deploying a bundle will always try to provision new machines, not map to the machines in the bundle :(14:50
admcleodbdx: well, thats peculiar since the rest of the error says:14:50
admcleodcannot add a new machine: use "juju add-machine ssh:[user@]<host>" to provision machines14:50
admcleodright. but they're already there...14:50
bdxyeah .. that would make sense, because the bundle doesn't know about them, and because manual provider, juju doesn't have access to a hoard of resources to create new machines with14:51
admcleodbdx: right. but if it says 'add a machine manually', then if i was to do that, presumably it wouldnt know, and ask me to add another machine manually14:52
bdxyea .. exactly14:52
admcleodbdx: that seems like a bug14:52
bdxbundle deploys <-> manual provider have never worked for me ... yea totally14:53
sparkiegeekwasn't there a ML thread about this?14:53
bdxadmcleod: but what if you want new machines? ^ doesn't fit the rest of the use cases though14:53
admcleodsparkiegeek: ??14:54
sparkiegeekadmcleod: https://lists.ubuntu.com/archives/juju/2016-December/008367.html14:54
admcleodsparkiegeek: thanks14:55
sparkiegeekadmcleod: np14:55
stokachuaisrael, were you able to get conjure-up going?15:03
aisraelstokachu, not yet. Just finished with my talk, though, so I'll give it a try15:30
Zichi, if somebody of the CDK team is here, I got the same error (Unable to authenticate the request due to an error: crypto/rsa: verification error)on a fresh brand new cluster15:56
Zicwith no operation post-install15:56
Zicwe just poweroff-ed one master to test the HA, and when we poweron-ed it, it came back with this error which totally mess the whole cluster :/15:57
Zichttp://paste.ubuntu.com/23941613/15:57
magicaltrouthmm dunno if jcastro has irc open to ping one of the CDK guys15:59
magicaltroutthey're all in Gent15:59
Zicthe last time, this error appeared few days after some operations on our side, so we didn't know how to reproduce16:01
Zicbut here, it's *just* after the end of the install (when all juju status was green)16:02
magicaltrouti've tried to ping them on twitter Zic16:02
Zicthanks :)16:02
magicaltroutwe shall see how connected they are16:02
Budgie^Smoreor the mailing list, I know lazy said he would see things there16:02
magicaltroutyeah but its not real time is it! ;)16:03
rick_hThe signal fires are lit16:04
magicaltrouthehe16:04
lazyPowermagicaltrout - state the nature of your kubernetes emergency16:05
magicaltrouthehe16:05
magicaltroutnot mine16:05
Budgie^Smorespeak of the devil! ;-)16:05
ZiclazyPower: guess who is crashing everything? \o/16:05
magicaltroutZic here has broken his cluster16:05
lazyPowerZic - ah, i could have guessed :) I spoke about you today in my talk16:05
lazyPowerok so what seems to be the trouble Zic?16:05
magicaltroutwas it something like "poor guy, keeps testing our code and breaking things?" ;)16:06
ZiclazyPower: you remembered the credential error with "Unable to authenticate the request due to an error: crypto/rsa: verification error"?16:06
lazyPowermagicaltrout - nah, it was actually describing how he was able to make an irrecoverable state change, and i was quite impressed16:06
ZicI had it after some days of operation and it was not clear how we can reproduce it16:06
lazyPowerZic i do remember something about this16:06
lazyPowerZic did you manage to reproduce it? or is just showing up again without a clear indicator as to why?16:07
Zichere, I just deployed a brand new cluster and just after the end of install (`juju status` showed all green) it starter immediately after we rebooted one of the 3 masters16:07
Zicstarted*16:07
lazyPowerahhh16:07
lazyPowerso restarting a master has triggered a cryptographic verification error?16:07
Zicyeah16:07
lazyPowerZic - we definitely need to document that in a bug and take a look16:08
Zicwe did the check that you pointed me to the last time : check that /srv/kubernetes corresponding with openssl x509, it does not change16:09
Zicmaybe it's the k8s token? but I don't know how they work16:09
lazyPowerah good, so the TLS cert was verified16:09
lazyPowergood call, i bet i know whats happened Zic16:09
lazyPowerZic - those tokens need to be syc'd among the masters, and they aren't right now. so you've got a new token that it loaded into the k8s object store during turn up16:10
lazyPowerand its not agree'ing with what the other msters have.16:10
lazyPowerwhich would be why it showed up after reboot, not before16:10
lazyPoweractually i'm kind of perturbed we didn't find this before reboot16:11
Zicwe think about his because one of the event of the Ingress controller was about readyness16:12
Zickube-dns also16:12
lazyPowerZic - they mount the default service token right?16:13
Zic-> requests that ask the API with token16:13
lazyPowerZic - so the token is different across all 3 masters16:13
ZiclazyPower: how can I confirm that?16:13
lazyPower1 sec i'm turning up k8s on my laptop16:13
lazyPowerZic - check in /srv/kubernetes/known_tokens.csv16:14
lazyPowerall 3 masters will have a different token in that file is my guess16:14
Zichttp://paste.ubuntu.com/23941724/16:15
lazyPowerZic - seems like thats the culprit16:16
lazyPowerZic - great pointer on this one16:16
ZiclazyPower: just before this error (and just after the install, no additionnal extra OP) we discovered that something went wrong because kube-dns and one of the Ingress controller was in CLBO16:19
Zicdo you confirm it can be tied to this problem? as it probes the readiness through API I believe16:19
lazyPowerZic - yeah you should be able to confirm that with a kubectl describe16:21
lazyPowerZic - if its failed contacting the API with that token, i'm mostly certain you'll get that as output in the kubectl describe of that pod16:21
lazyPoweri may be incorrect and it only say it entered a failed/crashed state16:21
lazyPowerbut i'm mostly certain it'll give you more detailed info w/ regards to an api contact failure16:21
Zichttp://paste.ubuntu.com/23941761/16:22
Zic(for kube-dns)16:22
lazyPoweryap, i'm wrong16:22
lazyPowerit would have to be verified with log output16:22
aisraelstokachu, I'm running into a problem building the snap. It wants me to install the core snap, but I try and it won't because ubuntu-core is already installed (and uninstallable). Run into that before?16:23
Zicmp          10m         11m         18        zk-0               Pod                                 Warning   FailedSync         {kubelet mth-k8s-01}          Error syncing pod, skipping: failed to "SetupNetwork" for "zk-0_mp" with SetupNetworkError: "Failed to setup network for pod \"zk-0_mp(c5559d40-eafe-11e6-a57e-0050569efde9)\" using network plugins \"cni\": \"cni0\" already has an IP address different16:24
Zicfrom 10.1.6.1/24; Skipping pod"16:24
stokachuaisrael, what about `sudo snap refresh ubuntu-core`16:24
stokachuaisrael, see if it'll update and rename it16:24
aisraelno updates available16:24
stokachutry sudo snap refresh core16:24
ZiclazyPower: I have this one also, but it's on the old cluster (not the fresh one) don't know if we can link all this bugs16:24
lazyPowerZic - that CNI failure seems to be bubbling up FROM CNI itself though16:25
lazyPowerZic - wsa this similar in symptom though?16:26
aisraelstokachu, no luck. can't refresh 'core', can't find 'core'16:26
lazyPowerit started with cryptographic errors and yielded broken behavior?16:26
stokachuaisrael, what does `snap version` show16:26
lazyPowerZic - also, why aren't you here :) :) We could be hacking on this in real time16:26
aisraelsnap    2.2116:27
aisraelsnapd   2.2116:27
aisraelseries  1616:27
aisraelubuntu  16.0416:27
stokachuwell wth16:27
ZiclazyPower: because I'm stick with the customer to debug this x)16:27
lazyPowerZic - also, thanks for following up with this.16:28
stokachuaisrael, can you hop on #snappy on freenode16:28
lazyPowerZic ah good reason. I'll see if i can get this fixed up in a branch tonight fo you to deploy and test with.16:28
ZiclazyPower: but I'm counting on you to tell me all your stories about the Juju Summit :p16:30
lazyPowerZic ah, let me show you this :D16:30
lazyPowerhttps://docs.google.com/presentation/d/1kJ-OeazxivyMkQbDC20wD-NDkW4T-H8BUTfQNxw3wOE/pub?start=false&loop=false&delayms=3000&slide=id.g1c62525350_1_9016:30
Zicor we'll be glad to welcome you at my company x)16:30
Zicit's not to far from where you are :)16:30
lazyPowerhehe great :D16:31
ZicGent -> Paris, and we offer you pizza :D16:31
ZiclazyPower: the best part is about "Operators and devs be like" slide :p16:37
ZiclazyPower: can I do something for you before the hotfixing comes? to help you with?16:40
lazyPowerZic -well16:40
lazyPoweryou're gonna be the primary driver of testing16:41
lazyPowerthat's enough for me. I don't think there's anything other than ensuring the bug is filed so i can link it on the commit.16:41
Budgie^SmoreSo etcd is becoming the new duck tape is what I like to say :P16:41
lazyPowerBudgie^Smore - hahaha16:41
lazyPowernailed it16:41
Budgie^SmorelazyPower was your talk recorded by chance?16:42
xnoxstokachu, what is the difference between "kubernetes core" and "canonical distribution of kubernetes" when doing conjure up?16:43
lazyPowerBudgie^Smore - it was16:43
lazyPowerxnox - kubernetes-core is a much lighter bundle, it only rqeuires 3 units16:43
lazyPowerxnox canonical-distribution-of-kubernetes requires 9 units out of the gate, and introduces an API LoadBalancer to work around kubernetes not properly contacting the masters in an HA formation (they only contact the first)16:44
lazyPowerso those are the 2 initial differences16:44
xnoxlazyPower, is the charm and the binary that runs kubernetes the same?16:44
lazyPowerxnox correct16:44
xnoxgood16:44
xnoxlazyPower, thanks a lot!16:44
Budgie^SmorelazyPower I think I found a problem with the loadbalancer, was trying to use helm the other day and it couldn't do a port-forward command16:45
lazyPowerxnox no problem :) happy to help16:45
lazyPowerBudgie^Smore - we documented a work-around for the mean time16:45
xnoxlazyPower, where can i do the lookup to where the conjure up "recipe is", bundle, and charms?16:45
lazyPowerits known that helm hates our LB, it doesn't talk SPDY16:45
Budgie^SmoreOK :) just checking, decided not to use helm at the moment and just use their charts as templates to self deploy16:45
Zicok, I will fill the bug lazyPower :) https://github.com/juju/juju is the right place?16:48
lazyPowerZic - github.com/kubernetes/kubernetes16:48
lazyPowerBudgie^Smore - ok, we're goig to be replacing the current APILB with a layer4 reverse proxy16:49
lazyPowerso that issue should have a shelf life16:49
Budgie^Smoreawersome, might have a use for helm stuff then :)16:49
stokachuxnox, https://github.com/conjure-up/spells/tree/master/canonical-kubernetes16:50
stokachuxnox, metadata will have a link to the bundle location16:50
xnoxspell - that's the word!16:50
lazyPowerman, as much as i like interfacing with you people but im ready to get back to engineering :P16:51
lazyPoweras i'm sure you who are patiently waiting for fixes are ready for too16:51
Budgie^Smorenope, coming up with work aroundsi in the meantime ;-)16:51
stokachuxnox, were you able to navigate the ui without it freezing?16:52
stokachuxnox, i saw you mention that earlier16:52
xnoxstokachu, i have not. but i was advised at fosdem that maybe i should not be trying to do this with: zesty, on ipv6 only network16:53
stokachuxnox, ah16:53
stokachuxnox, yea ipv6 :(16:53
xnoxstokachu, at fosdem they have ipv6 networking only by default with dns6416:54
xnox( https://fosdem.org/2017/ )16:54
stokachuxnox, fancy :)16:54
xnoxit's a free event as well =)16:54
xiletWorking on my first charm, are there any simple ways to have a templated config file brought in that does a simple replace on hostname? ( All of the samples I found were specific to applications requiring a good bit of python ). I mean I can always use a language to do it but I don't know if the charm system has something already in place.17:18
=== mskalka is now known as mskalka|lunch
=== mskalka|lunch is now known as mskalka
stokachuxilet, you doing it in bash?17:33
stokachuxilet, are using charmhelpers?17:33
xiletRight now all of the hooks are in bash17:38
xiletLet me ask a concrete example. If I wanted to add a line to /etc/hosts with <ip address> juju_server_<unit ID>     what would be the best way to accomplish that?17:56
vagarwalthanks to all the speakers and organizers - nice talks today18:06
=== menn0 is now known as menn0-otp
=== frankban is now known as frankban|afk
stokachuaisrael, your talk go ok today?19:37
stokachuarosales, conjure-up work out for you all?19:37
arosalesstokachu: so far so good. More work shop time tomorrow19:38
stokachuarosales, cool man19:38
arosalesbut encouraged folks to try what they saw in presos today with conjure19:38
stokachuarosales, sweet hopefully you'll get some good feedback19:39
arosalesand increased usage of conjure19:39
stokachuyea im looking at the numbers now19:39
stokachu10 for spark processing today19:40
stormmoreso has anyone worked on securing the CDK API load balancer so it doesn't use HTTP?20:10
=== menn0-otp is now known as menn0
catbus1stokachu: hi, I tried  to install conjure-up on 16.04 but it says --classic flag is unknown.22:40
catbus1stokachu: do you know if there is any change to the flag? This is what usermanaul says $ sudo snap install conjure-up --classic --beta22:41
catbus1using apt install instead22:48
magicaltroutany charmers awake?23:02
magicaltroutkwmonroe: you alive?23:09
kwmonroeyo yo magicaltrout23:11
kwmonroehow may i direct you to cory_fu today?23:11
magicaltroutlol23:11
magicaltroutall your counterparts over here are shit23:11
magicaltroutand have gone to bed23:11
magicaltroutcharm push . cs:~spiculecharms/apache-solr23:12
magicaltroutERROR cannot post archive: unauthorized: access denied for user "spicule"23:12
magicaltroutany idea?23:12
cory_fumagicaltrout: Sorry, I'm EOD and headed to bed.  ;)23:12
cory_fumagicaltrout: Kidding.  Have you tried doing "charm logout" and "charm login"?23:12
magicaltroutjust did that23:13
magicaltroutsame result23:13
cory_fuHrm23:13
magicaltroutwe were discussing you earlier. I made the point drinking is so much easier when you aren't around! ;)23:14
cory_fumagicaltrout: Go to jujucharms.com and log out / in there, then charm logout/in again23:14
cory_fuha23:14
magicaltrouti have this cool trick of hitting weird bugs the night before a talk23:15
kwmonroemagicaltrout: i'm gonna guess this is because you have all these identities.  who does "charm whoami" think you are?23:17
magicaltroutbugg@tom-laptop2:~/Projects/charms/builds/apache-solr$ charm login23:17
magicaltroutLogin to Ubuntu SSO23:17
magicaltroutPress return to select a default value.23:17
magicaltroutE-Mail: tom@analytical-labs.com23:17
magicaltroutPassword:23:17
magicaltroutTwo-factor auth (Enter for none):23:17
magicaltroutbugg@tom-laptop2:~/Projects/charms/builds/apache-solr$ charm push . cs:~spiculecharms/apache-solr23:17
magicaltroutERROR cannot post archive: unauthorized: access denied for user "spicule"23:17
magicaltroutbugg@tom-laptop2:~/Projects/charms/builds/apache-solr$ charm whoami23:17
magicaltroutUser: spicule23:17
magicaltroutGroup membership: apachefoundation, apachesoftwarefoundation, charm-contributors, containers23:17
magicaltroutbugg@tom-laptop2:~/Projects/charms/builds/apache-solr$23:17
magicaltrouthmm23:18
magicaltroutso i should be a member of my own group right?23:18
magicaltroutgroup/team23:18
magicaltroutwhatever launchpad calls it23:18
kwmonroeindeed you should23:20
kwmonroealso, how in the heck did you get into the containers group?  those folks are picky.23:20
magicaltroutcause i'm bloody amazing23:20
magicaltroutanyway23:20
kwmonroelol23:20
magicaltrouthow did i end up out of my own group23:20
magicaltroutand how do i get back into it?23:20
cory_fumagicaltrout: You are a member of the group, but the charm store needs to refresh from LP.  Did you go to jujucharms.com like I said?23:21
kwmonroeyeah magicaltrout, it does look like spicule is in the group:  https://launchpad.net/~spiculecharms/+members.  do what cory said.23:22
magicaltrouti did cory_fu23:22
magicaltroutas if i wouldn't do something cory_fu suggested!23:23
cory_fumagicaltrout: Hrm.  I don't know.  That's a known issue and that's the fix for it.  Maybe try charm logout, jujucharms.com logout, jujucharms.com login, charm login?23:23
cory_fu:)23:23
cory_fuI've definitely had that fix that issue before23:24
kwmonroeyeah - it's gotta be in an order iirc.  luckily there aren't many permutations to try ;)23:24
kwmonroemagicaltrout: as a workaround, could you push to ~spicule and use the charm from there?23:26
kwmonroethen sort out the group membership at a more reasonable hour?23:26
magicalt1outsame fail23:29
magicalt1outgreat i'm locked out of pushing my own charms \o/23:30
cory_fuO_O23:30
magicalt1outi'll deploy locally and complain on the mailing list23:30
magicalt1outweird though23:30
kwmonroemagicaltrout: can you push to one of those other listed groups?  apachefoundation, apachesoftwarefoundation, charm-contributors, containers23:31
magicalt1outtesting23:32
magicalt1outyes kwmonroe23:33
magicalt1outthat works fine23:33
catbus1Hi, I tried to add maas cloud to conjure-up, but it errors out with trackback: unable to find: <home folder>/.local/share/juju/accounts.yaml. And there is also 'cattle' not found in juju's bootstrap-config.yaml error message. Is there something I need to do between maas and juju prior to setting this up in conjure-up?23:34
=== kina__ is now known as kina_
=== mramm_ is now known as mramm
=== iatrou_ is now known as iatrou
=== Dmitrii-Sh_ is now known as Dmitrii-Sh
=== petevg_ is now known as petevg
=== mpontillo_ is now known as mpontillo
=== chigang__ is now known as chigang
=== wolsen_ is now known as wolsen
=== plars_ is now known as plars
=== cargonza_ is now known as cargonza
cory_fucatbus1: I believe those errors are due to not having had run any juju commands before.  I thought conjure-up was updated to resolve that; can you verify if conjure-up is at the latest version?23:37
cory_fucatbus1: Otherwise, you should be able to run any juju command, such as: juju status23:37
cory_fumagicalt1out: Sorry I couldn't be of more help.  I really do have to EOD now23:38
magicalt1outyeah no worries23:38
catbus1cory_fu: I have the latest conjure-up in 16.04: 2.1.0-0~201701041302~ubuntu16.04.123:39
catbus1cory_fu: I don't have any juju controller yet.23:40
catbus1I assume a juju controller will be created afterwards. conjure-up bootstraps juju.23:41
cory_fucatbus1: Yes, that's fine.  It's just the bootstrap logic that needs to run.  The relevant issue for conjure-up is https://github.com/conjure-up/conjure-up/issues/64123:42
cory_fuIt seems that it has not been fixed yet after all23:42
magicalt1outcory_fu: your support services are failing this evening! ;)23:43
cory_fumagicalt1out: Indeed.  Happens when I try to support things I haven't worked on.  ;)23:43
magicalt1outhehe23:44
magicalt1outi blame your colleagues for going to bed....23:44
magicalt1outand certainly not the fact i'm trying to stand up a demo 12 hours before my presentation....23:44
cory_fumagicalt1out: Me too23:44
magicalt1outthanks... that means a lot!23:46
magicalt1outi also blame kwmonroe23:46
cory_fucatbus1: I'm afraid I do have to head out for the evening.  I'm not sure what timezone you're in, but if my suggestion of running `juju status` followed by re-running conjure-up doesn't work, I would have to direct you to stokachu tomorrow during the day according to EST23:46
cory_fumagicalt1out: That goes without saying, I'm sure.  ;)  But for reals, heading out.23:47
cory_fuo/23:47
kwmonroemagicalt1out: does pushing to one of your other groups get you un-stuck for the preso tomorrow?23:47
magicalt1outyeah kwmonroe its not a blocker23:48
magicalt1outjust a bit shit :)23:48
catbus1cory_fu: thanks, I will ping stokachu tomorrow23:48
kwmonroetotally agreed magicalt1out23:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!