rmcadamsjuju makes me want to bang my head on the desk sometimes :)04:31
=== mup_ is now known as mup
kjackalGood morning Juju world!07:40
=== CyberJacob is now known as zz_CyberJacob
spaokheya, is there a way to force juju 1.25 to use a particular controller?09:05
=== zz_CyberJacob is now known as CyberJacob
=== jamespag` is now known as jamespage
SpauldingHi folks!14:27
Spauldinghttps://gist.github.com/pananormalny/11144709446aa80e956d7977df3d88d1 anyone know how to run it just once? @only_once seems to be not working here...14:27
magicaltroutonly_once is really a thing?!14:28
Spauldingalso i have no idea what is the best way to check if db is already filled, should I check that some tables exists or there is some other fancy way of doing this14:28
magicaltroutso it is14:28
magicaltroutI don't know how fixed it is Spaulding but a issue here says:14:29
magicaltrout"Another consideration that needs to be documented because it is very non-obvious is that, because of an implementation detail, @only_once must be the innermost decorator, if combined with @when, etc."14:29
magicaltroutyours is the outer most14:29
Spauldinglet's try it then :)14:30
magicaltrouti know... my google-fu is amazing on a Friday14:30
Spauldingfriday the 13th14:31
magicaltrouti didn't even notice14:31
Spauldingso now you know ;)14:31
Spauldingyour magic abilities works even today!14:31
BlackDexhello there15:01
BlackDexi used juju with maas and i created several network aliasses on one vlan/interface15:01
BlackDexmaas deploy's it correctly15:02
BlackDexi can access the servers on every network15:02
BlackDexbut when i let juju generate lxd's on these hosts, they only receive one ip15:03
Zichi here, I'm (always) testing canonical-kubernetes bundle charms, and I test to shut an etcd member to see if the 2 others deployed by default will be hurted15:08
Zicwhen I shut the etcd/1 or etcd/2 it's fine15:08
Zicbut if I shut etcd/0, the cluster go unhealthy15:08
magicaltroutlazyPower: one for you -^15:08
lazyPowerZic which versin of the etcd charm?15:08
Zichave you any idea about this issue?15:08
Ziccharm: cs:~containers/etcd-2115:09
lazyPowerI have done disaster testing where we tear down nodes, and i haven't encountered this behavior.  Is the broken deploy still around? Can you collect the logs from one of the etcd nodes thats failing after you've destroyed etcd/0?15:11
Zicyep, I collected it, it just try to relink to the etcd/0 in loop (nothing unexpected) : http://paste.ubuntu.com/23792623/15:12
ZicI have the same type of message if I shut etcd/1 or etcd/2, but the cluster stay healthy and Kubernetes cluster is working15:13
Zicbut if I shut etcd/0, all kubectl or even the kubernetes-dashboard is dead, with a "etcd cluster unhealthy or misconfigured" error message15:13
lazyPoweroh Zic  i know what happened here. it looks like the etcd cluster lost quorem and didn't re-elect a new master because it couldn't reach consensus via the 2 nodes that were still active, and not acting as coordinator15:14
ZiclazyPower: oh, it seems logic, but how can I prevent that ?15:16
lazyPowerZic juju add-unit etcd -n 2  -- that will bring the total number of etcd units to 5, and you should be able to test the disaster recovery at that point15:16
Zicnote that I put etcd on the same machine that's running kubernetes-master charm, instead of the default model where etcd is running alone in a machine15:17
Zicdon't know if it's a safe way15:17
lazyPowerZic thats not recommended for production deployments :)15:17
magicaltroutetcd needs to man up and make a decision! ;)15:17
lazyPoweras etcd is the core database tracking the state of your cluster.15:17
Zicyeah, but I scale 3 kubernetes-master15:17
Zic(and test kube-api-loadbalancer ;))15:18
lazyPowerbut its fine to colocate etcd for testing purposes15:18
Ziceven if I run through 3 kubernetes-master, do you recommend to put etcd separately ?15:18
Zicto put it correctly: the default model deploy 1 kubernetes-master, 1 etcd, 1 etcd, 1 etcd15:19
ZicI add two more kubernetes-master, and place etcd charms alongside15:19
Zicis it a poor design choice for production cluster? :|15:19
lazyPoweri think thats fine15:22
lazyPowerextend the etcd cluster onto those principal units, 3 independent, 2 shared15:22
lazyPowerwhere 2 is N really15:22
ZiclazyPower: thanks, I will try that15:23
Zicbonus question: is kube-api-loadbalancer really experimental? I hesitate to run it for production environment, bus as I lurked in what it is doing, it's just an automatically configured nginx which act as reverse-proxy15:24
lazyPowerZic we've found some interesting behavior with the proxy, there's still some tuning to do before we can call it GA15:26
lazyPower it works well, and needs no further tuning in 90% of the scenarios in which its deployed15:26
lazyPowerbut if you're using addons like HELM, we've found that it doesn't work when routed through the LB15:27
ZiclazyPower: 5 etcd units in total permits what fault-tolerance? 2 nodes down as they stay 3 to make a quorum decision?15:43
lazyPowerZic https://coreos.com/etcd/docs/latest/v2/admin_guide.html15:44
lazyPowersee the table under Optimal Cluster Size15:44
Zicoh yeah, I found this link yesterday and forgot it, sorry :)15:44
lazyPowerZic no worries :) I just find it useful to spot check my own knowledge against their upstream docs15:45
lazyPowerit keeps me honest15:45
magicaltroutnooo lazyPower make him pay!!!!15:45
magicaltroutmake him paaaaaaaay!15:45
magicaltroutsorry, friday.15:45
lazyPowerand according to magicaltrout you owe us pizza and beer... because we posted links.15:45
lazyPowerand he pinged me15:45
* lazyPower pokes magicaltrout 15:45
magicaltroutmarcoceppi_: i should be in DC in the last week in March15:45
magicaltroutconsider yourself warned15:46
magicaltroutconference season hasn't even started and i'm already juggling trips15:46
Zictrue! come in France, but I must warn you, we don't have good pizzas (and our 'good' bear is tolen from Belgian neighbors)15:46
magicaltrouti was in Paris last year, that was fun15:47
lazyPowerZic i'll gladly take your belgian brews15:49
magicaltroutwise lazyPower you need to get practicing ;)15:49
lazyPowerpracticing what exactly?15:50
magicaltroutdrinking 15% beer for Ghent ;)15:50
lazyPowerdear lord15:50
lazyPoweri am not ready15:50
lazyPowermagicaltrout you have lofty goals ;)15:50
magicaltroutits always achievable... just requires dedication and practice... and the distinct possiblity of doing talks whilst still drunk15:51
lazyPowerThat would clearly follow in the footsteps of my hero Jamie Windsor15:51
lazyPoweri think it was chef conf 2012, where he took the stage blitzed and tried to run a live siri-controlled infra deploy and it failed fantastically. RIP that demo, but it certainly was memorable15:52
Zichehe, here we use Puppet generally... but for some new projects (like K8S) we're testing Juju :)15:53
ZiclazyPower: if I don't care about ressources, do you advice me to split up etcd from machines which already host kubernetes-master? or staying with 3 kubernetes-master+etcd and 2 others etc separately is also fine? (more question more beers, I know :()15:56
lazyPoweri do, i think its the best strategy to keep them out of the kubernetes units themselves for instances of scale15:56
lazyPowerif you colocate etcd with kubernetes-master-3,4,5 for example15:57
lazyPowerand your cluster goes dormant, and you remove master-4,515:57
lazyPoweryou'll still have tainted machines running with those etcd units15:57
lazyPowerPlus, in juju land, when you colocate services (we call this  hulk-smashing), it *can sometimes* have unintended behavior, like say we add an addon that requires an isolated etcd container to be running... (i'm looking @ you CNI networking providers)15:58
lazyPowerif you've got a colocated etcd unit on that node, you'll hit port collision15:58
Zicok, good advices, I will apply them :)15:58
lazyPowerand in some cases, it will cause the addon workload to fail15:58
spaokso, my juju is broken badly, mongo won't cluster and keeps restarting, I can't issue ensure-availability, I tried restoring a lxc snapshot but the last two didn't help, before I try something drastic, like restoring the first snapshot (removing the newer to do so) anyone have any ideas how I might recover?18:40
lazyPowerspaok have you been able to collect the logs from your container? which juju version?19:25
spaokshould be 1.2519:25
spaokits an older install we are trying to decomm, but juju broke so it making that harder19:26
lazyPoweryeah, thats a rough situation. sorry spaok i'm not certain how to resolve mongo clustering issues but i would lead with a bug with those logs so we can attempt to triage, but it sound slike you're on a time table19:26
=== Guest87783 is now known as med_
=== med_ is now known as medberry
magicaltroutfriday is here \o/ gin o'clock21:25
magicaltroutthis ones for you kwmonroe https://youtu.be/u6mJMJzDD-M?t=1m13s21:26
kwmonroeno way i'm clicking that21:26
magicaltroutits music! :P21:26
magicaltroutit popped up on spotify on the way home and made me think of you21:26
kwmonroeugh.. fine.  i'll click it.21:27
kwmonroeha!  i like it.  you're back on my nice list magicaltrout!21:28
magicaltroutthanks :P21:28
elbalaaspaok: did you try sshing into the mongo machine and looking at the logs?21:41
spaokelbalaa: ya there was some corruption or something, mongo would try to cluster and then restart23:39
spaokwe were able to finally find a mix between a previous snapshot and turning off the repl23:39
spaokthen recovering from there23:39
spaokso it's up enough that we can finish the decomm process23:40
elbalaaspaok:  "corruption or something" story of my life when working with mongo23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!