[00:28] <jac_cplane> how do i unset no-proxy env variable in juju 2.2?  this was not set in 2.1   - juju model-config no-proxy="" give error ERROR no-proxy: expected string, got nothing
[00:33] <tvansteenburgh1> jac_cplane: juju model-config --reset no-proxy
[00:33] <jac_cplane> no --reset will set it to the default value - which is set to localhost
[00:34] <jac_cplane> there is no --unset command
[08:36] <Hetfield> good morning. i have a hook failed: "config-changed" error on a unit. someone did something but i don't know what. is there a sort of log of juju action on a units?
[11:20] <Bringe> Hey, i do have a question about juju.
[11:22] <zeestrat> Bringe: Fire away.
[11:23] <Bringe> What is the best practice if a machine with active services breaks down, how am i supposed to replace that? I'd like to "redeploy" it without having to build a "new" machine and adding every service to that new machine.
[11:23] <Bringe> Is there a way to do that?
[11:25] <Bringe> In a private cloud with maas.
[11:43] <rick_h> Bringe: so you'd add the new machine and then run add-unit to the applications using --to to specify where to place it?
[11:44] <rick_h> Bringe: not sure on how it was setup to start with
[11:44] <rick_h> Bringe: but that way relations/config/etc are kept and you're just adding a new unit for things
[11:46] <Bringe> I agree, i can do that. Is there no more automated way of saying "put services that were on machine X onto machine Y"? :(
[11:48] <digv> juju bootstrap fails Error: ld.so:object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded , Running module ntp failed
[11:48] <rick_h> Bringe: no, I suppose that's something someone could play with as a plugin using libjuju to script out a search of what's there andrecreate
[11:48] <digv> any idea?
[11:50] <rick_h> digv: wow, that's a new one for me. Doing a quick look
[11:50] <rick_h> digv: what cloud/series is this?
[11:50] <digv> MAAS
[11:51] <rick_h> digv: do the machines come up cleanly if you just start them from MAAS?
[11:51] <rick_h> digv: what OS are they in MAAS? trusty/xenial/?
[11:51] <Bringe> Well thanks for the information, rick_h!
[11:52] <rick_h> Bringe: sorry, always wish I could say "yea we can do that" but sometimes, not there yet
[11:52] <digv> yes.. it is clean setup with ubuntu xenial
[11:53] <Bringe> digv: We encountered a similiar problem, try forcing the kernel to be at least at xenial (hwe-16.04), that fixed it for us.
[11:54] <Bringe> rick_h: No problem, we will find a solution.
[11:54] <digv> Bringe: ok.. Let me try it quickly
[12:12] <julen> Hi there! I'm also having some problem bootstrapping juju with MaaS. It stops at:
[12:12] <julen> 13:50:14 INFO  juju.juju api.go:67 connecting to API addresses: [10.0.100.47:17070]
[12:12] <julen> 14:00:12 DEBUG juju.api apiclient.go:837 error dialing websocket: Forbidden
[12:12] <digv> Bringe: It's working.. thanks a lot :)
[12:49] <julen> Is it possible to define multiple "no-proxy" addresses as:
[12:49] <julen> juju bootstrap machine --config http-prox=http...  --config no-proxy=192.168.1.2,10.0.0.2
[12:54] <julen> At the node bootstrapped by juju, I can run "nc -vz 10.0.0.51 17070"  and it works, but juju still gets stuck on that step
[12:57] <julen> If I make curl to that address I get some binary stuff, so the http-proxy stuff seems right, but the API is still not responding
[12:58] <cnf> wpk: poke
[12:59] <wpk> cnf: pokeback
[12:59] <cnf> \o/
[12:59] <cnf> i'm done with meetings for today
[12:59] <cnf> back to fixing the juju thingies
[12:59] <wpk> yay!
[12:59] <wpk> Could you start 'from scratch'?
[13:00] <wpk> kill this machine, then add it, and step by step see what happens?
[13:00] <cnf> i can juju destroy-model, and juju deploy, if you want
[13:01] <cnf> it'll take a while, booting HP machines isn't fast
[13:01] <cnf> wpk: will that work, or did you want something more specific?
[13:01] <wpk> destroy-model, add-model, model-config logging-config "<root>=DEBUG", deploy
[13:01] <wpk> that should be enough
[13:01] <cnf> ok
[13:02] <cnf> Waiting on model to be removed, 16 machine(s), 16 application(s)...
[13:03] <cnf> ERROR expected "key=value", got "logging-config" ?
[13:04] <cnf> juju model-config logging-config="<root>=DEBUG" works
[13:08] <wpk> Ah, sorry, right
[13:15] <cnf> hmz, juju is being weird
[13:19] <cnf> wpk: how do i force remove a model from juju?
[13:19] <wpk> what do you mean by 'weird'?
[13:20] <cnf> MaaS shows all machines as ready
[13:20] <cnf> juju says it can't release the machines
[13:21] <cnf> i'm stuck at Waiting on model to be removed, 4 machine(s)...
[13:22] <cnf> oh, seems it timed out, and did it anyway
[13:22] <cnf> o,O
[13:24] <cnf> ok, now we wait half an hour
[13:37] <cnf> wpk: machines are up, juju is doing its thing
[13:44] <wpk> cnf: cool
[14:13] <cnf> wpk: it's almost there
[14:13] <cnf> wpk: what's the next step?
[14:16] <wpk> cnf: what's the current status?
[14:16] <wpk> is the machine dead as it was previously?
[14:16] <wpk> well, network-wise dead
[14:16] <cnf> yep
[14:16] <cnf> on that vlan, anyway
[14:16] <wpk> could you copy /var/log/juju/* from it somewhere?
[14:17] <cnf> https://bpaste.net/show/21ea96526ca7
[14:17] <cnf> do you need unit-neutron-gateway-0.log and unit-ntp-3.log as well?
[14:20] <wpk> cnf: could you also provide output of grep "" /etc/network/interfaces/* and ip a ?
[14:20] <cnf> you mean /etc/network/interfaces.d/* ?
[14:20] <wpk> /etc/network/interfaces*
[14:21] <wpk> all files there, new and backups
[14:21] <cnf> ok
[14:21] <cnf> you had a / there that made me wonder :P
[14:21] <wpk> typo :)
[14:23] <cnf> wpk: https://bpaste.net/show/c2b08cb540ab
[14:23] <cnf> i still don't get where it gets 172.20.19.248 from, it should not have _antyhing_ in that subnet
[14:23] <wpk> and ip a?
[14:24] <wpk> and ip r
[14:24] <cnf> oh, https://bpaste.net/show/55a4b7746004
[14:24] <cnf> i forgot to link it :P
[14:25] <wpk> ok, so I don't see bond0.4013 there
[14:25] <wpk> I see br-bond0.4013 but no bond0.4013
[14:25] <cnf> wpk: https://bpaste.net/show/4fe735871db0
[14:25] <cnf> wpk: where?
[14:26] <cnf> item 30 in ip a
[14:26] <wpk> ah, sorry, yues, it's there
[14:26] <wpk> but why master ovs-system?
[14:27] <wpk> could you also do brctl show ?
[14:27] <cnf> brctl show br-bond0.4013 ?
[14:28] <cnf> https://bpaste.net/show/ecb674077e42
[14:29] <wpk> brctl show ovs-system ?
[14:29] <wpk> where is this getting from, it's not in the e/n/i anywhere
[14:30] <cnf> bridge name	bridge id		STP enabled	interfaces
[14:30] <cnf> ovs-system		can't get info Operation not supported
[14:31] <cnf> wpk: i'm betting that's a neutron thing
[14:31] <wpk> hm, it might be. and it's eating bond0.4013
[14:32] <cnf> hmm
[14:32] <wpk> cnf: did it worked on 2.1?
[14:33] <cnf> yes, but now that you point to that, some other details changed
[14:33] <cnf> let me try some shizzle
[14:33] <cnf> i need to destroy everything though
[14:33] <cnf> anything you'd like me to check before I do?
[14:34] <wpk> cnf: copy of syslog and unit-neutron-gateway-0.log
[14:34] <wpk> cnf: and then do your thing
[14:38] <cnf> k, deploying again
[14:38] <cnf> it'll take half an hour :P
[14:59] <cnf> ugh, now juju isn't asking for all the machines it needs o,O
[15:01] <cnf> why is it not asking for more machines? o,O
[15:04] <cnf> wpk: any reason juju isn't asking MaaS for all its machines?
[15:05] <wpk> whaa?
[15:05] <cnf> i have 4 machines configured in my model
[15:05] <cnf> juju is only asking for 1 from maas
[15:06] <cnf> i'm watching the maas logs, it's just not getting any other requests
[15:06] <cnf> i can't juju retry-provisioning 0
[15:06] <cnf> because machine 0 isn't in an error state
[15:07] <cnf> i see in MaaS
[15:07] <cnf> Jul  7 16:58:50 MAAS maas.api: [info] Request from user juju to acquire a machine with constraints: [('tags', ['compute', 'openstack']), ('interfaces', ['0:space=5;1:space=1;2:space=2']), ('agent_name', ['a702d342-3c94-4ff5-8548-c296fb5d2dae']), ('zone', ['default'])]
[15:07] <cnf> which is right
[15:08] <cnf> but no requests for the other 3 machines
[15:09] <wpk> So the machine is in proper state in juju but not it's not provisioned?
[15:09] <wpk> (brb)
[15:11] <cnf> wpk: https://bpaste.net/show/77d220e2a785
[15:11] <cnf> 3 machines in "pending"
[15:11] <cnf> nothing happening
[15:13] <cnf> now the 1 machine is deployed
[15:13] <cnf> the other 3 are not moving
[15:15] <cnf> wpk: so i just ran the same juju deploy openstack.yml again
[15:15] <cnf> now it's getting the other 3 machines
[15:15] <cnf> O,o
[15:20] <julen> Hi there!
[15:20] <julen> I'm having a weird problem while bootstrapping
[15:20] <julen> the process stops waiting for the websocket, at:
[15:20] <julen> INFO  juju.juju api.go:67 connecting to API addresses: [10.0.0.52:17070]
[15:21] <julen> and then... DEBUG juju.api apiclient.go:837 error dialing websocket: Forbidden
[15:22] <julen> I am behind a http-proxy, but I the no_proxy and no-proxy variables are defined and from the shell I can access that IP:port with curl
[15:23] <julen> how can it be that curl can connect but juju not?
[15:25] <cnf> well, curl isn't connecting to the websocket
[15:25] <julen> really? ... but after the request, it gets some stuff... binary, but something comes out
[15:28] <julen> the error says "error dialing websocket: Forbidden", but I do see the IP and the port, and the proxy is not blocking it...
[15:32] <julen> cnf: from my limited knowledge, I got the picture that a websocket is a port listening to http requests, right?
[15:32] <cnf> it's listening to websocket requests
[15:32] <julen> so, curl http://10.0.100.52:17070  is equivalent to what juju is trying to do
[15:33] <cnf> on the same port as your http request, generally
[15:34] <cnf> websockets do an upgrade on a http request, and then become a websocket
[15:34] <cnf> your curl is just getting the html/css/js files
[15:35] <julen> aha!
[15:35] <julen> so, my error might have to do with some settings within the API
[15:35] <cnf> it might be you not being authenticated
[15:35] <cnf> or a proxy not allowing it
[15:36] <julen> hmm... that sounds possible...
[15:37] <julen> but, I don't think our http-proxy has such a complex settings, to allow traffic but drop authentication
[15:38] <julen> is there any way I could check that?
[15:39] <cnf> not authentication
[15:39] <cnf> just not allowing websockets
[15:39] <cnf> you send a websocket upgrade through the http proxy
[15:39] <cnf> it responds with "nope!"
[15:40] <cnf> that _might_ be one of the reasons
[15:40] <cnf> not saying that is the reason
[15:40] <julen> ... and I probably can not check that with some curl request or something, right?
[15:41] <cnf> well, use your browser, and enable webdev / debug tools
[15:41] <cnf> you should be able to see all requests, and the responses
[15:41] <cnf> generally under network you can see the headers sent and received
[15:46] <julen> well.. the browser seems to work. I get the headers...
[15:47] <julen> the response  is just a couple of red dots, I don't know what that means
[15:55] <julen> cnf: I have an openstack with nova with same settings and it works fully, so I guess I can discard the option of the proxy blocking the websockets, right?
[16:15] <cnf> wpk: prod
[16:16] <wpk> cnf: ?
[16:16] <cnf> wpk: shizzle works, i'm going home :P
[16:17] <wpk> cnf: could you explain what have you done?
[16:17] <wpk> cnf: for future generations :)
[16:18] <cnf> wpk: changed the data-port: in my model
[16:24] <cnf> wpk: so i shuffled some machines around
[16:24] <cnf> and my data-port was set to the wrong interface
[16:25] <cnf> i still don't know where it gets the 172.20.19 network
[16:25] <cnf> or why it wasn't aquireing the machines
[16:25] <cnf> but it was ovs claiming an interface that was already provisioned (which i think it should not be allowed to, tbh)
[16:28] <cnf> wpk: so i guess it was me being an idiot
[16:28] <cnf> anyway, i'm going home
[16:28] <cnf> wpk: thanks a lot for your help! and have a nice WE!
[18:40] <Budgie^Smore> o/ juju world'