[00:28] how do i unset no-proxy env variable in juju 2.2? this was not set in 2.1 - juju model-config no-proxy="" give error ERROR no-proxy: expected string, got nothing [00:33] jac_cplane: juju model-config --reset no-proxy === tvansteenburgh1 is now known as tvansteenburg === tvansteenburg is now known as tvansteenburgh [00:33] no --reset will set it to the default value - which is set to localhost [00:34] there is no --unset command === jac__ is now known as jac_cplane === rogpeppe2 is now known as rogpeppe [08:36] good morning. i have a hook failed: "config-changed" error on a unit. someone did something but i don't know what. is there a sort of log of juju action on a units? === frankban|afk is now known as frankban [11:20] Hey, i do have a question about juju. [11:22] Bringe: Fire away. [11:23] What is the best practice if a machine with active services breaks down, how am i supposed to replace that? I'd like to "redeploy" it without having to build a "new" machine and adding every service to that new machine. [11:23] Is there a way to do that? [11:25] In a private cloud with maas. [11:43] Bringe: so you'd add the new machine and then run add-unit to the applications using --to to specify where to place it? [11:44] Bringe: not sure on how it was setup to start with [11:44] Bringe: but that way relations/config/etc are kept and you're just adding a new unit for things [11:46] I agree, i can do that. Is there no more automated way of saying "put services that were on machine X onto machine Y"? :( [11:48] juju bootstrap fails Error: ld.so:object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded , Running module ntp failed [11:48] Bringe: no, I suppose that's something someone could play with as a plugin using libjuju to script out a search of what's there andrecreate [11:48] any idea? [11:50] digv: wow, that's a new one for me. Doing a quick look [11:50] digv: what cloud/series is this? [11:50] MAAS [11:51] digv: do the machines come up cleanly if you just start them from MAAS? [11:51] digv: what OS are they in MAAS? trusty/xenial/? [11:51] Well thanks for the information, rick_h! [11:52] Bringe: sorry, always wish I could say "yea we can do that" but sometimes, not there yet [11:52] yes.. it is clean setup with ubuntu xenial [11:53] digv: We encountered a similiar problem, try forcing the kernel to be at least at xenial (hwe-16.04), that fixed it for us. [11:54] rick_h: No problem, we will find a solution. [11:54] Bringe: ok.. Let me try it quickly [12:12] Hi there! I'm also having some problem bootstrapping juju with MaaS. It stops at: [12:12] 13:50:14 INFO juju.juju api.go:67 connecting to API addresses: [10.0.100.47:17070] [12:12] 14:00:12 DEBUG juju.api apiclient.go:837 error dialing websocket: Forbidden [12:12] Bringe: It's working.. thanks a lot :) [12:49] Is it possible to define multiple "no-proxy" addresses as: [12:49] juju bootstrap machine --config http-prox=http... --config no-proxy=192.168.1.2,10.0.0.2 [12:54] At the node bootstrapped by juju, I can run "nc -vz 10.0.0.51 17070" and it works, but juju still gets stuck on that step [12:57] If I make curl to that address I get some binary stuff, so the http-proxy stuff seems right, but the API is still not responding [12:58] wpk: poke [12:59] cnf: pokeback [12:59] \o/ [12:59] i'm done with meetings for today [12:59] back to fixing the juju thingies [12:59] yay! [12:59] Could you start 'from scratch'? [13:00] kill this machine, then add it, and step by step see what happens? [13:00] i can juju destroy-model, and juju deploy, if you want [13:01] it'll take a while, booting HP machines isn't fast [13:01] wpk: will that work, or did you want something more specific? [13:01] destroy-model, add-model, model-config logging-config "=DEBUG", deploy [13:01] that should be enough [13:01] ok [13:02] Waiting on model to be removed, 16 machine(s), 16 application(s)... [13:03] ERROR expected "key=value", got "logging-config" ? [13:04] juju model-config logging-config="=DEBUG" works [13:08] Ah, sorry, right [13:15] hmz, juju is being weird [13:19] wpk: how do i force remove a model from juju? [13:19] what do you mean by 'weird'? [13:20] MaaS shows all machines as ready [13:20] juju says it can't release the machines [13:21] i'm stuck at Waiting on model to be removed, 4 machine(s)... [13:22] oh, seems it timed out, and did it anyway [13:22] o,O [13:24] ok, now we wait half an hour [13:37] wpk: machines are up, juju is doing its thing [13:44] cnf: cool [14:13] wpk: it's almost there [14:13] wpk: what's the next step? [14:16] cnf: what's the current status? [14:16] is the machine dead as it was previously? [14:16] well, network-wise dead [14:16] yep [14:16] on that vlan, anyway [14:16] could you copy /var/log/juju/* from it somewhere? [14:17] https://bpaste.net/show/21ea96526ca7 [14:17] do you need unit-neutron-gateway-0.log and unit-ntp-3.log as well? [14:20] cnf: could you also provide output of grep "" /etc/network/interfaces/* and ip a ? [14:20] you mean /etc/network/interfaces.d/* ? [14:20] /etc/network/interfaces* [14:21] all files there, new and backups [14:21] ok [14:21] you had a / there that made me wonder :P [14:21] typo :) [14:23] wpk: https://bpaste.net/show/c2b08cb540ab [14:23] i still don't get where it gets 172.20.19.248 from, it should not have _antyhing_ in that subnet [14:23] and ip a? [14:24] and ip r [14:24] oh, https://bpaste.net/show/55a4b7746004 [14:24] i forgot to link it :P [14:25] ok, so I don't see bond0.4013 there [14:25] I see br-bond0.4013 but no bond0.4013 [14:25] wpk: https://bpaste.net/show/4fe735871db0 [14:25] wpk: where? [14:26] item 30 in ip a [14:26] ah, sorry, yues, it's there [14:26] but why master ovs-system? [14:27] could you also do brctl show ? [14:27] brctl show br-bond0.4013 ? [14:28] https://bpaste.net/show/ecb674077e42 [14:29] brctl show ovs-system ? [14:29] where is this getting from, it's not in the e/n/i anywhere [14:30] bridge name bridge id STP enabled interfaces [14:30] ovs-system can't get info Operation not supported [14:31] wpk: i'm betting that's a neutron thing [14:31] hm, it might be. and it's eating bond0.4013 [14:32] hmm [14:32] cnf: did it worked on 2.1? [14:33] yes, but now that you point to that, some other details changed [14:33] let me try some shizzle [14:33] i need to destroy everything though [14:33] anything you'd like me to check before I do? [14:34] cnf: copy of syslog and unit-neutron-gateway-0.log [14:34] cnf: and then do your thing [14:38] k, deploying again [14:38] it'll take half an hour :P [14:59] ugh, now juju isn't asking for all the machines it needs o,O [15:01] why is it not asking for more machines? o,O [15:04] wpk: any reason juju isn't asking MaaS for all its machines? [15:05] whaa? [15:05] i have 4 machines configured in my model [15:05] juju is only asking for 1 from maas [15:06] i'm watching the maas logs, it's just not getting any other requests [15:06] i can't juju retry-provisioning 0 [15:06] because machine 0 isn't in an error state [15:07] i see in MaaS [15:07] Jul 7 16:58:50 MAAS maas.api: [info] Request from user juju to acquire a machine with constraints: [('tags', ['compute', 'openstack']), ('interfaces', ['0:space=5;1:space=1;2:space=2']), ('agent_name', ['a702d342-3c94-4ff5-8548-c296fb5d2dae']), ('zone', ['default'])] [15:07] which is right [15:08] but no requests for the other 3 machines [15:09] So the machine is in proper state in juju but not it's not provisioned? [15:09] (brb) [15:11] wpk: https://bpaste.net/show/77d220e2a785 [15:11] 3 machines in "pending" [15:11] nothing happening [15:13] now the 1 machine is deployed [15:13] the other 3 are not moving [15:15] wpk: so i just ran the same juju deploy openstack.yml again [15:15] now it's getting the other 3 machines [15:15] O,o [15:20] Hi there! [15:20] I'm having a weird problem while bootstrapping [15:20] the process stops waiting for the websocket, at: [15:20] INFO juju.juju api.go:67 connecting to API addresses: [10.0.0.52:17070] [15:21] and then... DEBUG juju.api apiclient.go:837 error dialing websocket: Forbidden [15:22] I am behind a http-proxy, but I the no_proxy and no-proxy variables are defined and from the shell I can access that IP:port with curl [15:23] how can it be that curl can connect but juju not? [15:25] well, curl isn't connecting to the websocket [15:25] really? ... but after the request, it gets some stuff... binary, but something comes out [15:28] the error says "error dialing websocket: Forbidden", but I do see the IP and the port, and the proxy is not blocking it... [15:32] cnf: from my limited knowledge, I got the picture that a websocket is a port listening to http requests, right? [15:32] it's listening to websocket requests [15:32] so, curl http://10.0.100.52:17070 is equivalent to what juju is trying to do [15:33] on the same port as your http request, generally [15:34] websockets do an upgrade on a http request, and then become a websocket [15:34] your curl is just getting the html/css/js files [15:35] aha! [15:35] so, my error might have to do with some settings within the API [15:35] it might be you not being authenticated [15:35] or a proxy not allowing it [15:36] hmm... that sounds possible... [15:37] but, I don't think our http-proxy has such a complex settings, to allow traffic but drop authentication [15:38] is there any way I could check that? [15:39] not authentication [15:39] just not allowing websockets [15:39] you send a websocket upgrade through the http proxy [15:39] it responds with "nope!" [15:40] that _might_ be one of the reasons [15:40] not saying that is the reason [15:40] ... and I probably can not check that with some curl request or something, right? [15:41] well, use your browser, and enable webdev / debug tools [15:41] you should be able to see all requests, and the responses [15:41] generally under network you can see the headers sent and received [15:46] well.. the browser seems to work. I get the headers... [15:47] the response is just a couple of red dots, I don't know what that means [15:55] cnf: I have an openstack with nova with same settings and it works fully, so I guess I can discard the option of the proxy blocking the websockets, right? [16:15] wpk: prod [16:16] cnf: ? [16:16] wpk: shizzle works, i'm going home :P [16:17] cnf: could you explain what have you done? [16:17] cnf: for future generations :) [16:18] wpk: changed the data-port: in my model [16:24] wpk: so i shuffled some machines around [16:24] and my data-port was set to the wrong interface [16:25] i still don't know where it gets the 172.20.19 network [16:25] or why it wasn't aquireing the machines [16:25] but it was ovs claiming an interface that was already provisioned (which i think it should not be allowed to, tbh) [16:28] wpk: so i guess it was me being an idiot [16:28] anyway, i'm going home [16:28] wpk: thanks a lot for your help! and have a nice WE! === frankban is now known as frankban|afk [18:40] o/ juju world' === scuttle|afk is now known as scuttlemonkey === scuttlemonkey is now known as scuttle|afk