[08:54] <cnf> morning
[08:57] <cnf> so anyone around that can help me figure out why juju isn't putting ip's in my containers even though i have the correct constraint on the services?
[08:58] <wpk> What's the setup?
[09:00] <cnf> i used to do it fine, it stopped doing it when i upgraded to the latest version
[09:00] <cnf> wpk: running on MaaS
[09:01] <wgrant> cnf: It's not adding the extra interfaces to the containers?
[09:01] <cnf> wgrant: right
[09:02] <wgrant> cnf: Juju 2.2.0? That's fixed in 2.2.1.
[09:02] <cnf> wgrant: it _is_ getting ip's from maas...
[09:02] <cnf> oh
[09:02] <cnf> so i need to upgrade again
[09:02] <cnf> can you link the bug?
[09:02] <wgrant> cnf: https://bugs.launchpad.net/juju/+bug/1698443
[09:02] <mup> Bug #1698443: Juju-2.2 does not create interfaces in an LXD for all spaces <4010> <canonical-bootstack> <containers> <cpe> <cpe-sa> <eda> <maas-provider> <network> <juju:Fix Committed by wpk> <juju 2.2:Fix Released by wpk> <https://launchpad.net/bugs/1698443>
[09:02] <cnf> thanks
[11:25] <cnf> hmz
[11:28] <cnf> right, so juju messed up my networking, again
[11:28] <cnf> >,<
[11:40] <cnf> so when MaaS configures the machine, it works
[11:40] <cnf> juju takes over, and i can't even make arp work anymore on that vlan
[11:51] <cnf> damn machine stops replying to arp requests o,O
[11:55] <wpk> cnf: do you have console access to that machine? e/n/i would be useful
[11:57] <wpk> or could you set log level to debug, try to add the machine then and look what's in the log
[12:01] <cnf> i do
[12:04] <wpk> cnf: Could you post /etc/network/interfaces and /etc/network/interfaces.(backup...) somewhere?
[12:10] <cnf> wpk: https://bpaste.net/show/59224fa597dc
[12:11] <cnf> i have 2 backups
[12:11] <cnf> and a .new
[12:17] <wpk> cnf: paste everything if you can
[12:18] <cnf> wpk: https://bpaste.net/show/4873574782b0
[12:19] <wpk> and the .new?
[12:21] <wpk> also, what's happening when you try to do ifup -a ?
[12:29] <cnf> https://bpaste.net/show/a13728fa885b
[12:29] <cnf> sorry if i'm a bit slow, helping a coworker install a VM
[12:29] <cnf> ifup -a just gives a new prompt
[12:29] <cnf> no messages
[12:30] <cnf> exit code 0
[12:30] <wpk> yet the machine has no network connectivity?
[12:31] <wpk> I can't see anything wrong with those files, the bridges seem to be configured correctly
[12:37] <cnf> wpk: only the vlan 4013
[12:37] <cnf> the rest works fine
[12:37] <cnf> i can see arp requests coming in
[12:37] <cnf> but no replies going back out
[12:44] <wpk> cnf: could you remove post-up/pre-down lines from bond0.4013 definition (leave them for br-bond0.4013) and then do ifdown -a; ifup -a ?
[12:53] <cnf> ok, i'm back :P
[12:54] <cnf> and if i do a ifdown -a, won't i lose connectivity? PP
[12:54] <cnf> wpk: ^^
[13:00] <cnf> i just rebooted
[13:02] <wpk> I thought you were using a console of some sort, you probably would :)
[13:03] <cnf> no, juju ssh
[13:04] <cnf> it's running on MaaS
[13:06] <cnf> ok, no difference
[13:08] <cnf> also, all interfaces have MTU 9100 or 9000
[13:08] <cnf> except br-bond0.4013
[13:13] <cnf> wait, i just noticed the routes in /e/n/i are wrong
[13:13] <cnf> and also not what is actually set on the system
[13:14] <cnf> wt...
[13:14] <wpk> that is a wtf
[13:14] <wpk> where is it getting them from then?
[13:15] <cnf> i have no idea
[13:15] <cnf> it should not even be touching that  subnet
[13:16] <cnf> apart form that, i can't even ping the gateway on the 4013 vlan
[13:16] <cnf> it works when maas brings it up
[13:17] <cnf> juju takes over, and it stops working
[13:18] <cnf> hmm, and it's not outputting icmp packets, either, on that vlan o,O
[13:21] <cnf> not even when i ping -I
[13:23] <cnf> i'm flabbergasted o,O
[13:27] <wpk> The fact that e/n/i is inconsistent with what's really there is weird for me
[13:30] <cnf> yes
[13:30]  * cnf sighs
[13:30] <cnf> i'm getting a serious juju burnout...
[13:37] <wpk> hm, one more idea - could you disable jujud on this machine 'permanently', reboot it and see if e/n/i is consistent with configured networking?
[13:38] <wpk> systemctl disable jujud-machine-0
[14:07] <wpk> cnf: juju might be messing with it, but we have to rule out that something is broken in Ubuntu itself
[14:30] <julen> Hi there! does someone happen to know the right API endpoint address for MaaS?
[14:30] <julen> in the docu it says that http://$ip:5240/MAAS
[14:31] <julen> but my MaaS controller is not listening on port 5240 for IPv4
[14:31] <julen> I also tried http://$ip/MAAS/api/2.0/  but it's also not working
[14:36] <joedborg> hey everyone
[14:37] <joedborg> is there a "set_staus()" eqv for old bash charms?
[14:40] <admcleod> joedborg: i think its just status-set ?
[14:40] <admcleod> joedborg: https://github.com/juju-solutions/charms.reactive/blob/7322a9bc13ffde1960bcfcb9166e3c09341d27bb/docs/index.rst
[14:53] <joedborg> cheers @admcleod that looks promising
[14:56] <admcleod> joedborg: !
[14:59] <cnf> wpk: rebooting
[15:04] <cnf> wpk: still the same
[15:06] <joedborg> admcleod works perfectly!  thank you sir
[15:06] <admcleod> joedborg: !
[15:10] <admcleod> joedborg: Success rate is 100 percent (2/2), round-trip min/avg/max = 4/6/8 ms
[15:14] <wpk> cnf: and jujud is not running?
[15:14] <cnf> uhm, a lot of jujud things are running
[15:15] <cnf> https://bpaste.net/show/7f8d48b33757
[15:15] <cnf> is that normal?
[15:16] <wpk> cnf: have you disabled jujud-machine-0 before rebooting?
[15:16] <cnf> i did  sudo systemctl disable jujud-machine-0
[15:21] <cnf> hmm
[15:25] <cnf> wpk: was that not enough?
[15:30] <wpk> cnf: hm, it should be...
[15:30] <wpk> cnf: systemctl list-unit-files \*juju\*
[15:30] <cnf> juju-clean-shutdown.service          enabled
[15:30] <cnf> jujud-unit-neutron-gateway-0.service enabled
[15:30] <cnf> jujud-unit-ntp-3.service             enabled
[15:31] <wpk> disable all of those, reboot and see
[15:31] <wpk> hmm
[15:31] <wpk> but before that
[15:31] <wpk> set logging to DEBUG, reboot, and copy logs somewhere (or mail them to me)
[15:32] <cnf> in syslog?
[15:32] <wpk> in /var/log/juju/
[15:32] <cnf> how do i set logging to debug in juju?
[15:32] <cnf> especially as i just disabled all things juju?
[15:33] <wpk> reenable jujud-machine-0
[15:34] <wpk> something really bad is happening there, I'd guess that multiple units in LXCs are trying to start up container networking all at once
[15:34] <cnf> ok, how do i systemd jujud-machine-0 ?
[15:35] <wpk> systemctl start jujud-machine-0
[15:35] <cnf> Failed to start jujud-machine-0.service: Unit jujud-machine-0.service not found.
[15:38] <wpk> juju model-config logging-config="<root>=DEBUG"  should work
[15:39] <cnf> ok
[15:39] <cnf> but now juju is still disabled, right?
[15:40] <wpk> But the units should receive this logging
[15:41] <cnf> but if juju isn't running, how can it log things?
[15:42] <wpk> jujud-unit-* are running and will be logging
[15:43] <cnf> $ systemctl list-unit-files \*juju\*
[15:43] <cnf> UNIT FILE                   STATE
[15:43] <cnf> juju-clean-shutdown.service disabled
[15:43] <cnf> 1 unit files listed.
[15:43] <cnf> nothing is running?
[15:44] <wpk> what about the jujud-unit-ntp3.service and jujud-unit-neutron-gateway-0?
[15:44] <cnf> no, i stopped them all
[15:46] <wpk> Ah, ok. Reboot is a way to go then probably
[15:47] <wpk> I have a suspiction of what might be happening, but having a debug log from the units would be really helpful
[15:47] <wpk> to see -why- it's happening
[15:48] <cnf> i'm rebooting
[15:51] <cnf> ok wpk, what log do you want?
[15:51] <cnf> machine-0 ?
[15:52] <cnf> wpk: https://bpaste.net/show/c60636a36174
[15:58] <cnf> anything in there?
[16:00] <cnf> k, wpk it's time to go home here, can I poke you tomorrow?
[16:00] <cnf> I have meetings till 13:00 CET
[16:11] <wpk> cnf: Please do.
[16:12] <wpk> cnf: The only unusual thing in the logs is "2017-07-06 13:27:25 INFO juju.api.common network.go:118 no addresses observed on interface "bond0.4013""
[16:52] <cnf> hmm