=== StoneTable is now known as aisrael [00:44] Bug #1645477 changed: Problem deploying openstack-ha with juju deploy [00:49] thumper: ping [00:53] Bug #1645477 opened: Problem deploying openstack-ha with juju deploy [01:05] Bug # changed: 1195040, 1268364, 1457575, 1504272, 1528261, 1544796, 1552237, 1571457, 1613992, 1645477, 1645729, 1653888 [01:06] anastasiamac_: hey [01:07] thumper: it looks like memory/disk issues are prevailent on latest 1.25.x too... could it be related to what we are seeing in juju 2? [01:07] thumper: and hence related to work we did at EOY? [01:07] anastasiamac_: all different memory and disk issues [01:07] so much changed between 1.25 and 2.0 [01:07] hazaah [01:08] Bug # opened: 1195040, 1268364, 1457575, 1504272, 1528261, 1544796, 1552237, 1571457, 1613992, 1645729, 1653888 [01:08] anastasiamac_, thumper: We're definitely still seeing memory issues on latest stable versions of both 1.25 and 2.0 [01:08] * thumper sighs... yeah [01:09] we suck [01:09] thumper: we dont! [01:09] but let's just focus on adding features [01:09] it'll be fine [01:11] thumper: did u find perfscaling CI collects helpful in triaging 2.x memory issues? don't we track goroutines there? [01:11] veebers: ^^ [01:12] anastasiamac_: we don't currently track goroutines there [01:12] (but that's a possible feature we could add in the near future) [01:14] Bug # changed: 1195040, 1268364, 1457575, 1504272, 1528261, 1544796, 1552237, 1571457, 1613992, 1645729, 1653888 [01:25] anastasiamac_, thumper, veebers: I'm about to do a check of my envs (1.25.9 & 2.0.2) for memory issues; any info I could provide on bug 1645729 or others that would help? [01:25] Bug #1645729: environment unstable after 1.25.8 upgrade

[01:25] blahdeblah: not right now... I don't think [01:30] blahdeblah: I can't suggest anything useful [01:35] Bug #1650401 changed: Kubernetes-core bundle fails to deploy. Easy-rsa results with failed hook [01:35] Bug #1650405 changed: Juju Embedded - Juju logout/login not working for multiple users connected to same controller [01:37] thumper, veebers: FWIW, I have a 2.0.2 Canonistack controller running 1 env which has up since Dec 27 05:51:53 2016 [01:37] and [01:38] So it seems a lot better on 2.0.2 than 1.25.9 which has been restarted several times since. [01:38] * thumper nods [01:42] anastasiamac_ asked for an update on that bug, so I'll add the above there. [01:43] blahdeblah: thank you! [01:50] * thumper screams [01:50] loudly [01:50] FFS [01:50] * thumper headdesks [01:50] thumper: wot? [01:51] very stupid code [01:51] * thumper blames ian [01:52] good! u can only blame ppl that r not here to defend themselves :D [01:52] thumper: which part of code? [01:52] cmr [01:53] I'm trying to trace the run away state references... [01:53] awesome \o/ should it not have been behind a feature flag? [01:53] 75 machines [01:54] call it average 6 units per machine [01:54] or even three [01:54] that is 300 api connections [01:55] each api connection creates an extra state because stupid [01:55] wow :( [01:55] so... found that [01:55] but it isn't the cause of this leak [01:56] i just... don't even... [02:11] * redir goes eod [03:30] veebers: i have a q about MM functional tests. r u the person to ask? [03:33] anastasiamac_: https://github.com/juju/juju/pull/6789 [03:33] * anastasiamac_ looking [03:35] thumper: Stata? funny :) that alone is an indication of the rush in which this code was produced... [03:36] yeah [03:37] well, done - i think it should b added to Hall of Fame or at leats 'Treasure Hunt 2017' repository... i suspect there will b a few of these in the months to come [03:38] lgtm'ed [03:39] natefinch: ping... r u really here? [03:43] frobware: ping [03:57] anastasiamac_: hey, I probably am. What's the query? :-) [03:58] veebers: was looking at https://bugs.launchpad.net/juju/+bug/1648063 [03:58] Bug #1648063: kill-controller removes machines from migrated model

[03:58] veebers: which looks like it could have a pretty simple functional test... do we have one? [03:58] veebers: i was hoping that we did and could easily verify if the issue has been fixed.. [03:59] veebers: there were couple of MM fixes over the last couple of weeks, it seems [04:03] anastasiamac_: there is no test for this currently, but it's on the books to be added [04:04] veebers: k. thnx [04:09] thumper: funny: https://bugs.launchpad.net/juju/+bug/1519147... somehow I do not thinkg that Dave's on it [04:09] Bug #1519147: worker/rsyslog: data races <2.0-count> [04:09] yeah... probably not [06:22] can anyone tell me from https://jujucharms.com/docs/stable/authors-charm-actions#example-schema [06:22] looking at the example schema [06:22] what is going to be in the files under actions/ [06:22] ie actions/report [06:22] from reading that it looks like everything is defined in actions.yaml [06:22] and then the implementation is handled in the reactive layer [06:31] t0mb0: this looks like a question best suited for #juju channel :) [06:52] anastasiamac_: We've got another occurrence of what looks like 1645729 on juju 2.0.2 - do you want any further info gathered? [06:53] blahdeblah: and i was planning to hold my breath for the next 7mins... [06:53] blahdeblah: yes plz ;) [06:53] What needs gathering? [06:54] Just the standard go routines dump? [06:54] blahdeblah: for now, yes. [06:54] blahdeblah: altho m a little confused, the bug is about 1.25.x.. [06:55] It seems to be assigned to all current versions, including 2.1 [06:55] presumably because axw found the same issue in them [06:55] blahdeblah: ah i see... and u have been sayint that equivalent 2.0.2 is fine.. until 5mins away from eod ;D [06:55] blahdeblah: got it.. [06:56] blahdeblah: thank you for providing more info! [06:56] So on my canonistack env, 2.0.2 has been good so far, but this is another env on the azure provider [06:58] blahdeblah: \o/ loving it! and yes, update on the bug would be awesome [06:58] anastasiamac_: Mind having a 1-minute look over https://wiki.canonical.com/InformationInfrastructure/WebOps/Juju#Gathering_debug_info_with_pprof to make sure I'm gathering the right stuff? [06:59] * anastasiamac_ looking [07:02] blahdeblah: looks awesome \o/ someone who worte it up should get a medal [07:03] OK - gathering now [07:06] TBC, this might be lp:1635311 rather than lp:1645729 - I can't be sure until checking the symptoms [07:07] blahdeblah: ack [08:13] anastasiamac_: pong & HNY [08:13] frobware: HNY to u 2 :D [08:14] frobware: m about to do kids/dinner.. could I pm u later-ish? maybe in 3hrs? [08:14] anastasiamac_: sure [08:14] frobware: \o/ === rogpeppe is now known as rogpeppe1 === rogpeppe1 is now known as rogpeppe [09:48] when using maas as the provider, what is deciding the ip of the machines? for some reason, the machine ip of the node seem to be choosen arbitrary even though im using an external dhcp (not the one maas provides) [09:49] when i first booted up the maas node i got the correct ip from my dhcp server, but when i run "juju add-machine" the node is created with a different ip [09:49] that ip is even in one of the "reserved ranges" that i have stated in the maas configuration, which makes me even more perplexed [11:17] SimonKLB: what is the network configuration of the node? DHCP? Auto Assign? Static Assign? [11:18] SimonKLB: Correction - "what is the network configuration of the node that was allocated from MAAS" when you did juju add-machine [11:51] natefinch: hey, you around? [11:59] natefinch: I'm off to the dentist, I'll be in touch on my return === deanman_ is now known as deanman [12:38] frobware: ah, youre correct, it was set to auto-assign, tried static first and then DHCP which worked, still curious though how auto-assigned resulted in a different ip which was reserved [12:38] frobware: does juju have anything to do with how the ip is chosen, or is it purely maas? [12:38] SimonKLB: MAAS [12:38] SimonKLB: consider MAAS as your IP address management source [12:39] frobware: gotcha [12:39] frobware: what about LXD containers, for example in the openstack bundle? [12:39] frobware: from where are they grabbing their ips? [12:40] SimonKLB: if you do `juju add-machine lxd:0` we as MAAS for an IP address. It's static in /e/n/i in the container but is statically allocated from MAAS. [12:40] will it fetch the ip from the same subnet as the host? [12:41] frobware: and if you have multiple nics, which subnet? [12:41] SimonKLB: for Juju 2.0 it will use interfaces that have a subnet and an IP address [12:43] SimonKLB: otp - will answer in a bit [12:44] frobware: yea, just tried it, it has the same behavoir as auto-assign in MAAS it seems [12:44] frobware: it grabs an ip that is reserved, which is problematic :/ [12:46] SimonKLB: as in the reserved range? What's the issue? [12:47] frobware: yea, i want to restrict the subnet to a range, from what i understand the reserved ranges are supposed to be "those that are already taken" [12:47] frobware: but when i deploy an LXD container it still grabs an ip from a range that is reserved [12:50] frobware: this is where i read it https://docs.ubuntu.com/maas/2.1/en/intro-concepts [12:50] SimonKLB: you have a reserved range setup in MAAS? [12:50] "Reserved range An IP range that MAAS will never use. You can use it for anything you want (e.g. infrastructure systems, network hardware, external DHCP, or the namespace for an OpenStack cloud you will be building)." [12:51] frobware: i have a subnet with a CIDR that includes a number of ips, most of them i have in the reserved range so that they are restricted from being used by MAAS [12:51] SimonKLB: hmm. so the container has been allocated a IP addr from your reserved range. [12:51] frobware: correct [12:51] frobware: and this also happened to a machine before [12:51] frobware: when it had "auto-assign" as the ip mode option === deanman is now known as deanman_ [12:57] SimonKLB: I have two ranges on my 10.100.0.0/24. 10.100.0.1->10.100.0.99 as reserved and 10.100.0.250..10.100.0.254 as dynamic. [12:57] frobware: the dynamic ranges are only used when you use the MAAS DHCP right? [12:57] frobware: im running an external DHCP [12:57] SimonKLB: I just added two containers and they ended up as 10.100.0.101 and 10.100.0.102 - which seems to be correct [12:58] SimonKLB: yes to your DHCP question [12:58] frobware: good, with you so far! [12:58] SimonKLB: so, it seems to be allocating out of the correct range. agreed? [12:59] frobware: yea [12:59] frobware: but i wonder if it could be behaving differently with an external DHCP instead of the MAAS one [13:00] SimonKLB: perhaps. difficult for me to repro that quickly. [13:00] SimonKLB: I did try without a DHCP range as well. it was allocated out of the non-reserved range. [13:00] frobware: yea np, but is there a way to simulate the DHCP "IP Mode" option that you have for machines? [13:01] frobware: for lxd containers that is [13:01] frobware: because changing it from auto-assigned to DHCP seemed to have fixed it for me at least for the machines [13:03] SimonKLB: you're using MAAS 2.1.1, correct? [13:05] frobware: maas/xenial,now 2.1.2+bzr5555-0ubuntu1~16.04.1 [13:05] SimonKLB: I wonder if there is some difference on 2.1.2. [13:06] SimonKLB: I just created: reserved range: 10.100.0.40->10.100.0.99 and dynamic as 10.100.0.100->10.100.0.199 [13:06] SimonKLB: allocated another container and it came back as 10.100.0.3 i.e., before the reserved range [13:07] frobware: what happens when you try to allocate more containers than available IPs? [13:07] SimonKLB: which to me says MAAS is finding the first applicable IP address, given the ranges. [13:07] frobware: yea i wish :) [13:07] SimonKLB: I would expect MAAS to say "no, IPs available", and for Juju to gracefully fail. [13:18] * frobware lunches === deanman_ is now known as deanman [15:31] natefinch: bug #1631254 [15:31] Bug #1631254: [2.0rc3] lxd containers do not autostart [15:32] natefinch: https://github.com/tych0/juju/commit/81156dfb3c1d21431cb3bd5047a51e13bd91fc5d [15:46] macgreagoir: thanks for the email === deanman is now known as deanman_ [21:30] thumper: can you have a look at https://github.com/juju/juju/pull/6783 ? [21:35] ack [22:20] thanks [23:17] what's the magic incantation to get cross model relations to work. I assume it is behind a feature flag [23:20] well I thikn i have it but still have an error [23:33] got it