[00:53] wallyworld: re the trusty tests, while looking to remove them, I recall that I left them in as they also test i386 constraints, which isn't supported in xenial anymore, IIUC> [00:53] wallyworld: with that in mind do you still want them removed? [00:54] redir: yeah, 386 is no longer supported in 2.0 [00:54] \o/ [00:54] ahh even in trusty and older. OK makes sense. [00:54] tx [01:09] wallyworld: how about the even older quantal tests? nix them too? [01:10] redir: depends. we tend to use quantal as a series for certain things in tests where we want to be sure the system on which the test runs is not that series [01:10] ok [01:10] * redir leaves them [01:10] only removes trusty [01:32] bbiab [02:06] Bug #1574272 changed: Juju agent uninstalls itself while adding machine [02:12] Bug #1574272 opened: Juju agent uninstalls itself while adding machine [02:21] Bug #1574272 changed: Juju agent uninstalls itself while adding machine [04:16] single character code review anyone? [04:16] http://reviews.vapour.ws/r/4704/ [04:17] wallyworld, thumper, menn0 ^ [04:17] looking [04:19] natefinch: now if that facade were not done separately and differently to every other facade it would not have missed having its version bumped [04:19] natefinch: and there are tests that check that all api and apiserver facades marry up [04:19] wallyworld: yeah. [04:19] which also do not pick this up [04:20] wallyworld: the lack of tests is the real problem here, I think. [04:20] we have tests [04:20] but because this was done differently they did not pick it up [04:20] wallyworld: I mean, yes, also the fact it's different is bad [04:22] I don't think it's necessarily bad to component1/api component2/api rather that api/component1 api/component2 ... it's just that we have a mix, which is confusing, and this code is missing tests that should have picked up the mismatch of version numbers. [04:22] anyway, bedtime for me. [04:39] Bug #1574949 opened: setting a valid config setting should not warn [04:47] wallyworld: http://reviews.vapour.ws/r/4691/diff/1-2/ updated per earlier conversation [04:48] great ty [04:49] redir: looks good, just missing that extra test [04:49] there are tests [04:51] wallyworld: https://github.com/juju/juju/blob/master/environs/config/config_test.go#L1855 [04:51] or it is covered [04:51] redir: ok, np, i was just going by the diff, let me check the link you just posted [04:53] redir: those tests don't check that FallbackLtsSeries is used as expected [04:53] that's the missing test i think we need [04:54] i added a couple using patch executable, but they didn't cover anything additional so I didn't commit them [04:55] * redir_ looks again [05:00] redir: i guess the fallback series is only used to set testing FakeLtsSeries so it doesn't really need any more tests [05:01] i just wanted to ensure that the now exported FallbackLtsSeries was used as expected now that it is exported [05:03] got it committing. [05:10] wallyworld: http://reviews.vapour.ws/r/4691/diff/2-3/ adds tests to ensure the exported variable is used [05:10] ok, ta [05:10] they don't add any coverage however. [05:11] I am going eod... but will merge in the morning if it looks good to you. [05:13] redir: ok, np. looking at the tests, it was not so much testing the setting of FallbaclLtsSeries (as you say that adds no extra coverage), but more testing places that used it got waht we expected ie was it logic to use fallbacklts wired up correctly. i think since we just use it to set a testing variable elsewhere, we can do without these tests [05:15] ok reverting those tests:) [05:16] OK they are gone again wallyworld :) [05:17] ty, sorry for misunderstanding [05:17] lgtm :-) [05:19] np, It is getting late here, so I may have been unclear. [05:19] I can ship it now if you want but I am not waiting around for the merge bot [05:20] wallyworld: ^ [05:20] redir: yep, np. i'll check the progress [05:25] k, the bot's got it [05:25] nite [05:25] btw, when do sleep? [05:25] when do you sleep? even... [05:54] Bug #1574963 opened: juju2 lxd launch hostname reverse lookup inconsistent [06:17] night all [07:58] dimitern: ping, running 10mins late... [07:58] frobware: no worries, thanks for reminding me :) [08:01] frobware: ping me when you're there [08:08] dimitern: let's go [08:09] frobware: omw [08:16] dimitern: let's try after standup [08:27] morning [08:50] dimitern: I've replied to two of your points on the review of my branch: http://reviews.vapour.ws/r/4700/ [08:50] dimitern: the error 500 on the subnet is unrelated to the points you make and could be a genuine error [08:50] dimitern: and one of the points (about creating the primary NIC) I'll talk to you about in standup [08:51] dimitern: the other point about null types I'm fixing. [08:59] dimitern: I'd like to see the output of subnets read on your maas server - the error appears to be that subnet "2" doesn't exist [09:01] dimitern: frobware: standup? [10:53] dimitern: When setting up a VLAN and subnet, if I want a machine on that VLAN to be able to get to the internet (or anywhere outside that subnet) I need to set up a default gateway, right? [10:54] dimitern: Should that be the address of the MAAS controller? Does that mean that it needs another interface that I can assign that VLAN and subnet? [10:58] dimitern: reading docs [11:02] babbageclunk: yeah, the controller usually has access to all VLANs (has NICs on each) [11:02] babbageclunk: and also acts as a gateway for the VLAN's subnets [11:03] Ok, so I need to add an interface on the controller for each VLAN I create? [11:03] babbageclunk: to make the setup closer to the real world, add gateways to each subnet pointing to the respective cluster addresses on that subnet [11:04] dimitern: ? [11:04] dimitern: cluster addresses? The interface on the controller? [11:04] babbageclunk: yeah, one NIC for each subnet on the controller is the easiest thing, both for physical and vlan subnets [11:04] babbageclunk: yep [11:05] dimitern: ok, thanks. [11:05] babbageclunk: np [11:15] dimitern: I get an error when I try to create a subnet: malformed array literal: "" [11:15] LINE 1: ...', 5005, 1, '10.10.0.0/24', 2, '10.10.0.1'::inet, '', true) ... [11:15] ^ [11:15] DETAIL: Array value must start with "{" or dimension information. [11:16] babbageclunk: what did you run to get this? [11:16] Just trying to create the subnet in the MAAS admin. [11:18] babbageclunk: is that in 2.0? [11:18] babbageclunk: I mean can you paste the command line? [11:18] I made a VLAN with the VID and name 10, and then tried to create a 10.10.0.0/24 subnet [11:18] I wasn't using the command line, hang on - I'll try that. [11:18] babbageclunk: have you tried setting up the VLAN NICs on the cluster first and rebooting? [11:19] dimitern: I thought that detection only happened at install time [11:19] ooh wow [11:19] * dimitern haven't see the web ui in 2.0 allows that [11:19] * frobware is jealous in 1.9 land... [11:19] dimitern: Added the nic and rebooted, but I can't set the vlan tag on the new nic. [11:20] frobware: unless it changed, it also happens at boot [11:20] dimitern: nice! [11:20] babbageclunk: check out this http://juju-sapphire.github.io/post/MAAS%20Spaces%20Demo/ [11:20] dimitern: I was only scripting that a few days ago [11:21] frobware: ah, ok [11:22] dimitern: I thought maybe that was because I'd need to assign a subnet to the nic first and then it would set it to the VLAN for that subnet [11:22] dimitern: but having trouble creating the subnet. [11:22] frobware: it seems to work, as I've recently "gained" a fabric-2 with a 192.168.122.0/24 subnet on it after installing libvird-bin and rebooting [11:23] dimitern: but this is 2.0 only, yes? [11:24] dimitern: oh yeah - I've got that as well, on fabric-1 (I assume because I installed libvirt-bin before adding the new nic) [11:24] babbageclunk: nope, the creation order is: fabrics and spaces, vlan on the fabric, subnet on a vlan and space [11:24] frobware: it might be 2.0 only, now that I think of it [11:25] dimitern: Ok - I've got fabrics 0 (original nic on the controller), 1 (bridge from libvirt-bin), 2 (new nic I added to be on the new vlan) [11:25] dimitern: I've got a private space [11:26] dimitern: Trying to add a subnet in the private space in the 10 vlan. [11:26] babbageclunk: but, you either do A or B, no need for both; A) edit /e/n/i on the maas controller to e.g. add eth0.11, eth0.12, eth0.13 vlan nics, which are statically configured with e.g. 10.11.1.1/24, 10.12.1.1/24, 10.13.1.1/24 (so maas can detect the subnet along with the vlan, and from the parent device eth0 it can detect the fabric) [11:26] dimitern: Does the vlan need a primary rack? [11:27] or B) create vlans, then spaces, then subnets within the vlans and spaces manually with the CLI (or if it works - web ui) [11:27] babbageclunk: I expect so, but you should only have one anyway [11:28] dimitern: Ah - when I added the nic did I need to add a new network in VMM as well? [11:28] dimitern: I fixed the "nil" issue by the way, so if you could complete the review sometime it would be appreciated [11:28] dimitern: http://reviews.vapour.ws/r/4700/ [11:28] doing some multi-nic testing. [11:28] well, attempting to without screwing up maas :-/ [11:28] babbageclunk: not necessarily - adding new NICs on the VM is only needed if you need more than one physycal NIC to appear on the VM when commissioned by MAAS [11:29] dimitern: but how else would I be able to make sure the controller had an address in the right vlan? [11:29] babbageclunk: you can add VLANs on top of existing physycal NICs [11:30] dimitern: how? [11:30] dimitern: do I do that in the UI, or somewhere else? [11:30] voidspace: ta, will look in a bit [11:30] babbageclunk: see for example my /e/n/i from the maas controller machine: http://paste.ubuntu.com/16061991/ [11:31] dimitern: (Sorry for all the questions - I feel like there's a lot of knowledge that I'm missing that everyone else has forgotten they know.) [11:32] babbageclunk: if I add e.g. "eth1.42" to that, with address 10.42.0.1/24 and vlan-raw-device eth1, then reboot maas should have created a 10.42.0.0/24 subnet linked to a newly created 42 on the same fabric eth1 is on [11:33] dimitern: ok, that makes sense - I thought that I could do that through the UI but had missed a step. [11:33] babbageclunk: well, you could also to that from the CLI (possibly the UI as well), but why bother if MAAS does it for you? [11:34] babbageclunk: once you have it working (the auto-detected way), you could try to replicate it manually, if you're interested [11:35] dimitern: ok, I'll do that. [11:36] babbageclunk: here's some more info that can help shed some light on the manual approach: http://blog.naydenov.net/2016/01/maas-setup-deploying-openstack-on-maas-1-9-with-juju/ [11:37] just you'd have to adapt it for the 2.0 API changes [11:38] dimitern: ok, that's great, thanks. [11:41] fwereade: jam: hey, will you be able to make the planning meeting in ~20m? [11:41] fwereade: doh, you responded. sorry to pester ;) [11:42] katco, thanks for the reminder, though :) [11:42] fwereade: ha ;p definitely time for some coffee [11:47] katco: yes [11:51] voidspace: reviewed [11:51] jam: awesome, ty! [11:52] frobware: hey were you able to get those estimates? i see you requested access to the spreadsheet [11:58] dimitern: ta [12:00] katco: nope. don't have any meaningful numbers [12:08] voidspace: live testing your branch now - I think I found the cause of "subnet not found" [12:08] voidspace: PUT http://10.14.0.1/MAAS/api/2.0//MAAS/api/2.0/nodes/4y3h8x/interfaces/112/, para [12:08] ms: [12:08] [12:09] we pass no params (I'd expect the vlan id there) when updating interface [12:11] dimitern: no vlan id [12:12] dimitern: ok, I'll look for that [12:13] voidspace: so what I think happens is maas tries to find a subnet with id=2 in the vlan with id=0 (which does not exist) [12:13] dimitern: right [12:13] dimitern: don't suppose you have any logging where it's coming from [12:13] dimitern: PUT is for update - which only happens on the device create I think [12:14] voidspace: ah, just killed the controller :/ [12:14] heh [12:14] I'm looking [12:14] voidspace: but the PUT 500 was right after the device was created [12:15] dimitern: so in gomaasapi/machine.go:310 you can see the code [12:16] dimitern: it's only adding a VLAN if the iface and Subnet don't match - which may well be incorrect [12:16] * dimitern takes a look [12:16] dimitern: that should probably be unconditional [12:19] voidspace: it makes sense not to set the vlan param if it's empty [12:20] voidspace: as you might want to e.g. just rename the interface or change its mac [12:20] that's where the vlan should be added [12:21] dimitern: should it be vlan id or vlan vid [12:21] we're setting id, which is consistent with the other entitites (use id) [12:21] in interface.go:137 [12:22] voidspace: it should be the vlan id, not vid (or alternatively vlan="vid:") [12:22] yeah [12:22] well, there's definitely code to handle it in that code path [12:24] dimitern: actually, that code doesn't specify a subnet (the update call), so it's probably not that one [12:24] dimitern: let me look at LinkSubnet - but that's a POST not a PUT [12:25] voidspace: the update call sets the vlan, the link-subnet call sets the subnet and mode [12:25] dimitern: right, but you said the error was specifying a subnet and no vlan [12:26] dimitern: and the update call (the PUT) doesn't specify a subnet [12:26] dimitern: so it can't be that [12:27] I need a setup I can try this on [12:27] I'm shooting in the dark here [12:27] lunch first [12:27] voidspace: sorry, I didn't say we need subnet and vlan in the link-subnet call [12:28] dimitern: no, you said you'd expect to see a vlan id in the update interface call [12:28] voidspace: however, not setting a valid vlan on the interface won't allow you to link any subnet to it [12:28] dimitern: but it doesn't take that parameter [12:29] oh, it does [12:29] dimitern: right [12:29] voidspace: yep [12:29] dimitern: however, we do have code to set the vlan id - so it's weird that it's not getting there [12:31] * dimitern *facepalms* [12:32] go on... [12:32] dimitern: I've added the vlan in ENI, but it doesn't show up in the controller. Do I need to do something (other than rebooting) after changing ENI? [12:33] voidspace: of course it won't work :) we're not actaully updating device's eth0 vlan at all [12:33] dimitern: explain [12:34] voidspace: we're creating a device with a mac, then reading back the interface_set from it, skipping the primary in the loop around 2308 in environ.go, and only recording its id [12:34] dimitern: but machine.CreateDevice does the update [12:34] dimitern: in gomaasapi [12:34] Bug #1574949 changed: setting a valid config setting should not warn [12:35] dimitern: which is why we skip it [12:36] voidspace: yeah, it looks like Update() uses the subnet linked to the primary device NIC to figure out which vlan id should use [12:36] voidspace: sorry, not Update(), but CreateDevice() [12:37] voidspace: so the problem is in line 2272 in environs.go, not gomaasapi [12:38] dimitern: ah, so CIDR is not unique - so a simple hash is not sufficient [12:38] dimitern: it's CIDR/vlan id pairs [12:39] dimitern: it pulls out the wrong subnet [12:39] ? [12:39] voidspace: cidr is unique [12:39] voidspace: but I suspect preparedInfo['eth0'].CIDR is empty [12:39] dimitern: the maas 1.9 code is doing the same [12:40] voidspace: hmmm.. true [12:40] mystery [12:40] * voidspace lunches [12:41] dimitern: ooh, while he's gone, can we hangout again? I'm hopelessly confused. (Unless you also want to lunch?) [12:42] babbageclunk: can you give me 10m ? [12:42] dimitern: if you do I can go for a run now and we can chat after? [12:42] babbageclunk: sounds better - please ping me when you're back then [12:42] dimitern: ok, thanks [12:44] voidspace: the problem is using Spaces(), followed by Subnets() on each does not give you the vlan of the subnet [12:48] voidspace: and CreateMachineDeviceArgs.Validate must also check a.Subnet.VLAN() != nil [12:48] dimitern: Space.Subnets() doesn't set vlan properly [12:48] that's a bug if true [12:48] anyway, really going on lunch [12:48] (you may be right) [12:49] voidspace: enjoy :) [13:04] Bug #1573410 changed: trusty juju 1.25.5 having issues deploying xenial lxc containers [13:50] dimitern: back! [13:50] babbageclunk: hey, so let's use the standup HO? [13:51] frobware: hey, thanks for showing up. sorry it wasn't useful to you. [13:51] dimitern: yup [14:04] katco, ping? [14:04] mattyw: pong [14:05] katco, hey there, I'm struggling to deploy local charms on beta6 (it also fails on master) I'm getting connection timed out on ec2 but nothing more useful in the log than that (charmstore charms work fine) shall I raise a bug? [14:06] mattyw: s/charms/resources/ ? or do you really mean charms? [14:08] katco, charms [14:08] mattyw: does it work on a different substrate? i.e. is it a fluke? [14:09] katco, it did work on lxd, but I had other problems on lxd so had to stop using it [14:09] mattyw: this seems like something CI would be catching... hm [14:10] mattyw: i don't think it would hurt to raise a bug, but the inability to deploy a local charm seems very severe and something everyone would be seeing [14:11] sinzui: cherylj: mgz: is ci seeing anything like this? [14:13] katco: matty, ci deploys many 10s of local charms and is not seeing an issue [14:13] mattyw: in good news, lxd will probably work for you now :) [14:14] mgz, lxd was sort of working, I was trying to install lxd in lxd though, and that was giving me problems [14:15] mattyw: this is an example of a deploy froma few hours ago "juju --show-log deploy" [14:15] mattyw: what connection is timing out? the actual deploy command? [14:15] sinzui: I see no instances in eu-west-1. Did you do cleanup, or did we finally get a clean run? [14:15] cherylj, seems to be yeah [14:15] abentley: I have too many disasters to deal with to clean up. I guess all went well [14:16] mattyw: can you paste the juju deploy --debug output? [14:16] abentley: I saw the region being used yesterday [14:16] cherylj, https://pastebin.canonical.com/155238/ [14:16] sinzui: Yes. industrial tests were still running last I checked, but it looked reasonable then. [14:17] mattyw: and this is ec2? [14:17] cherylj, yep [14:18] mattyw: yeah, it's not filtering lxd addresses [14:18] jam has a PR up for that [14:18] babbageclunk: http://rickardnobel.se/the-vlan-802-1q-tag-part-1/ [14:18] cherylj, I don't believe lxd is being used in this situation [14:18] mattyw: can you ssh to your controller and run an ifconfig? [14:19] mattyw: I'm guessing you have a bridge device on the controller that's not being filtered, but I could be wrong [14:19] dimitern: you froze up! Are you ok over there? Thanks heaps anyway! [14:19] babbageclunk: yeah, to it appeared you dropped out btw [14:19] s/it/me/ [14:19] argh [14:19] s/it/me it/ :) [14:20] mattyw: can you also go to your ec2 dashboard and see what IPs that machine has assigned to it? [14:20] dimitern: ah well, I think we're finished anyway - I'll read through that and get the vlans set up in my maas. [14:20] Thanks! [14:22] babbageclunk: I've sent you a few links in prvmsg not to flood the channel :) [14:23] cherylj, so I can't seem to ssh into it anymore, seems like none of the juju commands work, I've found the instance in ec2 though [14:24] cherylj, it's eu-west, I guess it might be a not commonly used region [14:24] mattyw: yeah, that doesn't surprise me that you couldn't ssh through juju [14:26] frobware: https://github.com/juju/juju/pull/5281 may look very similar to something you reviewed yesterday... if you could take a quick look. [14:26] cherylj, how do I ssh to the controller now? [14:27] mattyw: can you ssh directly to ubuntu@54.195.25.5 ? [14:28] mattyw: try juju ssh --proxy 0 ? [14:28] cherylj, yes: https://pastebin.canonical.com/155242/ [14:28] dimitern, ERROR machine 0 not found (not found) [14:28] hmm, I guess that region does actually use a 10. internal IP [14:28] I kinda figured it was a bridge IP [14:29] mattyw: juju switch admin then the above? [14:29] dimitern, that seems to work [14:29] or more verbosely juju ssh --proxy -m admin 0 [14:30] mattyw: see anything interesting in machine-0.log? [14:30] well, for machine 0 of the admin model you shouldn't need to pass --proxy, but for any other machine (or container on machine 0), --proxy is needed unless you can ping the private address of the machine directly [14:33] cherylj, nothing really [14:34] mattyw: and the deploy command just hangs in the POST? [14:35] cherylj, yeah [14:35] cherylj, I've just discovered the charm is >600mb, I wonder If I just have to wait for ages, and expect some errors to happen [14:35] could be [14:35] too bad there's not a progress indicator [14:36] I mean, you could watch df on the controller to see if something is happening [14:36] or use jnettop [14:38] cherylj, yeah will do, I must admit to not realising the charm was that big [14:39] mattyw: yeah, that is surprising [14:45] cherylj: mattyw: I'm trying to get that landed today [14:54] cherylj, yep panic over, just took ages [14:54] yay! [14:54] (well, not yay for taking ages, but yay to not being a regression) [14:54] cherylj, thanks for sticking with me [14:54] mattyw: of course :) [15:02] Bug #1575229 opened: juju/utils/GetAddressesForInterface only returns IPv4 addresses [15:07] cherylj, sinzui, alexisb: xenial amis are available [15:07] rcj, awesome [15:09] wee [15:10] thank you cherylj , I will update the merge job [15:10] rcj: <3 [15:10] dimitern: you're wrong by the way - I fetch the spaces from MAAS directly there, and it does get VLAN information [15:11] dimitern: those are gomaasapi.Subnet not network.SubnetInfo [15:11] voidspace: oh, it does? hmm.. [15:11] dimitern: well, I'm pretty sure you're wrong anyway [15:11] Bug #1575229 changed: juju/utils/GetAddressesForInterface only returns IPv4 addresses [15:11] voidspace: well, CreateMachineDeviceArgs.Validate has to check Subnet.VLAN is != nil [15:12] dimitern: that's true enough, but it's not the cause of the error [15:12] still working on a multi-nic setup [15:13] voidspace: well, something is wrong with the 2.0 implementation if AllocateContainerAddresses or something in gomaasapi, as the 1.0 API path still works [15:14] dimitern: sure :-) [15:14] dimitern: once I can reproduce I'm confident I can track it down [15:15] voidspace: +1 [15:15] dimitern: I think your diagnosis about the VLAN might be right [15:15] dimitern: I just can't see how from the code [15:15] dimitern: but some debugging info should reveal it [15:17] voidspace: I'd suggest commenting out the deferred device deletion on failure to inspect it [15:20] dooferlad: just a couple of coments, but LGTM [15:21] cherylj: got a sec to chat about a bug? [15:21] katco: sure [15:21] cherylj: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1 [15:23] voidspace, babbageclunk, dooferlad, dimitern: please remind me, who ran into the iface names too long issues yesterday? And was a bug raised for this? [15:23] Bug #1575229 opened: juju/utils/GetAddressesForInterface only returns IPv4 addresses [15:24] frobware: I thought it was dimitern - at least he told me about it :-) [15:24] dimitern: I deleted a node from my maas (because I screwed up its network) and want to recommission it, but it won't pxe boot, always falls back to the disk. [15:25] frobware: yeah, I think it was dimitern [15:25] babbageclunk: check the boot order [15:25] babbageclunk: boot order in the VM settings [15:25] voidspace, frobware: thanks, but I checked that. It shows that it's trying to boot from the network but nothing happens. [15:26] dimitern: ^^ - please could you raise a bug for the iface name issue [15:27] fwereade, I will be a bit late for our 1x1 [15:27] babbageclunk: weird. And the subnet that exists is one with managed dhcp? [15:28] voidspace: ooh, checking [15:31] voidspace: hmm, I don't know where to check that [15:31] babbageclunk: in the maas ui - networks [15:31] alexisb, chatting to dimitern, let me know when you're ready [15:31] babbageclunk: find the subnet you're interested in and click on "untagged" (vlan) [15:32] babbageclunk: and it will tell you if DHCP is enabled (you need it enabled to pxe boot) [15:32] anybody up for the fastest review of the week? http://reviews.vapour.ws/r/4710/ [15:32] voidspace: yup, DHCP is enabled on the untagged VLAN in fabric-0 [15:33] bogdanteleaga: shipit === mhilton-lunch is now known as mhilton [15:34] babbageclunk: :-/ [15:34] babbageclunk: clone the vm and commission a new one instead [15:35] babbageclunk: maybe maas won't commission a deleted machine? [15:35] babbageclunk: or ask in #maas [15:35] voidspace: might try creating a new one and see if that gets picked up ok [15:39] voidspace: nope - ok, something wrong with the controller rather than the node. [15:40] babbageclunk: :-/ [15:41] voidspace: undoing the vlan change to see if that helps [15:41] dimitern: ok, I can reproduce [15:44] Bug #1575245 opened: Juju 2.0 LXD install instructions don't work on trusty [15:45] voidspace: \o/ [15:45] voidspace: what was it? [15:46] dimitern: I said I can repro it [15:46] dimitern: working out what causes it is the next step [15:47] voidspace, dimitern: what the crap - my ENI is empty now! Just has loopback in it now. That might explain the problem (although I don't know how that happened). [15:47] frobware: which iface name issue? about biosdevnames and enxxaabbccddeef0 ? [15:47] dimitern: yep [15:47] Bug #1575245 changed: Juju 2.0 LXD install instructions don't work on trusty [15:47] babbageclunk: hah, nice [15:48] yay, lucky I put it in a pastebin while chatting to dimitern [15:48] frobware: I did yesterday - bug 1572070 [15:48] Bug #1572070: MAAS 2.0 cannot link physical device interfaces to tagged vlans, breaking juju 2.0 multi-NIC containers [15:49] babbageclunk: sorry, I was talking to fwereade - catching up on scrollback [15:50] no worries - it was a false alarm, my ENI is still there. [15:50] babbageclunk: ah, good! :) [15:50] dimitern: is that the right bug#? [15:51] frobware: it's against maas only, not juju [15:52] dimitern: I'm confused. How is that related to enxxaabbccddeef0 and our resultant bridge name? [15:52] * dimitern *d'oh* [15:52] frobware: sorry - looking more carefully now [15:53] frobware: there it is - bug 1572070 [15:53] Bug #1572070: MAAS 2.0 cannot link physical device interfaces to tagged vlans, breaking juju 2.0 multi-NIC containers [15:53] argh [15:53] bug 1574771 [15:53] Bug #1574771: MAAS/curtin generate invalid /e/n/i and failed deployment for nodes with long (biosdevname) interface names, which in turn have VLANs [15:53] Bug #1575245 opened: Juju 2.0 LXD install instructions don't work on trusty [15:54] alexisb: this ^^ was the bug I mentioned earlier (bug 1574771) [16:07] dimitern: yup - if I remove the vlan, I can commission. If I add it back in, I can't. [16:08] babbageclunk: add/remove from the maas machine? [16:11] babbageclunk: I'm not sure what are you talking about :) [16:11] sinzui: is it safe to drop code that is only compiled for older versions of Go? [16:11] (older than 1.6) [16:11] dimitern: sorry - add/remove it to/from ENI [16:12] babbageclunk: on the maas controller maching? [16:12] machine* [16:17] dimitern: yup [16:18] babbageclunk: how does the commissioning fail? [16:18] babbageclunk: do you see the console of the VM node doing stuff? [16:19] dimitern: I get this error: 0x040ee119 [16:19] babbageclunk: so pxe is failing [16:19] dimitern: yup [16:19] dimitern: so presumably that means DHCP? [16:20] babbageclunk: yeah, among other things [16:20] dimitern: could it be setting the dns servers on the subnet? [16:21] dimitern: I don't know why that would cause it though. [16:21] babbageclunk: hmm.. let me check something here on my maas [16:22] dimitern: so it's the call to machine.CreateDevice that errors - however as far as I can tell the juju side of that is correct (right subnet with the right vlan) [16:22] dimitern: so instrumenting gomaasapi to see what it actually sends... [16:22] voidspace: it gets deeper .. :/ [16:23] babbageclunk: how did you trigger the commissioning - via the UI or CLI? [16:23] Bug #1564622 changed: Suggest juju1 upon first use of juju2 if there is an existing JUJU_HOME dir [16:23] cherylj, katco: btw, got clarity from security... basically we just need to blacklist the RC4 ciphersuitem which is fairly straightforward... I'll add a function to juju/utils to create a default tls.Config and make sure we use that everywhere. [16:23] dimitern: yeah, as far as I can tell the arguments to machine.CreateDevice are correct, and note that at this point it's the same code path as for a single nic [16:23] dimitern: it hasn't even got into the part that's different! [16:23] babbageclunk: and what series was used - xenial or trusty? [16:24] dimitern: it *could* still be a MAAS bug :-) [16:24] voidspace: hmm the difference between single-nic and multi-nic should only be the way we handle the primary nic (update the vlan + link-subnet) [16:26] natefinch: cool, glad to hear it's manageable [16:26] dimitern: xenial [16:27] awesome, thanks for following up on that, natefinch [16:27] dimitern: although it wouldn't ever get to that point. [16:27] babbageclunk: and that particular node has 1 NIC on the same bridge where the controller's managed interface is on? [16:28] dimitern: I think so. What do you mean by managed interface? [16:28] natefinch: good question. We are wtill running windows and centos unit test with go 1.2 because they do not pass on go 1.6. I don't think you can require go 1.6 today [16:29] babbageclunk: how many NICs does your maas controller KVM has configured? [16:29] sinzui: booooooooooooooooooooooooooo [16:29] oooooo [16:29] dimitern: 2 [16:29] * natefinch sadface [16:29] babbageclunk: ok, and they are attached to difference virbrX bridges? [16:29] dimitern: I added one when I thought I needed another for the vlan. [16:30] babbageclunk: is it possible that second bridge has DHCP enabled on it?\ [16:30] babbageclunk: (last bridge you added) [16:30] sinzui: are we building the binaries with go 1.6? [16:30] sinzui: for centos and windows? [16:30] natefinch: yes [16:30] dimitern: I'm not sure how to see what bridge it's connected to. [16:31] sinzui: eq [16:31] ew [16:31] natefinch: we only build with golang 1.6 [16:31] dimitern: in VMM? They're both on the maas2 network - I guess that's the bridge. [16:31] babbageclunk: if you have virt-manager installed and connected to qemu:///system it will be easy :) [16:31] cherylj: this seems like a pretty bad thnig - our tests and our production code are being built with different versions of go. which means we're not actually testing 100% what we're distributing [16:32] babbageclunk: ah, well - not quite the network is "backed" by a virbrX bridge, but it's a separate entity as far as libvirt is concerned [16:32] dimitern: I'm going to remove the second nic anyway now - it's not needed. [16:32] babbageclunk: so yeah, how is that maas2 network configured in terms of dhcp, ipv4/ipv6, etc? [16:33] babbageclunk: that might help indeed :)_ [16:33] dimitern: IPv4, DCHP off, NAT on [16:34] babbageclunk: ok, that's correct [16:34] babbageclunk: then, if you go to the Nodes | Controllers | your maas controler [16:34] dimitern: ok, I'll ditch the other one and put the vlan back, then try recommissioning. [16:35] dimitern: yup [16:35] ? [16:35] natefinch: I'm not sure what you're referencing. Since we made the move to go 1.6, we use it everywhere. The only exceptions are windows unit tests using go 1.6 are still non-voting, pending some changes from perrito666. Oh and, centos unit tests too [16:35] babbageclunk: what does it show in the "served vlans" section and interfaces below it? [16:35] natefinch: I think you misunderstand what is happening. we switched to go 1.6 as core asked, but core is not fixing the test as QA asks [16:36] babbageclunk: sorry for the 20 questions :) I suspect the issue might be enabling DHCP on the newly added VLAN [16:36] dimitern: no worries! I appreciate the help. [16:36] dimitern: [16:37] dimitern: both urls seem valid, but ill make the change [16:37] dimitern: so served VLANs has fabric-0 (the nic) and fabric-1 (the bridge that got added when installing libvirt-bin) [16:38] babbageclunk: ok, so the first fabric-0 is the managed one, and its vlans should all have dhcp enabled [16:38] dimitern: under Interfaces there are ens3 (original nic), ens9 (second nic I added) and virbr0 (the libvirt bridge) [16:39] dimitern: yup (there's only the one VLAN, I removed the new one) [16:39] dimitern: fabric-1's vlan has DHCP off [16:39] babbageclunk: ok, so the ens9 is the (only) managed interface, but with you should also see the newly added VLAN interface there as well - e.g. ens9.42 [16:40] babbageclunk: that's fine though - it can't be on as libvirt also provides dhcp for 192.168.122.0/24 by default (virbr0) [16:41] babbageclunk: ah, you've removed the new vlan, ok [16:41] babbageclunk: and you're saying that adding it blocks commissioning - adding it how? manually with the CLI or editting /e/n/i and rebooting to let maas autodiscover it? [16:42] dimitern: so ens9 is the managed interface? should I be adding the vlan to that interface instead of ens3? [16:42] dimitern: the latter [16:42] babbageclunk: aha! that's it I think :) [16:43] dimitern: (and then using the cli to set space/gateway/dns_ [16:43] babbageclunk: ok, so that misconfigured ens3 was the issue I suspect [16:44] dimitern: so you're suggesting - keep ens9, configure it in ENI, and add the vlan to that instead? [16:44] sinzui, cherylj: sorry, had to step away... I just mean - we should prioritize work to get the windows and centos tests passing on 1.6. [16:44] dimitern: ok, I can see that in your ENI from the pastebin [16:45] babbageclunk: sorry - which one did you add last - ens9 or ens3 ? [16:45] ens9 [16:45] natefinch: yeah, perrito666 was doing some work for windows [16:45] natefinch: yes we agree [16:45] sinzui: is there a bug open for the centos / 1.6 failures? [16:45] cherylj: sorry I thought I merged those last night but got flakytestwalled [16:45] babbageclunk: ah, ok then add the vlan to the ens3 and drop ens9 (on the vm as well) [16:46] cherylj: bug 1570883 [16:46] perrito666: I love that term, totally stealing it [16:46] Bug #1570883: imageSuite.TestEnsureImageExistsCallbackIncludesSourceURL fails on centos go 1.6 [16:46] sinzui: ah, I see it now [16:46] babbageclunk: as ens3 was the "original" managed interface :) [16:46] dimitern: ok [16:46] dimitern: (by virtue of being the only one0 [16:46] ) [16:47] babbageclunk: yep [16:47] babbageclunk: my maas-es setups all use dual-nic lxc containers for the controllers - one managed, one "external" unmanaged [16:48] babbageclunk: but that's not a requirement, just me making my life interesting :D [16:48] dimitern: :) [16:51] babbageclunk: to be on the safe side, I'd delete that node which failed commissioning and re-add it [16:53] babbageclunk: for kvm-based nodes by far the easiest way is 'Add Hardware|V|' -> Chassis -> Power type 'virsh', Address like 'qemu+ssh:///...', etc. and set a prefix filter on the node names (e.g. mine are called maas-19-node-0, maas-18-node-2, etc.) [16:54] dimitern: Ooh, that's nice. [16:54] so I'd use 'maas-19-node-' as prefix [16:54] dimitern: I tried adding a new one though, and it also wouldn't pxe boot [16:54] babbageclunk: yeah :) nice trick [16:54] babbageclunk: so a "new" node can't pxe boot just like that, maas won't allow it [16:55] babbageclunk: it needs to be turned on and try pxe booting (with priority over local disk boot) [16:56] dimitern: yeah, I had done that. [16:56] babbageclunk: then it shows up as "new" and you can edit a few things like name, zone, etc. and then you "accept" it [16:56] dimitern: ok - just tried with the vlan, commissioning works (at least, got past the bit where it got stuck) [16:56] babbageclunk: but using "Add Hardware" like that bypasses most of that - it should add them as new and the start commissioning [16:57] babbageclunk: awesome! [16:57] dimitern: just adding the gateway/dns/space back on to make sure. [16:57] dimitern: thanks for all of the detective work! [16:58] babbageclunk: :) glad to help [16:58] ok I think it's beer o'clock already [16:59] * dimitern eod-s [17:00] dimitern: yay, still works with the extra settings. Enjoy that beer! [17:00] babbageclunk: nice! I'll leave you to it then :) cheers! [17:17] we don't support any i386 in 2.x correct? [17:33] redir: correct [17:34] tx natefinch [17:44] Bug #1571053 opened: container networking lxd 'invalid parent device' [17:44] Bug #1575283 opened: Juju 2 status doesn't show error reason in default format [18:12] yay, headed my way: https://weather.com/weather/radar/interactive/l/USMO0105:1:US?layer=radarConus&zoom=7 [18:21] mgz: sinzui: did something change on the merge bot recently? It looks like LXD is installed but not configured, as I tried to land a change and I got a lot of "LXD is not configured" failures. [18:21] which I don't *think* I touched myself [18:21] (I am working in that area, so I might have tweaked something and not realized it) [18:22] jzm: for merging? why yes we did. We switched back to xenial images when the AMIs appeared [18:22] ah, at least fwereade's patch failed as well [18:23] sinzui: so it looks like the merge bot is rejecting patches now because as it tests the LXD provider, it finds that the local LXD is unconfigured [18:23] jam: So either we revert to trusty, or we we change someting to configure LXD. I am still operating on the assumption LXD doesn't just work [18:25] jam: Some trickery is need to configure LXD without a prompt. I think reverting to trust is safest for now [18:25] sinzui: well, I'm heading to bed, unfortunately. Most expedient is probably to revert to trusty [18:25] jam: I can rety your branch in a few minutes [18:25] dpkg-reconfigure -p medium still prompts you ? [18:26] (i can see that it could, I just noticed that the dpkg-reconfigure changed, and our prompt helping you to get working changed as well) [18:31] jam: katco fwereade : your merges are re-queued. The landing bot will use trusty images again because xenial images don't come with a working lxd, and "lxd init" required a interactive prompt [18:31] sinzui: ty, was wondering about that [18:32] sinzui: so 'lxd init' is the wrong thing to run, as that won't setup lxdbr0, 'dpkg-reconfigure -p medium lxd' is apparently the right thing to run, but it might also need a prompt [18:32] sinzui, jam: is this something we need to ping the lxd guys about? [18:32] jam: does -p medium lxd not require interactive? [18:33] sinzui, jam: it prompts me [18:33] sinzui: not sure. '-p' seems to change what level of questions get asked (so I'm guessing dpkg-reconfigure asks you to enter the subnet, but -p medium doesn't [18:33] jam :( one question is enough to kill a provisioning script [18:34] jam: does the failure look like "initialisation.go:94 configuring zfs failed with exit status 1: LXD init cannot be used at this time"? [18:34] sinzui: dpkg-reconfigure -p high doesnt ask any questions, but I don't know if it works :) [18:35] jam: :) If it works, I can switch back to xenial in a few hours [18:35] tych0: ^^ [18:35] tych0: do you know if "dpkg-reconfigure -p high' will just use the auto-selected subnet? [18:36] frobware: the current failure we're seeing is different [18:36] frobware: but that also looks like one we should be tracking if you're seeing that as well. [18:36] jam: ok, it might be because my tip is a little behind upstream/master [18:36] not being able to configure ZFS shouldn't be fatal [18:37] jam: it may also be due to local changes I'm making in terms of detecting whether we're already setup (bridge-wise) [18:51] anyone know if there's any reason to support a TLS version less than 1.2? [18:59] jam: i don't know, but you can probably preseed it [19:03] Bug #1575310 opened: Add "juju status --verbose". [19:44] natefinch, gui, possibly [19:46] other than that, no [19:47] cmars: IE (of course) is the only browser that hasn't had 1.2 support for a long time. Wikipedia says you need IE 11 for 1.2 support to be on by default... and google says IE 11 is the lowest supported version for win 7+ [20:03] Bug #1575332 opened: add-cloud method not documented by "juju help" [20:33] Bug #1544890 changed: "ERROR the name of the model must be specified" when 'juju init' required <2.0-count> [21:58] I have to go pick up sick kid and vet supplies (For horse not kid); will be back online later [22:39] Bug #1467715 changed: worker/peergrouper: data race in package [22:39] Bug #1575400 opened: juju2: no maas nodes available: error message mentions 'zone=default' [22:43] cherylj: reviewed the add-model change. Ship it with a few small suggestions. [23:09] Bug # opened: 1575403, 1575405, 1575409, 1575410 [23:27] menn0: I'll have to address in a follow up PR. I was just merging feature branch I had created to test the requisite CI changes. [23:27] cherylj: no worries [23:39] Bug #1575409 changed: status hides container error in tabular format [23:39] Bug #1575410 changed: juju2-beta5: tools mismatch error on lxd container