/srv/irclogs.ubuntu.com/2016/04/26/#juju-dev.txt

redirwallyworld: re the trusty tests, while looking to remove them, I recall that I left them in as they also test i386 constraints, which isn't supported in xenial anymore, IIUC>00:53
redirwallyworld: with that in mind do you still want them removed?00:53
wallyworldredir: yeah, 386 is no longer supported in 2.000:54
anastasiamac\o/00:54
redirahh even in trusty and older. OK makes sense.00:54
redirtx00:54
redirwallyworld: how about the even older quantal tests? nix them too?01:09
wallyworldredir: depends. we tend to use quantal as a series for certain things in tests where we want to be sure the system on which the test runs is not that series01:10
redirok01:10
* redir leaves them01:10
redironly removes trusty01:10
redirbbiab01:32
mupBug #1574272 changed: Juju agent uninstalls itself while adding machine <juju-core:Incomplete> <https://launchpad.net/bugs/1574272>02:06
mupBug #1574272 opened: Juju agent uninstalls itself while adding machine <juju-core:Incomplete> <https://launchpad.net/bugs/1574272>02:12
mupBug #1574272 changed: Juju agent uninstalls itself while adding machine <juju-core:Incomplete> <https://launchpad.net/bugs/1574272>02:21
natefinchsingle character code review anyone?04:16
natefinchhttp://reviews.vapour.ws/r/4704/04:16
natefinchwallyworld, thumper, menn0 ^04:17
wallyworldlooking04:17
wallyworldnatefinch: now if that facade were not done separately and differently to every other facade it would not have missed having its version bumped04:19
wallyworldnatefinch: and there are tests that check that all api and apiserver facades marry up04:19
natefinchwallyworld: yeah.04:19
wallyworldwhich also do not pick this up04:19
natefinchwallyworld: the lack of tests is the real problem here, I think.04:20
wallyworldwe have tests04:20
wallyworldbut because this was done differently they did not pick it up04:20
natefinchwallyworld: I mean, yes, also the fact it's different is bad04:20
natefinchI don't think it's necessarily bad to component1/api  component2/api    rather that api/component1 api/component2 ... it's just that we have a mix, which is confusing, and this code is missing tests that should have picked up the mismatch of version numbers.04:22
natefinchanyway, bedtime for me.04:22
mupBug #1574949 opened: setting a valid config setting should not warn <landscape> <juju-core:New> <https://launchpad.net/bugs/1574949>04:39
redir_wallyworld: http://reviews.vapour.ws/r/4691/diff/1-2/ updated per earlier conversation04:47
wallyworldgreat ty04:48
wallyworldredir: looks good, just missing that extra test04:49
redir_there are tests04:49
redir_wallyworld: https://github.com/juju/juju/blob/master/environs/config/config_test.go#L185504:51
redir_or it is covered04:51
wallyworldredir: ok, np, i was just going by the diff, let me check the link you just posted04:51
wallyworldredir: those tests don't check that FallbackLtsSeries is used as expected04:53
wallyworldthat's the missing test i think we need04:53
redir_i added a couple using patch executable, but they didn't cover anything additional so I didn't commit them04:54
* redir_ looks again04:55
wallyworldredir: i guess the fallback series is only used to set testing FakeLtsSeries so it doesn't really need any more tests05:00
wallyworldi just wanted to ensure that the now exported FallbackLtsSeries was used as expected now that it is exported05:01
redir_got it committing.05:03
redir_wallyworld: http://reviews.vapour.ws/r/4691/diff/2-3/ adds tests to ensure the exported variable is used05:10
wallyworldok, ta05:10
redir_they don't add any coverage however.05:10
redir_I am going eod... but will merge in the morning if it looks good to you.05:11
wallyworldredir: ok, np. looking at the tests, it was not so much testing the setting of FallbaclLtsSeries (as you say that adds no extra coverage), but more testing places that used it got waht we expected ie was it logic to use fallbacklts wired up correctly. i think since we just use it to set a testing variable elsewhere, we can do without these tests05:13
redir_ok reverting those tests:)05:15
redir_OK they are gone again wallyworld :)05:16
wallyworldty, sorry for misunderstanding05:17
wallyworldlgtm :-)05:17
redir_np, It is getting late here, so I may have been unclear.05:19
redir_I can ship it now if you want but I am not waiting around for the merge bot05:19
redir_wallyworld: ^05:20
wallyworldredir: yep, np. i'll check the progress05:20
redir_k, the bot's got it05:25
redir_nite05:25
redir_btw, when do sleep?05:25
redir_when do you sleep? even...05:25
mupBug #1574963 opened: juju2 lxd launch hostname reverse lookup inconsistent <kanban-cross-team> <landscape> <juju-core:New> <https://launchpad.net/bugs/1574963>05:54
thumpernight all06:17
frobwaredimitern: ping, running 10mins late...07:58
dimiternfrobware: no worries, thanks for reminding me :)07:58
dimiternfrobware: ping me when you're there08:01
frobwaredimitern: let's go08:08
dimiternfrobware: omw08:09
frobwaredimitern: let's try after standup08:16
TheMuemorning08:27
voidspacedimitern: I've replied to two of your points on the review of my branch: http://reviews.vapour.ws/r/4700/08:50
voidspacedimitern: the error 500 on the subnet is unrelated to the points you make and could be a genuine error08:50
voidspacedimitern: and one of the points (about creating the primary NIC) I'll talk to you about in standup08:50
voidspacedimitern: the other point about null types I'm fixing.08:51
voidspacedimitern: I'd like to see the output of subnets read on your maas server - the error appears to be that subnet "2" doesn't exist08:59
voidspacedimitern: frobware: standup?09:01
babbageclunkdimitern: When setting up a VLAN and subnet, if I want a machine on that VLAN to be able to get to the internet (or anywhere outside that subnet) I need to set up a default gateway, right?10:53
babbageclunkdimitern: Should that be the address of the MAAS controller? Does that mean that it needs another interface that I can assign that VLAN and subnet?10:54
babbageclunkdimitern: reading docs10:58
dimiternbabbageclunk: yeah, the controller usually has access to all VLANs (has NICs on each)11:02
dimiternbabbageclunk: and also acts as a gateway for the VLAN's subnets11:02
babbageclunkOk, so I need to add an interface on the controller for each VLAN I create?11:03
dimiternbabbageclunk: to make the setup closer to the real world, add gateways to each subnet pointing to the respective cluster addresses on that subnet11:03
babbageclunkdimitern: ?11:04
babbageclunkdimitern: cluster addresses? The interface on the controller?11:04
dimiternbabbageclunk: yeah, one NIC for each subnet on the controller is the easiest thing, both for physical and vlan subnets11:04
dimiternbabbageclunk: yep11:04
babbageclunkdimitern: ok, thanks.11:05
dimiternbabbageclunk: np11:05
babbageclunkdimitern: I get an error when I try to create a subnet: malformed array literal: ""11:15
babbageclunkLINE 1: ...', 5005, 1, '10.10.0.0/24', 2, '10.10.0.1'::inet, '', true) ...11:15
babbageclunk                                                             ^11:15
babbageclunkDETAIL:  Array value must start with "{" or dimension information.11:15
dimiternbabbageclunk: what did you run to get this?11:16
babbageclunkJust trying to create the subnet in the MAAS admin.11:16
frobwarebabbageclunk: is that in 2.0?11:18
dimiternbabbageclunk: I mean can you paste the command line?11:18
babbageclunkI made a VLAN with the VID and name 10, and then tried to create a 10.10.0.0/24 subnet11:18
babbageclunkI wasn't using the command line, hang on - I'll try that.11:18
dimiternbabbageclunk: have you tried setting up the VLAN NICs on the cluster first and rebooting?11:18
frobwaredimitern: I thought that detection only happened at install time11:19
dimiternooh wow11:19
* dimitern haven't see the web ui in 2.0 allows that11:19
* frobware is jealous in 1.9 land...11:19
babbageclunkdimitern: Added the nic and rebooted, but I can't set the vlan tag on the new nic.11:19
dimiternfrobware: unless it changed, it also happens at boot11:20
frobwaredimitern: nice!11:20
dimiternbabbageclunk: check out this http://juju-sapphire.github.io/post/MAAS%20Spaces%20Demo/11:20
frobwaredimitern: I was only scripting that a few days ago11:20
dimiternfrobware: ah, ok11:21
babbageclunkdimitern: I thought maybe that was because I'd need to assign a subnet to the nic first and then it would set it to the VLAN for that subnet11:22
babbageclunkdimitern: but having trouble creating the subnet.11:22
dimiternfrobware: it seems to work, as I've recently "gained" a fabric-2 with a 192.168.122.0/24 subnet on it after installing libvird-bin and rebooting11:22
frobwaredimitern: but this is 2.0 only, yes?11:23
babbageclunkdimitern: oh yeah - I've got that as well, on fabric-1 (I assume because I installed libvirt-bin before adding the new nic)11:24
dimiternbabbageclunk: nope, the creation order is: fabrics and spaces, vlan on the fabric, subnet on a vlan and space11:24
dimiternfrobware: it might be 2.0 only, now that I think of it11:24
babbageclunkdimitern: Ok - I've got fabrics 0 (original nic on the controller), 1 (bridge from libvirt-bin), 2 (new nic I added to be on the new vlan)11:25
babbageclunkdimitern: I've got a private space11:25
babbageclunkdimitern: Trying to add a subnet in the private space in the 10 vlan.11:26
dimiternbabbageclunk: but, you either do A or B, no need for both; A) edit /e/n/i on the maas controller to e.g. add eth0.11, eth0.12, eth0.13 vlan nics, which are statically configured with e.g. 10.11.1.1/24, 10.12.1.1/24, 10.13.1.1/24 (so maas can detect the subnet along with the vlan, and from the parent device eth0 it can detect the fabric)11:26
babbageclunkdimitern: Does the vlan need a primary rack?11:26
dimiternor B) create vlans, then spaces, then subnets within the vlans and spaces manually with the CLI (or if it works - web ui)11:27
dimiternbabbageclunk: I expect so, but you should only have one anyway11:27
babbageclunkdimitern: Ah - when I added the nic did I need to add a new network in VMM as well?11:28
voidspacedimitern: I fixed the "nil" issue by the way, so if you could complete the review sometime it would be appreciated11:28
voidspacedimitern: http://reviews.vapour.ws/r/4700/11:28
voidspacedoing some multi-nic testing.11:28
voidspacewell, attempting to without screwing up maas :-/11:28
dimiternbabbageclunk: not necessarily - adding new NICs on the VM is only needed if you need more than one physycal NIC to appear on the VM when commissioned by MAAS11:28
babbageclunkdimitern: but how else would I be able to make sure the controller had an address in the right vlan?11:29
dimiternbabbageclunk: you can add VLANs on top of existing physycal NICs11:29
babbageclunkdimitern: how?11:30
babbageclunkdimitern: do I do that in the UI, or somewhere else?11:30
dimiternvoidspace: ta, will look in a bit11:30
dimiternbabbageclunk: see for example my /e/n/i from the maas controller machine: http://paste.ubuntu.com/16061991/11:30
babbageclunkdimitern: (Sorry for all the questions - I feel like there's a lot of knowledge that I'm missing that everyone else has forgotten they know.)11:31
dimiternbabbageclunk: if I add e.g. "eth1.42" to that, with address 10.42.0.1/24 and vlan-raw-device eth1, then reboot maas should have created a 10.42.0.0/24 subnet linked to a newly created 42 on the same fabric eth1 is on11:32
babbageclunkdimitern: ok, that makes sense - I thought that I could do that through the UI but had missed a step.11:33
dimiternbabbageclunk: well, you could also to that from the CLI (possibly the UI as well), but why bother if MAAS does it for you?11:33
dimiternbabbageclunk: once you have it working (the auto-detected way), you could try to replicate it manually, if you're interested11:34
babbageclunkdimitern: ok, I'll do that.11:35
dimiternbabbageclunk: here's some more info that can help shed some light on the manual approach: http://blog.naydenov.net/2016/01/maas-setup-deploying-openstack-on-maas-1-9-with-juju/11:36
dimiternjust you'd have to adapt it for the 2.0 API changes11:37
babbageclunkdimitern: ok, that's great, thanks.11:38
katcofwereade: jam: hey, will you be able to make the planning meeting in ~20m?11:41
katcofwereade: doh, you responded. sorry to pester ;)11:41
fwereadekatco, thanks for the reminder, though :)11:42
katcofwereade: ha ;p definitely time for some coffee11:42
jamkatco: yes11:47
dimiternvoidspace: reviewed11:51
katcojam: awesome, ty!11:51
katcofrobware: hey were you able to get those estimates? i see you requested access to the spreadsheet11:52
voidspacedimitern: ta11:58
frobwarekatco: nope. don't have any meaningful numbers12:00
dimiternvoidspace: live testing your branch now - I think I found the cause of "subnet not found"12:08
dimiternvoidspace:  PUT http://10.14.0.1/MAAS/api/2.0//MAAS/api/2.0/nodes/4y3h8x/interfaces/112/, para12:08
dimiternms:12:08
dimitern 12:08
dimiternwe pass no params (I'd expect the vlan id there) when updating interface12:09
voidspacedimitern: no vlan id12:11
voidspacedimitern: ok, I'll look for that12:12
dimiternvoidspace: so what I think happens is maas tries to find a subnet with id=2 in the vlan with id=0 (which does not exist)12:13
voidspacedimitern: right12:13
voidspacedimitern: don't suppose you have any logging where it's coming from12:13
voidspacedimitern: PUT is for update - which only happens on the device create I think12:13
dimiternvoidspace: ah, just killed the controller :/12:14
voidspaceheh12:14
voidspaceI'm looking12:14
dimiternvoidspace: but the PUT 500 was right after the device was created12:14
voidspacedimitern: so in gomaasapi/machine.go:310 you can see the code12:15
voidspacedimitern: it's only adding a VLAN if the iface and Subnet don't match - which may well be incorrect12:16
* dimitern takes a look12:16
voidspacedimitern: that should probably be unconditional12:16
dimiternvoidspace: it makes sense not to set the vlan param if it's empty12:19
dimiternvoidspace: as you might want to e.g. just rename the interface or change its mac12:20
voidspacethat's where the vlan should be added12:20
voidspacedimitern: should it be vlan id or vlan vid12:21
voidspacewe're setting id, which is consistent with the other entitites (use id)12:21
voidspacein interface.go:13712:21
dimiternvoidspace: it should be the vlan id, not vid (or alternatively vlan="vid:<vid>")12:22
voidspaceyeah12:22
voidspacewell, there's definitely code to handle it in that code path12:22
voidspacedimitern: actually, that code doesn't specify a subnet (the update call), so it's probably not that one12:24
voidspacedimitern: let me look at LinkSubnet - but that's a POST not a PUT12:24
dimiternvoidspace: the update call sets the vlan, the link-subnet call sets the subnet and mode12:25
voidspacedimitern: right, but you said the error was specifying a subnet and no vlan12:25
voidspacedimitern: and the update call (the PUT) doesn't specify a subnet12:26
voidspacedimitern: so it can't be that12:26
voidspaceI need a setup I can try this on12:27
voidspaceI'm shooting in the dark here12:27
voidspacelunch first12:27
dimiternvoidspace: sorry, I didn't say we need subnet and vlan in the link-subnet call12:27
voidspacedimitern: no, you said you'd expect to see a vlan id in the update interface call12:28
dimiternvoidspace: however, not setting a valid vlan on the interface won't allow you to link any subnet to it12:28
voidspacedimitern: but it doesn't take that parameter12:28
voidspaceoh, it does12:29
voidspacedimitern: right12:29
dimiternvoidspace: yep12:29
voidspacedimitern: however, we do have code to set the vlan id - so it's weird that it's not getting there12:29
* dimitern *facepalms*12:31
voidspacego on...12:32
babbageclunkdimitern: I've added the vlan in ENI, but it doesn't show up in the controller. Do I need to do something (other than rebooting) after changing ENI?12:32
dimiternvoidspace: of course it won't work :) we're not actaully updating device's eth0 vlan at all12:33
voidspacedimitern: explain12:33
dimiternvoidspace: we're creating a device with a mac, then reading back the interface_set from it, skipping the primary in the loop around 2308 in environ.go, and only recording its id12:34
voidspacedimitern: but machine.CreateDevice does the update12:34
voidspacedimitern: in gomaasapi12:34
mupBug #1574949 changed: setting a valid config setting should not warn <landscape> <juju-core:New> <https://launchpad.net/bugs/1574949>12:34
voidspacedimitern: which is why we skip it12:35
dimiternvoidspace: yeah, it looks like Update() uses the subnet linked to the primary device NIC to figure out which vlan id should use12:36
dimiternvoidspace: sorry, not Update(), but CreateDevice()12:36
dimiternvoidspace: so the problem is in line 2272 in environs.go, not gomaasapi12:37
voidspacedimitern: ah, so CIDR is not unique - so a simple hash is not sufficient12:38
voidspacedimitern: it's CIDR/vlan id pairs12:38
voidspacedimitern: it pulls out the wrong subnet12:39
voidspace?12:39
dimiternvoidspace: cidr is unique12:39
dimiternvoidspace: but I suspect preparedInfo['eth0'].CIDR is empty12:39
voidspacedimitern: the maas 1.9 code is doing the same12:39
dimiternvoidspace: hmmm.. true12:40
dimiternmystery12:40
* voidspace lunches12:40
babbageclunkdimitern: ooh, while he's gone, can we hangout again? I'm hopelessly confused. (Unless you also want to lunch?)12:41
dimiternbabbageclunk: can you give me 10m ?12:42
babbageclunkdimitern: if you do I can go for a run now and we can chat after?12:42
dimiternbabbageclunk: sounds better - please ping me when you're back then12:42
babbageclunkdimitern: ok, thanks12:42
dimiternvoidspace: the problem is using Spaces(), followed by Subnets() on each does not give you the vlan of the subnet12:44
dimiternvoidspace: and CreateMachineDeviceArgs.Validate must also check a.Subnet.VLAN() != nil12:48
voidspacedimitern: Space.Subnets() doesn't set vlan properly12:48
voidspacethat's a bug if true12:48
voidspaceanyway, really going on lunch12:48
voidspace(you may be right)12:48
dimiternvoidspace: enjoy :)12:49
mupBug #1573410 changed: trusty juju 1.25.5 having issues deploying xenial lxc containers <canonical-bootstack> <juju-core:Invalid> <https://launchpad.net/bugs/1573410>13:04
babbageclunkdimitern: back!13:50
dimiternbabbageclunk: hey, so let's use the standup HO?13:50
katcofrobware: hey, thanks for showing up. sorry it wasn't useful to you.13:51
babbageclunkdimitern: yup13:51
mattywkatco, ping?14:04
katcomattyw: pong14:04
mattywkatco, hey there, I'm struggling to deploy local charms on beta6 (it also fails on master) I'm getting connection timed out on ec2 but nothing more useful in the log than that (charmstore charms work fine) shall I raise a bug?14:05
katcomattyw: s/charms/resources/ ? or do you really mean charms?14:06
mattywkatco, charms14:08
katcomattyw: does it work on a different substrate? i.e. is it a fluke?14:08
mattywkatco, it did work on lxd, but I had other problems on lxd so had to stop using it14:09
katcomattyw: this seems like something CI would be catching... hm14:09
katcomattyw: i don't think it would hurt to raise a bug, but the inability to deploy a local charm seems very severe and something everyone would be seeing14:10
katcosinzui: cherylj: mgz: is ci seeing anything like this?14:11
sinzuikatco: matty, ci deploys many 10s of local charms and is not seeing an issue14:13
mgzmattyw: in good news, lxd will probably work for you now :)14:13
mattywmgz, lxd was sort of working, I was trying to install lxd in lxd though, and that was giving me problems14:14
sinzuimattyw: this is an example of a deploy froma  few hours ago "juju --show-log deploy"14:15
cheryljmattyw: what connection is timing out?  the actual deploy command?14:15
abentleysinzui: I see no instances in eu-west-1.  Did you do cleanup, or did we finally get a clean run?14:15
mattywcherylj, seems to be yeah14:15
sinzuiabentley: I have too many disasters to deal with to clean up. I guess all went well14:15
cheryljmattyw: can you paste the juju deploy --debug output?14:16
sinzuiabentley: I saw the region being used yesterday14:16
mattywcherylj,  https://pastebin.canonical.com/155238/14:16
abentleysinzui: Yes.  industrial tests were still running last I checked, but it looked reasonable then.14:16
cheryljmattyw: and this is ec2?14:17
mattywcherylj, yep14:17
cheryljmattyw: yeah, it's not filtering lxd addresses14:18
cheryljjam has a PR up for that14:18
dimiternbabbageclunk: http://rickardnobel.se/the-vlan-802-1q-tag-part-1/14:18
mattywcherylj, I don't believe lxd is being used in this situation14:18
cheryljmattyw: can you ssh to your controller and run an ifconfig?14:18
cheryljmattyw: I'm guessing you have a bridge device on the controller that's not being filtered, but I could be wrong14:19
babbageclunkdimitern: you froze up! Are you ok over there? Thanks heaps anyway!14:19
dimiternbabbageclunk: yeah, to it appeared you dropped out btw14:19
dimiterns/it/me/14:19
dimiternargh14:19
dimiterns/it/me it/ :)14:19
cheryljmattyw: can you also go to your ec2 dashboard and see what IPs that machine has assigned to it?14:20
babbageclunkdimitern: ah well, I think we're finished anyway - I'll read through that and get the vlans set up in my maas.14:20
babbageclunkThanks!14:20
dimiternbabbageclunk: I've sent you a few links in prvmsg not to flood the channel :)14:22
mattywcherylj, so I can't seem to ssh into it anymore, seems like none of the juju commands work, I've found the instance in ec2 though14:23
mattywcherylj, it's eu-west, I guess it might be a not commonly used region14:24
cheryljmattyw: yeah, that doesn't surprise me that you couldn't ssh through juju14:24
dooferladfrobware: https://github.com/juju/juju/pull/5281 may look very similar to something you reviewed yesterday... if you could take a quick look.14:26
mattywcherylj, how do I ssh to the controller now?14:26
cheryljmattyw: can you ssh directly to ubuntu@54.195.25.5 ?14:27
dimiternmattyw: try juju ssh --proxy 0 ?14:28
mattywcherylj, yes: https://pastebin.canonical.com/155242/14:28
mattywdimitern, ERROR machine 0 not found (not found)14:28
cheryljhmm, I guess that region does actually use a 10. internal IP14:28
cheryljI kinda figured it was a bridge IP14:28
dimiternmattyw: juju switch admin then the above?14:29
mattywdimitern, that seems to work14:29
dimiternor more verbosely juju ssh --proxy -m admin 014:29
cheryljmattyw: see anything interesting in machine-0.log?14:30
dimiternwell, for machine 0 of the admin model you shouldn't need to pass --proxy, but for any other machine (or container on machine 0), --proxy is needed unless you can ping the private address of the machine directly14:30
mattywcherylj, nothing really14:33
cheryljmattyw: and the deploy command just hangs in the POST?14:34
mattywcherylj, yeah14:35
mattywcherylj, I've just discovered the charm is >600mb, I wonder If I just have to wait for ages, and expect some errors to happen14:35
cheryljcould be14:35
cheryljtoo bad there's not a progress indicator14:35
cheryljI mean, you could watch df on the controller to see if something is happening14:36
cheryljor use jnettop14:36
mattywcherylj, yeah will do, I must admit to not realising the charm was that big14:38
cheryljmattyw: yeah, that is surprising14:39
jamcherylj: mattyw: I'm trying to get that landed today14:45
mattywcherylj, yep panic over, just took ages14:54
cheryljyay!14:54
cherylj(well, not yay for taking ages, but yay to not being a regression)14:54
mattywcherylj, thanks for sticking with me14:54
cheryljmattyw: of course :)14:54
mupBug #1575229 opened: juju/utils/GetAddressesForInterface only returns IPv4 addresses <ipv6> <lxd> <juju-core:Triaged> <https://launchpad.net/bugs/1575229>15:02
rcjcherylj, sinzui, alexisb: xenial amis are available15:07
alexisbrcj, awesome15:07
sinzuiwee15:09
sinzuithank you cherylj , I will update the merge job15:10
mgzrcj: <315:10
voidspacedimitern: you're wrong by the way - I fetch the spaces from MAAS directly there, and it does get VLAN information15:10
voidspacedimitern: those are gomaasapi.Subnet not network.SubnetInfo15:11
dimiternvoidspace: oh, it does? hmm..15:11
voidspacedimitern: well, I'm pretty sure you're wrong anyway15:11
mupBug #1575229 changed: juju/utils/GetAddressesForInterface only returns IPv4 addresses <ipv6> <lxd> <juju-core:Triaged> <https://launchpad.net/bugs/1575229>15:11
dimiternvoidspace: well, CreateMachineDeviceArgs.Validate has to check Subnet.VLAN is != nil15:11
voidspacedimitern: that's true enough, but it's not the cause of the error15:12
voidspacestill working on a multi-nic setup15:12
dimiternvoidspace: well, something is wrong with the 2.0 implementation if AllocateContainerAddresses or something in gomaasapi, as the 1.0 API path still works15:13
voidspacedimitern: sure :-)15:14
voidspacedimitern: once I can reproduce I'm confident I can track it down15:14
dimiternvoidspace: +115:15
voidspacedimitern: I think your diagnosis about the VLAN might be right15:15
voidspacedimitern: I just can't see how from the code15:15
voidspacedimitern: but some debugging info should reveal it15:15
dimiternvoidspace: I'd suggest commenting out the deferred device deletion on failure to inspect it15:17
frobwaredooferlad: just a couple of coments, but LGTM15:20
katcocherylj: got a sec to chat about a bug?15:21
cheryljkatco: sure15:21
katcocherylj: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=115:21
frobwarevoidspace, babbageclunk, dooferlad, dimitern: please remind me, who ran into the iface names too long issues yesterday? And was a bug raised for this?15:23
mupBug #1575229 opened: juju/utils/GetAddressesForInterface only returns IPv4 addresses <ipv6> <lxd> <juju-core:Triaged> <https://launchpad.net/bugs/1575229>15:23
voidspacefrobware: I thought it was dimitern - at least he told me about it :-)15:24
babbageclunkdimitern: I deleted a node from my maas (because I screwed up its network) and want to recommission it, but it won't pxe boot, always falls back to the disk.15:24
babbageclunkfrobware: yeah, I think it was dimitern15:25
voidspacebabbageclunk: check the boot order15:25
frobwarebabbageclunk: boot order in the VM settings15:25
babbageclunkvoidspace, frobware: thanks, but I checked that. It shows that it's trying to boot from the network but nothing happens.15:25
frobwaredimitern: ^^ - please could you raise a bug for the iface name issue15:26
alexisbfwereade, I will be a bit late for our 1x115:27
voidspacebabbageclunk: weird. And the subnet that exists is one with managed dhcp?15:27
babbageclunkvoidspace: ooh, checking15:28
babbageclunkvoidspace: hmm, I don't know where to check that15:31
voidspacebabbageclunk: in the maas ui - networks15:31
fwereadealexisb, chatting to dimitern, let me know when you're ready15:31
voidspacebabbageclunk: find the subnet you're interested in and click on "untagged" (vlan)15:31
voidspacebabbageclunk: and it will tell you if DHCP is enabled (you need it enabled to pxe boot)15:32
bogdanteleagaanybody up for the fastest review of the week? http://reviews.vapour.ws/r/4710/15:32
babbageclunkvoidspace: yup, DHCP is enabled on the untagged VLAN in fabric-015:32
perrito666bogdanteleaga: shipit15:33
=== mhilton-lunch is now known as mhilton
voidspacebabbageclunk: :-/15:34
voidspacebabbageclunk: clone the vm and commission a new one instead15:34
voidspacebabbageclunk: maybe maas won't commission a deleted machine?15:35
voidspacebabbageclunk: or ask in #maas15:35
babbageclunkvoidspace: might try creating a new one and see if that gets picked up ok15:35
babbageclunkvoidspace: nope - ok, something wrong with the controller rather than the node.15:39
voidspacebabbageclunk: :-/15:40
babbageclunkvoidspace: undoing the vlan change to see if that helps15:41
voidspacedimitern: ok, I can reproduce15:41
mupBug #1575245 opened: Juju 2.0 LXD install instructions don't work on trusty <juju-core:New> <https://launchpad.net/bugs/1575245>15:44
dimiternvoidspace: \o/15:45
dimiternvoidspace: what was it?15:45
voidspacedimitern: I said I can repro it15:46
voidspacedimitern: working out what causes it is the next step15:46
babbageclunkvoidspace, dimitern: what the crap - my ENI is empty now! Just has loopback in it now. That might explain the problem (although I don't know how that happened).15:47
dimiternfrobware: which iface name issue? about biosdevnames and enxxaabbccddeef0 ?15:47
frobwaredimitern: yep15:47
mupBug #1575245 changed: Juju 2.0 LXD install instructions don't work on trusty <juju-core:New> <https://launchpad.net/bugs/1575245>15:47
voidspacebabbageclunk: hah, nice15:47
babbageclunkyay, lucky I put it in a pastebin while chatting to dimitern15:48
dimiternfrobware: I did yesterday - bug 157207015:48
mupBug #1572070: MAAS 2.0 cannot link physical device interfaces to tagged vlans, breaking juju 2.0 multi-NIC containers <juju> <MAAS:Fix Committed by blake-rouse> <MAAS 1.9:Fix Committed by blake-rouse> <https://launchpad.net/bugs/1572070>15:48
dimiternbabbageclunk: sorry, I was talking to fwereade - catching up on scrollback15:49
babbageclunkno worries - it was a false alarm, my ENI is still there.15:50
dimiternbabbageclunk: ah, good! :)15:50
frobwaredimitern: is that the right bug#?15:50
dimiternfrobware: it's against maas only, not juju15:51
frobwaredimitern: I'm confused. How is that related to enxxaabbccddeef0 and our resultant bridge name?15:52
* dimitern *d'oh*15:52
dimiternfrobware: sorry - looking more carefully now15:52
dimiternfrobware: there it is - bug 157207015:53
mupBug #1572070: MAAS 2.0 cannot link physical device interfaces to tagged vlans, breaking juju 2.0 multi-NIC containers <juju> <MAAS:Fix Committed by blake-rouse> <MAAS 1.9:Fix Committed by blake-rouse> <https://launchpad.net/bugs/1572070>15:53
dimiternargh15:53
dimiternbug 157477115:53
mupBug #1574771: MAAS/curtin generate invalid /e/n/i and failed deployment for nodes with long (biosdevname) interface names, which in turn have VLANs <networking> <robustness> <MAAS:Invalid> <MAAS 1.9:Invalid> <https://launchpad.net/bugs/1574771>15:53
mupBug #1575245 opened: Juju 2.0 LXD install instructions don't work on trusty <juju-core:New> <https://launchpad.net/bugs/1575245>15:53
frobwarealexisb: this ^^ was the bug I mentioned earlier (bug 1574771)15:54
babbageclunkdimitern: yup - if I remove the vlan, I can commission. If I add it back in, I can't.16:07
dimiternbabbageclunk: add/remove from the maas machine?16:08
dimiternbabbageclunk: I'm not sure what are you talking about :)16:11
natefinchsinzui: is it safe to drop code that is only compiled for older versions of Go?16:11
natefinch(older than 1.6)16:11
babbageclunkdimitern: sorry - add/remove it to/from ENI16:11
dimiternbabbageclunk: on the maas controller maching?16:12
dimiternmachine*16:12
babbageclunkdimitern: yup16:17
dimiternbabbageclunk: how does the commissioning fail?16:18
dimiternbabbageclunk: do you see the console of the VM node doing stuff?16:18
babbageclunkdimitern: I get this error: 0x040ee11916:19
dimiternbabbageclunk: so pxe is failing16:19
babbageclunkdimitern: yup16:19
babbageclunkdimitern: so presumably that means DHCP?16:19
dimiternbabbageclunk: yeah, among other things16:20
babbageclunkdimitern: could it be setting the dns servers on the subnet?16:20
babbageclunkdimitern: I don't know why that would cause it though.16:21
dimiternbabbageclunk: hmm.. let me check something here on my maas16:21
voidspacedimitern: so it's the call to machine.CreateDevice that errors - however as far as I can tell the juju side of that is correct (right subnet with the right vlan)16:22
voidspacedimitern: so instrumenting gomaasapi to see what it actually sends...16:22
dimiternvoidspace: it gets deeper .. :/16:22
dimiternbabbageclunk: how did you trigger the commissioning - via the UI or CLI?16:23
mupBug #1564622 changed: Suggest juju1 upon first use of juju2 if there is an existing JUJU_HOME dir <juju-release-support> <juju-core:Fix Released by natefinch> <https://launchpad.net/bugs/1564622>16:23
natefinchcherylj, katco: btw, got clarity from security... basically we just need to blacklist the RC4 ciphersuitem which is fairly straightforward... I'll add a function to juju/utils to create a default tls.Config and make sure we use that everywhere.16:23
voidspacedimitern: yeah, as far as I can tell the arguments to machine.CreateDevice are correct, and note that at this point it's the same code path as for a single nic16:23
voidspacedimitern: it hasn't even got into the part that's different!16:23
dimiternbabbageclunk: and what series was used - xenial or trusty?16:23
voidspacedimitern: it *could* still be a MAAS bug :-)16:24
dimiternvoidspace: hmm the difference between single-nic and multi-nic should only be the way we handle the primary nic (update the vlan + link-subnet)16:24
katconatefinch: cool, glad to hear it's manageable16:26
babbageclunkdimitern: xenial16:26
cheryljawesome, thanks for following up on that, natefinch16:27
babbageclunkdimitern: although it wouldn't ever get to that point.16:27
dimiternbabbageclunk: and that particular node has 1 NIC on the same bridge where the controller's managed interface is on?16:27
babbageclunkdimitern: I think so. What do you mean by managed interface?16:28
sinzuinatefinch: good question. We are wtill running windows and centos unit test with go 1.2 because they do not pass on go 1.6. I don't think you can require go 1.6 today16:28
dimiternbabbageclunk: how many NICs does your maas controller KVM has configured?16:29
natefinchsinzui: booooooooooooooooooooooooooo16:29
natefinchoooooo16:29
babbageclunkdimitern: 216:29
* natefinch sadface16:29
dimiternbabbageclunk: ok, and they are attached to difference virbrX bridges?16:29
babbageclunkdimitern: I added one when I thought I needed another for the vlan.16:29
dimiternbabbageclunk: is it possible that second bridge has DHCP enabled on it?\16:30
dimiternbabbageclunk: (last bridge you added)16:30
natefinchsinzui: are we building the binaries with go 1.6?16:30
natefinchsinzui: for centos and windows?16:30
sinzuinatefinch: yes16:30
babbageclunkdimitern: I'm not sure how to see what bridge it's connected to.16:30
natefinchsinzui: eq16:31
natefinchew16:31
sinzuinatefinch: we only build with golang 1.616:31
babbageclunkdimitern: in VMM? They're both on the maas2 network - I guess that's the bridge.16:31
dimiternbabbageclunk: if you have virt-manager installed and connected to qemu:///system it will be easy :)16:31
natefinchcherylj: this seems like a pretty bad thnig - our tests and our production code are being built with different versions of go.  which means we're not actually testing 100% what we're distributing16:31
dimiternbabbageclunk: ah, well - not quite the network is "backed" by a virbrX bridge, but it's a separate entity as far as libvirt is concerned16:32
babbageclunkdimitern: I'm going to remove the second nic anyway now - it's not needed.16:32
dimiternbabbageclunk: so yeah, how is that maas2 network configured in terms of dhcp, ipv4/ipv6, etc?16:32
dimiternbabbageclunk: that might help indeed :)_16:33
babbageclunkdimitern: IPv4, DCHP off, NAT on16:33
dimiternbabbageclunk: ok, that's correct16:34
dimiternbabbageclunk: then, if you go to the Nodes | Controllers | your maas controler16:34
babbageclunkdimitern: ok, I'll ditch the other one and put the vlan back, then try recommissioning.16:34
babbageclunkdimitern: yup16:35
babbageclunk?16:35
cheryljnatefinch: I'm not sure what you're referencing. Since we made the move to go 1.6, we use it everywhere.  The only exceptions are windows unit tests using go 1.6 are still non-voting, pending some changes from perrito666.  Oh and, centos unit tests too16:35
dimiternbabbageclunk: what does it show in the "served vlans" section and interfaces below it?16:35
sinzuinatefinch: I think you misunderstand what is happening. we switched to go 1.6 as core asked, but core is not fixing the test as QA asks16:35
dimiternbabbageclunk: sorry for the 20 questions :) I suspect the issue might be enabling DHCP on the newly added VLAN16:36
babbageclunkdimitern: no worries! I appreciate the help.16:36
babbageclunkdimitern:16:36
perrito666dimitern: both urls seem valid, but ill make the change16:37
babbageclunkdimitern: so served VLANs has fabric-0 (the nic) and fabric-1 (the bridge that got added when installing libvirt-bin)16:37
dimiternbabbageclunk: ok, so the first fabric-0 is the managed one, and its vlans should all have dhcp enabled16:38
babbageclunkdimitern: under Interfaces there are ens3 (original nic), ens9 (second nic I added) and virbr0 (the libvirt bridge)16:38
babbageclunkdimitern: yup (there's only the one VLAN, I removed the new one)16:39
babbageclunkdimitern: fabric-1's vlan has DHCP off16:39
dimiternbabbageclunk: ok, so the ens9 is the (only) managed interface, but with you should also see the newly added VLAN interface there as well - e.g. ens9.4216:39
dimiternbabbageclunk: that's fine though - it can't be on as libvirt also provides dhcp for 192.168.122.0/24 by default (virbr0)16:40
dimiternbabbageclunk: ah, you've removed the new vlan, ok16:41
dimiternbabbageclunk: and you're saying that adding it blocks commissioning - adding it how? manually with the CLI or editting /e/n/i and rebooting to let maas autodiscover it?16:41
babbageclunkdimitern: so ens9 is the managed interface? should I be adding the vlan to that interface instead of ens3?16:42
babbageclunkdimitern: the latter16:42
dimiternbabbageclunk: aha! that's it I think :)16:42
babbageclunkdimitern: (and then using the cli to set space/gateway/dns_16:43
dimiternbabbageclunk: ok, so that misconfigured ens3 was the issue I suspect16:43
babbageclunkdimitern: so you're suggesting - keep ens9, configure it in ENI, and add the vlan to that instead?16:44
natefinchsinzui, cherylj: sorry, had to step away... I just mean - we should prioritize work to get the windows and centos tests passing on 1.6.16:44
babbageclunkdimitern: ok, I can see that in your ENI from the pastebin16:44
dimiternbabbageclunk: sorry - which one did you add last - ens9 or ens3 ?16:45
babbageclunkens916:45
cheryljnatefinch: yeah, perrito666 was doing some work for windows16:45
sinzuinatefinch: yes we agree16:45
cheryljsinzui: is there a bug open for the centos / 1.6 failures?16:45
perrito666cherylj: sorry I thought I merged those last night but got flakytestwalled16:45
dimiternbabbageclunk: ah, ok then add the vlan to the ens3 and drop ens9 (on the vm as well)16:45
sinzuicherylj: bug 157088316:46
natefinchperrito666: I love that term, totally stealing it16:46
mupBug #1570883: imageSuite.TestEnsureImageExistsCallbackIncludesSourceURL fails on centos go 1.6 <centos> <ci> <go1.6> <jujuqc> <lxd> <juju-core:Triaged> <https://launchpad.net/bugs/1570883>16:46
cheryljsinzui: ah, I see it now16:46
dimiternbabbageclunk: as ens3 was the "original" managed interface :)16:46
babbageclunkdimitern: ok16:46
babbageclunkdimitern: (by virtue of being the only one016:46
babbageclunk)16:46
dimiternbabbageclunk: yep16:47
dimiternbabbageclunk: my maas-es setups all use dual-nic lxc containers for the controllers - one managed, one "external" unmanaged16:47
dimiternbabbageclunk: but that's not a requirement, just me making my life interesting :D16:48
babbageclunkdimitern: :)16:48
dimiternbabbageclunk: to be on the safe side, I'd delete that node which failed commissioning and re-add it16:51
dimiternbabbageclunk: for kvm-based nodes by far the easiest way is 'Add Hardware|V|' -> Chassis -> Power type 'virsh', Address like 'qemu+ssh:///...', etc. and set a prefix filter on the node names (e.g. mine are called maas-19-node-0, maas-18-node-2, etc.)16:53
babbageclunkdimitern: Ooh, that's nice.16:54
dimiternso I'd use 'maas-19-node-' as prefix16:54
babbageclunkdimitern: I tried adding a new one though, and it also wouldn't pxe boot16:54
dimiternbabbageclunk: yeah :) nice trick16:54
dimiternbabbageclunk: so a "new" node can't pxe boot just like that, maas won't allow it16:54
dimiternbabbageclunk: it needs to be turned on and try pxe booting (with priority over local disk boot)16:55
babbageclunkdimitern: yeah, I had done that.16:56
dimiternbabbageclunk: then it shows up as "new" and you can edit a few things like name, zone, etc. and then you "accept" it16:56
babbageclunkdimitern: ok - just tried with the vlan, commissioning works (at least, got past the bit where it got stuck)16:56
dimiternbabbageclunk: but using "Add Hardware" like that bypasses most of that - it should add them as new and the start commissioning16:56
dimiternbabbageclunk: awesome!16:57
babbageclunkdimitern: just adding the gateway/dns/space back on to make sure.16:57
babbageclunkdimitern: thanks for all of the detective work!16:57
dimiternbabbageclunk: :) glad to help16:58
dimiternok I think it's beer o'clock already16:58
* dimitern eod-s16:59
babbageclunkdimitern: yay, still works with the extra settings. Enjoy that beer!17:00
dimiternbabbageclunk: nice! I'll leave you to it then :) cheers!17:00
redirwe don't support any i386 in 2.x correct?17:17
natefinchredir: correct17:33
redirtx natefinch17:34
mupBug #1571053 opened: container networking lxd 'invalid parent device' <ci> <lxd> <juju-core:Triaged> <https://launchpad.net/bugs/1571053>17:44
mupBug #1575283 opened: Juju 2 status doesn't show error reason in default format <juju-core:New> <https://launchpad.net/bugs/1575283>17:44
katcoyay, headed my way: https://weather.com/weather/radar/interactive/l/USMO0105:1:US?layer=radarConus&zoom=718:12
jammgz: sinzui: did something change on the merge bot recently? It looks like LXD is installed but not configured, as I tried to land a change and I got a lot of "LXD is not configured" failures.18:21
jamwhich I don't *think* I touched myself18:21
jam(I am working in that area, so I might have tweaked something and not realized it)18:21
sinzuijzm: for merging? why yes we did. We switched back to xenial images when the AMIs appeared18:22
jamah, at least fwereade's patch failed as well18:22
jamsinzui: so it looks like the merge bot is rejecting patches now because as it tests the LXD provider, it finds that the local LXD is unconfigured18:23
sinzuijam: So either we revert to trusty, or we we change someting to configure LXD. I am still operating on the assumption LXD doesn't just work18:23
sinzuijam: Some trickery is need to configure LXD without a prompt. I think reverting to trust is safest for now18:25
jamsinzui: well, I'm heading to bed, unfortunately. Most expedient is probably to revert to trusty18:25
sinzuijam: I can rety your branch in a few minutes18:25
jamdpkg-reconfigure -p medium  still prompts you ?18:25
jam(i can see that it could, I just noticed that the dpkg-reconfigure changed, and our prompt helping you to get working changed as well)18:26
sinzuijam: katco fwereade : your merges are re-queued. The landing bot will use trusty images again because xenial images don't come with a working lxd, and "lxd init" required a interactive prompt18:31
katcosinzui: ty, was wondering about that18:31
jamsinzui: so 'lxd init' is the wrong thing to run, as that won't setup lxdbr0, 'dpkg-reconfigure -p medium lxd' is apparently the right thing to run, but it might also need a prompt18:32
natefinchsinzui, jam: is this something we need to ping the lxd guys about?18:32
sinzuijam: does -p medium lxd not require interactive?18:32
natefinchsinzui, jam: it prompts me18:33
jamsinzui: not sure. '-p' seems to change what level of questions get asked (so I'm guessing dpkg-reconfigure asks you to enter the subnet, but -p medium doesn't18:33
sinzuijam :( one question is enough to kill a provisioning script18:33
frobwarejam: does the failure look like "initialisation.go:94 configuring zfs failed with exit status 1: LXD init cannot be used at this time"?18:34
jamsinzui: dpkg-reconfigure -p high doesnt ask any questions, but I don't know if it works :)18:34
sinzuijam: :) If it works, I can switch back to xenial in a few hours18:35
jamtych0: ^^18:35
jamtych0: do you know if "dpkg-reconfigure -p high' will just use the auto-selected subnet?18:35
jamfrobware: the current failure we're seeing is different18:36
jamfrobware: but that also looks like one we should be tracking if you're seeing that as well.18:36
frobwarejam: ok, it might be because my tip is a little behind upstream/master18:36
jamnot being able to configure ZFS shouldn't be fatal18:36
frobwarejam: it may also be due to local changes I'm making in terms of detecting whether we're already setup (bridge-wise)18:37
natefinchanyone know if there's any reason to support a TLS version less than 1.2?18:51
tych0jam: i don't know, but you can probably preseed it18:59
mupBug #1575310 opened: Add "juju status --verbose". <feature> <juju-core:New> <https://launchpad.net/bugs/1575310>19:03
cmarsnatefinch, gui, possibly19:44
cmarsother than that, no19:46
natefinchcmars: IE (of course) is the only browser that hasn't had 1.2 support for a long time.  Wikipedia says you need IE 11 for 1.2 support to be on by default... and google says IE 11 is the lowest supported version for win 7+19:47
mupBug #1575332 opened: add-cloud method not documented by "juju help" <juju-core:New> <https://launchpad.net/bugs/1575332>20:03
mupBug #1544890 changed: "ERROR the name of the model must be specified" when 'juju init' required <2.0-count> <bootstrap> <juju-release-support> <juju-core:Invalid> <https://launchpad.net/bugs/1544890>20:33
alexisbI have to go pick up sick kid and vet supplies (For horse not kid); will be back online later21:58
mupBug #1467715 changed: worker/peergrouper: data race in package <ci> <intermittent-failure> <race-condition> <regression> <juju-core:Fix Released by menno.smits> <https://launchpad.net/bugs/1467715>22:39
mupBug #1575400 opened: juju2: no maas nodes available: error message mentions 'zone=default' <landscape> <juju-core:New> <https://launchpad.net/bugs/1575400>22:39
menn0cherylj: reviewed the add-model change. Ship it with a few small suggestions.22:43
mupBug # opened: 1575403, 1575405, 1575409, 157541023:09
cheryljmenn0: I'll have to address in a follow up PR.  I was just merging feature branch I had created to test the requisite CI changes.23:27
menn0cherylj: no worries23:27
mupBug #1575409 changed: status hides container error in tabular format <kanban-cross-team> <landscape> <juju-core:Triaged> <https://launchpad.net/bugs/1575409>23:39
mupBug #1575410 changed: juju2-beta5: tools mismatch error on lxd container <landscape> <juju-core:New> <https://launchpad.net/bugs/1575410>23:39

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!