=== Spideyman is now known as Spideyman_afk
=== Spideyman_afk is now known as Spideyman
[00:37] <shimey> help
=== bjf[afk] is now known as bjf
[00:49] <bigjools> shimey: hello
=== CyberJacob is now known as CyberJacob|Away
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
[07:06] <gmb> Morning jtv. Sorry I didn't email you to tell you where I'd put my (very vestigial) branch. Did you find it, and was it useful?
[07:07] <gmb> (I got totally sidetracked yesterday, so I'd only just started iterating on it)
[07:10] <jtv> gmb: I did not find it, therefore no, not very useful.  :)
[07:10] <gmb> jtv: Damn, and it was my only maas-test branch, too. Oh well. Apologies again. Somehow I suspect you managed fine :)
[07:11] <jtv> gmb: But this may put you in the position of the most understanding potential reviewer for https://code.launchpad.net/~jtv/maas-test/create-admin/+merge/194284 :)
[07:11] <jtv> gmb: not the lint one!?
[07:11] <jtv> Ah.
[07:11] <gmb> Ah, well played sir :)
[07:11] <gmb> jtv: Branch, not MP.
[07:11] <gmb> And so no
[07:11] <jtv> *Now* I see extract-functions.  Why didn't I see that?  I thought I looked about 5 hours ago.
[07:11] <gmb> It wasn't my only one
[07:12] <gmb> jtv, I don't think you'll have suffered for it; I only just got started teasing things apart.
[07:12] <jtv> gmb: careful with the "test base class" stuff — you may hit a nerve with this team.  :)
[07:13] <jtv> We have tons and tons of testcase classes, base and leaf and mixed, in the main tree and it's been getting to us.
[07:13] <gmb> jtv: Yeah, I know. That was my second iteration or so... I just wanted to pull it out of the test case for starters and then I could start massaging it into a nice utility.
[07:13] <gmb> jtv: Anyway, I'll review your branch now.
[07:13] <jtv> Thanks for both then.  :-)
=== CyberJacob|Away is now known as CyberJacob
[07:20] <jtv> rbasak: morning, and Q about uvtool... does it set up a fixed username/password or something?
[07:21] <rbasak> jtv: the username is "ubuntu" (cloud-init default in our cloud images).
[07:21] <rbasak> jtv: the password is unset by default. uvtool defaults to adding your ~/.ssh/id_rsa.pub if you have one and aren't overriding cloud-init userdata.
[07:22] <rbasak> jtv: or use --password to set a password for the ubuntu user
[07:27] <jtv> rbasak: oh, and I forget — how do I actually reach the VM, networking-wise?
[07:27] <jtv> I tried <vmname>.local but no luck.
[07:28] <jtv> Heyyy, that just started working.  Maybe it just needed a while!
[07:28] <jtv> Thanks for the pointers.
[07:28] <jtv> No, not pointers.  The info?
[07:30] <rbasak> jtv: .local seems unreliable and I might drop it. It takes ages for the instance to install avahi-daemon.
[07:30] <jtv> Oh, will the uvt-wait thingy actually wait for the hostname to show up?
[07:30] <bigjools> what does this do over vagrant, out of interest?
[07:31] <rbasak> jtv: the latest trunk adds "wait" and "ip" subcommands. It actually just peeks at /var/lib/libvirt/dnsmasq/default.leases
[07:31] <rbasak> jtv: (so requires that you are using the default libvirt bridge/dnsmasq)
[07:33] <rbasak> bigjools: it does far less than vagrant, intentionally. It's just a wrapper over simplestreams, libvirt and cloud-init. So you get to use unmodified cloud images directly, with easy cloud-config definition or direct use of your own overrides.
[07:33] <bigjools> nice
[07:33] <bigjools> cloud images you say ...
[07:34] <bigjools> could these be the same cloud images that maas uses for ephemerals?
[07:34] <bigjools> rvba might see where I am going with this ...
[07:34] <rbasak> Remind me how ephemeral images differ again?
[07:35] <bigjools> I have no idea if they even do
[07:35] <rbasak> I'm sure we do; otherwise we wouldn't generate them.
[07:35] <rbasak> I just can't remember how.
[07:35] <bigjools> we mount them over iscsi as /
[07:36] <rvba> rbasak: right now maas-test uses the name published over avahi, but as we discussed yesterday, we will ditch that and use the ip (parsed from default.lease).  The only thing we need is for wait to wait until cloud-init has run.
[07:36] <bigjools> rvba: are you using cloud-init to install maas?
[07:36] <jtv> I suppose we could always just loop until either the hostname resolves or we lose patience...
[07:37] <jtv> bigjools: not yet.
[07:37] <rvba> bigjools: not currently, but we could.
[07:37] <jtv> Right now it's apt-get (good enough for me tbh)
[07:37] <bigjools> in other words is cloud-init an overhead we don't need?
[07:37] <rvba> bigjools: we need it for something else
[07:37] <bigjools> I would be looking to pare down the VM image as much as possible
[07:37] <rvba> To configure the bridged interface.
[07:37] <bigjools> how big is it BTW?
[07:38] <bigjools> I'm a bit concerned about the amount of downloading to run this stuff
[07:38] <rbasak> http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral/view/head:/maas-cloudimg2ephemeral is the cloud image to ephemeral image conversion script
[07:38] <rvba> I'm not going to disagree with that.
[07:39] <bigjools> rvba: can we use that script to cut some downloading out?
[07:39] <rbasak> uvtool can use an external image, provided it's in qcow format. It just needs it to have cloud-init installed. I'm not sure if ephemeral images will work, though.
[07:40] <bigjools> well if we just download cloud images and then run this tool ...
[07:40] <rbasak> I'm a little lost. What's your goal here?
[07:40] <rvba> bigjools: maybe, this is worth looking into.
[07:41] <bigjools> it's a shame it doesn't explain what it's actually doing
[07:41] <bigjools> rbasak: so we have to download images for uvtool and then again download ephemeral images for maas inside the vm.  We should be able to avoid that here
[07:41] <bigjools> they are largely identical
[07:41] <bigjools> apart from whatever this script does
[07:41] <rbasak> I see. That makes sense.
[07:42] <bigjools> I made sense \o/
[07:43] <rbasak> One possible difference is that ephemeral images need to ignore other disks that are on the machine. uvtool uses cloud-init to supply metadata without a metadata server by adding a second disk with the metadata and having cloud-init find it. I wonder if that's compatible.
[07:43] <bigjools> I don't see how it can even come into play
[07:43] <rbasak> uvt-kvm will take a --backing-image-file qcow2 format file directly if you have one
[07:44] <rbasak> Then it doesn't need to have downloaded anything previously; you can take care of it.
[07:48] <rvba> rbasak: btw, I managed to dhcp a node on the bridge interface all right.
[07:49] <jtv> Oh, you have the bridging working?
[07:49] <rbasak> \o/
[07:49] <bigjools> \o/
[07:50] <rvba> Yep, as a proof-of-concept for now: i.e. no code, just manual stuff.
[07:50] <rvba> But now it can be transformed into code.
[07:51] <rbasak> rvba: if I drop avahi-daemon installation, and you drive it from outside the guest, then do you still need the wait subcommand extended? I'm not exactly sure how to do that part right now.
[07:51] <rvba> rbasak: that would be ideal… but I think we can work around it.
[07:52] <rbasak> rvba: btw, you can wait for /var/lib/cloud/instance/boot-finished to appear. It just doesn't work in the general case. I've talked to smoser about something in /run so things don't get confused by reboots
[07:54] <rvba> rbasak: instead of using uvtool's cloud-init config and inject our stuff in the run-command, we could simple configure the entire cloud-init's config.  With this level of control we would be fine.
[07:56] <rbasak> rvba: sure, you can do that.
[08:01] <rvba> rbasak: hum, even if we do that, we would still a way to know when cloud-init has finished running.
[08:05] <rbasak> rvba: as a workaround, I think "while ! -f /var/lib/cloud/instance/boot-finished; do sleep 1; done" should do it.
[08:05] <rbasak> rvba: it just doens't work in the general case, because if you booted the backing image previously, then that file already exists.
[08:07] <rvba> rbasak: thanks, that should work in case so we can use that until something more solid is in place.
[08:10] <rvba> in our* case
[08:16] <rvba> rbasak: I also want to try using a "interface type='direct'" to see if we can avoid setting up the bridge.
[08:34] <rvba> rbasak: it seems to work just as well.
[08:36] <roaksoax> rvba: ping
[08:36] <roaksoax> err
[08:36] <roaksoax> rbasak: ping
[08:37] <rbasak> roaksoax: pong
[08:38] <roaksoax> rbasak: pm :)
[08:39] <rvba> rbasak: any reasons why we should use a bridged connection instead of a "direct" connection?  A "direct" connection seems a bit simpler to set up.
[08:40] <rbasak> rvba: for the connection to the independent NIC that connects to the MAAS node under test? "direct" would be fine in that case, if it works. I've never tried it.
[08:40] <rvba> rbasak: yes, for the connection to the node.  I just tried it, it seems to work fine.
[08:40] <rvba> We will start with this then.
=== CyberJacob is now known as CyberJacob|Away
[09:10] <jtv1> rvba: it's about a TODO (I think you probably wrote it) to check for an ssh key.
[09:11] <jtv1> AFAICS we can't connect to the VM without one, unless perhaps we start doing nasty stuff with stdin/stdout.
[09:11] <rvba> Right.
[09:11] <jtv1> And uvtool only uses the default key, which is something I'd love to change but for now seems to be something we just have to deal with.
[09:11] <rvba> Well, there is a way around this.
=== jtv1 is now known as jtv
[09:12] <jtv> Oh?
[09:12] <rvba> i.e. we could generate our key and use that…
[09:12] <jtv> That's the thing — the path is hard-coded in uvtool.
[09:12] <rbasak> If you generate your own cloud-config and supply uvtool with that, you can supply whiever key you want
[09:12] <jtv> $HOME/.ssh/id_rsa
[09:12] <rvba> That's exactly what I was thinking about.
[09:12] <jtv> Ah!
[09:12] <rvba> And we want to do that anyway.
[09:12] <rbasak> Or I could make the path overridable I suppose. I just need sensible logic that works reasonably in the general case.
[09:13] <rvba> Because we will want to add ppas and stuff.
[09:13] <rvba> So owning cloud-init's config seems like the logical thing to do.
[09:13] <jtv> rbasak: command-line option?  For our usage it'd be fine to have a disposable key.
[09:13] <rbasak> rvba: or you could do that stuff (apart from the key) from a script. The advantage of that is that it would be synchronous, and slightly easier to debug.
[09:14] <jtv> Easier in some ways, harder in others...
[09:14] <rbasak> jtv: a command line option would be fine
[09:15] <jtv> But anyway, for now userdata sounds like a good option.
[09:15] <rvba> rbasak: well, technically we could even use the run-cmd-once to set up another ssh key.
[09:15] <rbasak> I'm a fan of making everything synchronous where possible
[09:15] <rbasak> create, wait, grab the ip then run a script that calls ssh as needed
[09:15] <rbasak> Then you can do things synchronously across the host and the guest
[09:15] <jtv> (Out of interest, what is the advantage of setting up PPAs and installing packages through cloud-init, given that we need to ssh into the VM anyway?)
[09:15] <rbasak> jtv: right
[09:16] <rvba> Well, in my view, it's just that cloud-init is already designed to do most of the preparation things we will want to do:
[09:16] <jtv> (I wasn't making a statement, only asking)
[09:16] <rvba> set up ssh keys, install ppas, install packages.
[09:17] <rvba> Sure, we can do most of this (not the ssh key installation) manually using ssh.
[09:17] <rbasak> rvba: I think that's handy when you don't have an easy synchronous means to do the same thing. Eg. if you want to start an EC2 instance and don't want to hang around to ssh into it, but just want it deployed.
[09:18] <rbasak> rvba: OTOH, using ssh gives you stdin/out/err capability, for example. You aren't running scripts in the dark. So I think cloud-init's functionality makes sense when there is no ssh option, but ssh makes sense in the general case when there is an option to use it.
[09:18] <jtv> I think basic laziness is the most powerful argument here.  :)
[09:18] <jtv> I'll selfishly focus on the ssh key.
[09:19] <rbasak> Debugging and development velocity for me. Unix-land is focused on sending errors to stderr when something fails. With ssh, everything is hooked up so you see it.
[09:19] <rvba> rbasak: that's a good point.  So, could you add an option to uvtool to support using a user-defined ssh key?
[09:19] <rbasak> (and the user, too)
[09:19] <jtv> rbasak, is it a matter of passing something like userdata=<data> to uvt-kvm?
[09:19] <rbasak> rvba: sure. That's relatively easy.
[09:19] <rvba> jtv: yes
[09:20] <jtv> No base64 or anything like that?
[09:20] <rbasak> No base64. Although you do need a temporary file.
[09:20] <rvba> jtv: If rbasak can do, then all maas-test has to do is create and use that key, how does that sound?
[09:21] <jtv> Well... is uvtool in the cloud PPA?
[09:21] <jtv> In other words, is it safe to depend on a future version of uvtool?
[09:22] <rbasak> You mean the cloud archive? Yes, but we can't really put new functionality into Precise before Trusty's release, if that's what you're asking.
[09:22] <rbasak> (into Precise via the cloud archive, that is)
[09:22] <jtv> Then ISTM it's not quite enough to add an option to uvtool and make maas-test use it.
[09:22] <rbasak> It looks like "--user-data -" would work, so you can send it in stdin. I think. Not tested.
[09:23] <rvba> We can create a file, that's really not a problem.
[09:23] <jtv> And I can take a cue from create_default_user_data I suppose.  It sends yaml.
[09:23] <rbasak> Which release does maas-test need to be able to run in?
[09:23] <jtv> Good question.  Arguably, 14.04 would do because we run a VM anyway.
[09:24] <rvba> Well, uvtool needs to be installed on the host machine.
[09:24] <rvba> bigjools: ^ "Which release does maas-test need to be able to run in"
[09:24] <jtv> Yes, but what I mean is: if you want to test with MAAS on Precise, you can tell maas-test to fire up a Precise VM.
[09:25] <rbasak> Right. You might just be required to have a 14.04 laptop to run the test on though, for example. Right?
[09:25] <jtv> The problem with writing our own key right now is that its filename has to be $HOME/.ssh/id_rsa[.pub].
[09:25] <rbasak> Though uvtool is in the cloud-tools pocket, and that will be updated on release of Trusty (for Precise).
[09:25] <jtv> If there's already a key there and it has a passphrase, it gets harder.
[09:25] <rvba> rbasak: right, but still, I'd like bigjools to comment on that.
[09:25] <rbasak> OK
[09:25] <rvba> jtv: we should put maas-test's key somewhere else
[09:26] <jtv> Duh :)
[09:26] <rvba> and write our own cloud-init config with that key.
[09:26] <jtv> Yes...
[09:26] <rvba> I don't see where the problem is… ?
[09:26] <jtv> I think we were thinking different things when you said we can write a file...
[09:27] <rbasak> I can add an option to override the default path. Provided that it's OK for uvtool with that option to only be available in Trusty and Precise (with cloud-tools after Trusty's release), then you should be OK using it.
[09:27] <jtv> Anyway, yes, I'm looking at the code that generates the default user data.
[09:27] <rvba> rbasak: I think that option is useful to have.  But we need to figure out if we can rely on newer version of uvtool.
[09:27] <rvba> jtv: at the time we were talking about the user-data file.
[09:27] <jtv> Either that, or do the userdata thing.
[09:28] <jtv> rvba: I think I interpreted it as an alternative to what we were discussing.
[09:28] <rvba> jtv: ah ok
[09:28] <jtv> Tone of voice does not carry well on IRC... Let's try talking louder.  :)
[09:29] <rbasak> I THINK WE AGREE
[09:29] <rbasak> :)
[09:29] <jtv> YAY!
[09:29] <rvba> OK!
[09:29] <jtv> R U ON AOL 2 LOL
[09:29] <jtv> rbasak: I'll just imitate the yaml dump from create_default_user_data then...
[09:30] <rbasak> jtv: sure. Do you still want an option to override the default key path?
[09:30] <bigjools> rbasak: last LTS is all that counts, plus HWE kernels as necessary
[09:30] <rvba> jtv: as a reference, here is a real-world user-data created by uvtool: http://paste.ubuntu.com/6375343/
[09:32] <jtv> Thanks rvba.
[09:33] <rvba> jtv: line 7-13 is the base64-ized version of the script I gave to uvt-kvm using --run-cmd-once.
[09:33] <jtv> bigjools: so we don't need maas-test on Precise?
[09:34] <rvba> jtv: and you can probably get rid of the avahi-daemon stuff, I'll put together a branch today to use the IP rather than the name (because avahi takes ages to publish that name).
[09:34] <rvba> Use the IP rather than the name when sshing into the machine that is.
[09:35] <jtv> rvba: why use the IP?  And how do we obtain it?
[09:35] <jtv> I can look it up, but this sounds as if we already get it somewhere.
[09:35] <rvba> jtv: no, we use the name published by avahi.
[09:35] <rvba> jtv: like I said, it's slow, I want to use the IP.
[09:35] <rvba> utv-kvm gives us the IP
[09:35] <rbasak> "uvt-kvm ip" in trunk. It reads libvirt's dnsmasq leases file
[09:36] <rvba> (it parses the log file)
[09:36] <rvba> the lease file
[09:36] <rbasak> (but requires that you are using libvirt's dnsmasq for DHCP)
[09:36] <rvba> And we do that.
[09:36] <jtv> utvtool in 13.10 doesn't support that, does it?
[09:36] <rbasak> Right. I'm just paranoid that someone will apply it to a different use case and it won't work.
[09:37] <rbasak> jtv: no
[09:37] <rvba> arg
[09:37] <jtv> So then we might as well go with a "here's my ssh keypair" option.
[09:37] <jtv> And just not do any user-data unless we really want to.
[09:37] <rbasak> Do you need it to? You can use ppa:uvtool-dev/trunk for 13.10 this cycle, and after 14.04 release you won't care, right?
[09:38]  * rbasak unaways himself, as he's actually here.
[09:41] <jtv> rbasak: I suppose we could, but either way, if we require the PPA anywhere at all we might as well go with the ssh-key option.
[09:41] <jtv> At least it seems to me the cleanest, smallest, least fragile solution.
[09:42] <rvba> If we have to stick to what's released now, we could: stick with using avahi for name resolution, customize user-data to pass our own key.
[09:43] <jtv> But if we then also want to avoid using the hostname in the user-data...
[09:43] <rbasak> jtv: ie. avoid using avahi
[09:43] <jtv> ...it'd still be nice to have the "uvt-kvm ip" command.
[09:44] <rbasak> rvba: why would we need to stick to what's released now?
[09:44] <rvba> rbasak: I don't know.  I'm just unsure what the requirements are when it comes to using ppas and stuff.
[09:44] <rvba> rbasak: I'll ask Julian to clarify that.
[09:45] <rbasak> OK
[09:45] <rvba> jtv: rbasak But I'd like to use "uvt-kvm ip" as much as you do.
[09:45] <rbasak> :)
[10:08] <rvba> rbasak: okay, we will be using the ppa for saucy.
[10:09] <jtv> rbasak: need any help getting that option for the ssh key in?
[10:14] <rvba> rbasak: I filed bug 1248895. I phrased it like "Add option to…" because you seem to be filing bug this way yourself.
[10:14] <ubot5> bug 1248895 in uvtool "Add option to choose which ssh key gets added onto the machine." [Undecided,New] https://launchpad.net/bugs/1248895
[10:25] <jtv> rvba: all that's missing is the word "should"  :-P
[10:28] <rvba> heh
[10:29] <bigjools> snork
[10:29] <bigjools> I am EODing
[10:29] <bigjools> cheerio folks
[10:30] <jtv> nn bigjools
[10:30] <rvba> nn bigjools
[10:30] <jtv> I'll be following.
[10:30] <bigjools> 12h work day for me ...
[10:30] <jtv> gmb: don't forget to land that lint branch
[10:30] <jtv> bigjools: don't do that!
[10:30] <gmb> jtv: Good point, ta.
[10:30] <bigjools> making up for yesterday!
[10:30]  * bigjools departs
[10:31] <jtv> gmb: I now realize why I didn't find your branch — maas vs maas-test
[10:32] <jtv> Be careful come MP time...
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
[11:50] <gmb> Ooh, MemryError. Haven't seen one of those for a while.
[11:50] <gmb> *Memory
=== stokachu_ is now known as stokachu
=== Spideyman is now known as Spideyman_afk
[14:33] <gmb> rvba: Have at it: https://code.launchpad.net/~gmb/maas-test/extract-functions/+merge/194351
[14:34] <rvba> gmb: cool, I'll have a look in a minute.
[14:43] <rvba> gmb: reviewing your branch now.  Here is a tiny branch to review if you have time: https://code.launchpad.net/~rvb/maas-test/vm-network/+merge/194353
[14:50] <rvba> gmb: I get test failures when I run 'make test' in your branch: http://paste.ubuntu.com/6376687/
[14:51] <rvba> gmb: err, http://paste.ubuntu.com/6376689/
[14:51] <gmb> rvba: Oh! I didn't see those before, but I haven't make test since I merged trunk. Hang on.
=== Spideyman is now known as Spideyman_afk
[15:04] <rvba> gmb: also, this is what I get when I run the script: http://paste.ubuntu.com/6376753/
[15:04] <gmb> rvba: Yes, jtv's changes clobbered some of mine and caused problems. Fixing it now.
[15:07] <gmb> rvba: Should be fixed now.
=== Spideyman_afk is now known as Spideyman
[15:15] <rvba> gmb: approved with a couple of remarks.
[15:17] <gmb> rvba: Thanks
[15:28] <gmb> rvba: Fixed + asked a question re: check_call.
[15:29] <rvba> gmb: you're right, there is a bit of duplication: one one side we use addDetail and inside run_command itself we (optionally) check the retcode (but note that the Exception contains all the info)
[15:30] <gmb> rvba: Right. So, I guess I'm asking: is there a preference for test_install_maas_package?
[15:31] <rvba> gmb: I think I put the check_call stuff in run_command so that even the commands run at the fixture level can be checked.
[15:32] <gmb> rvba: When you say "run_command()" you mean utils.run_command(), right?
[15:32] <rvba> Right.
[15:32] <rvba> gmb: I'm trying to think what's better for test_install_maas_package…
[15:32] <rvba> gmb: on second thoughts, maybe it's better to keep the check at the test level.
[15:33] <rvba> If we do that everywhere in the test, it will give us more control.
[15:33] <gmb> rvba: I agree.
[15:33] <rvba> Cool, let's do that then.
[15:34] <gmb> rvba: So for clarity: do you mean that we stick with using asserts to check that a process has behaved properly (and therefore treat a failure as a failure rather than an error)?
[15:34] <freeflying> after a node get a new ip via dhcp, how long does maas take to update its dns record
[15:34] <rvba> gmb: yes… do you agree that it's best?
[15:35] <gmb> rvba: Yes, I do. After all, we're leveraging the testing infrastructure for a reason.
[15:35] <rvba> Right.
[15:35] <gmb> Cool.
[15:35] <rvba> freeflying: it can take up to 1 minute.
[15:35] <gmb> I'll get this landed then.
[15:36] <freeflying> rvba, if it doesn't, what shall I look into to troubleshoot?
[15:37] <rvba> freeflying: check that there is a record for your node in the dhcp lease file (/var/lib/maas/dhcp/dhcpd.leases)
[15:38] <freeflying> rvba, the lease file has been changed
[15:42] <rvba> freeflying: can you check that the cluster's celery log contains traces of the task named 'upload_dhcp_leases' being executed?
[15:43] <rvba> freeflying: also, please check that the region's celery log contains traces of tasks related to DNS… when new dhcp lease should have triggered a DNS zone rewrite on the region.
[15:44] <freeflying> rvba, [2013-11-08 00:43:28,566: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[7c0e286e-41c4-49c8-a58c-28ac1b62de6c] succeeded in 3.58162903786s: None
[15:44] <rvba> That looks all right.
[15:44] <freeflying> [2013-11-07 19:30:11,536: INFO/MainProcess] Task provisioningserver.tasks.rndc_command[7ba2f719-6988-43f6-b70e-64f2670db0dc] succeeded in 0.0125498771667s: None
[15:44] <freeflying> seems this is the problem
[15:45] <freeflying> last update was couple of hours back
[15:45] <rvba> Yeah, that's not right.
[15:46] <rvba> Is your cluster configured to handle DHCP *and* DNS?
[15:48] <freeflying> rbasak, cluster only handle dhcp
[15:49] <rvba> freeflying: Well, that's your problem then.
[15:49] <freeflying> rvba, any solution
[15:50] <rvba> freeflying: change the cluster's setting :)
[15:50] <freeflying> rvba, like what?
=== che is now known as Guest11248
[15:50] <rvba> freeflying: you can do that in the UI, or via the API.
[15:51] <freeflying> rvba, what do I need change
[15:51] <rvba> freeflying: where did you see that cluster was only handly dhcp?
[15:52] <rvba> freeflying: have a look at http://maas.ubuntu.com/docs/cluster-configuration.html
[15:52] <freeflying> rvba, the region controller and cluster are on different server
[15:53] <rvba> Right, so the DNS server is on the region and the DHCP server is on the other server, but still, you need to configure the cluster controller to say that it should relay the DNS info to region.
[15:53] <freeflying> rvba, so I only need cluster to manage dns? and then region controller will manage dns?
[15:54] <rvba> freeflying: yeah, you need to set the cluster to "Manage DHCP and DNS" and then the DNS server on the region will have all the DNS info.
[15:54] <freeflying> rvba,  that is what I have it set via web UI
[15:55] <rvba> freeflying: can you check that your dns server is running on the region machine?
=== Spideyman is now known as Spideyman_afk
=== CyberJacob|Away is now known as CyberJacob
=== Spideyman is now known as Spideyman_afk
=== Spideyman is now known as Spideyman_afk
=== Spideyman_afk is now known as Spideyman
[22:02] <freeflying> rvba, yes, it is
[22:16] <Azendale> I have a problem where if I boot too many machines in my MaaS cluster, it makes the DHCP server in MaaS stop responding until I reboot the MaaS server (which is both a region controller and the cluster controller). Anyone have any ideas where to start debugging?
[22:23] <freeflying> Azendale, how many node you booted simultaneously
[22:24] <Azendale> freeflying: 3 or 4 probably will cause it, 5, 6 or 7 will do it for sure
[22:25] <Azendale> freeflying: These are all KVM machines connected (with virtio drivers) to a bridge on the host machine
[22:25] <freeflying> Azendale, can you check /var/log/maas/pserv.log
[22:25] <freeflying> Azendale, uhmm
[22:26] <freeflying> Azendale, I had similar symptom before, but it were 30 vms
[22:28] <Azendale> freeflying: ok, I'll look at that log file, give me a few minutes
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
[22:37] <bigjools> Azendale: I'm interested in your log file
[22:44] <Azendale> bigjools: Ok, I can get you a copy. Should a paste bin it somewhere or upload it somewhere?
[22:45] <bigjools> Azendale: pastebin for now and then we can detemine if it's a bug
[22:45] <Azendale> ok, sounds good
[22:45] <bigjools> our QA lab does multiple simultaneous boots OK so this is curious
[22:47] <Azendale> it seems like usually it will boot (but booting can take a while with all the concurrent image loads). But then at some point after that, either when the installer is trying to get dhcp or on the reboot after the install, the DHCP fails and the machine gets stuck. Usually, I will restart the maas server, and then power off the nodes and start one or two at a time
[22:49] <Azendale> bigjools: It's going to take a while for stuff to install, and I'll have to run to class in a few minutes, but I'll get back to this in a few hours and get you a copy of the log
[22:49] <bigjools> Azendale: ok thanks
[22:50] <Azendale> bigjools: np, thank you for the help!
[22:50] <bigjools> is DHCP failing or the tftp?
[22:52] <Azendale> bigjools: it's the dhcp I believe. I've seen cases where the installer can't get an address, and gives up. I've also seen the PXE boot in the VM give up on an interface because it can't get an IP and go to the next interface, ad infinitum. (I may have seen the tftp get stuck before, but I'm not sure on that. I'll keep notes)
[22:54] <Azendale> ok, got to go for now, but I'll be back
[22:54] <bigjools> Azendale: ok thanks - that sounds like isc-dhcpd then. see you later
=== freeflying is now known as freeflying_away
=== CyberJacob is now known as CyberJacob|Away
=== freeflying_away is now known as freeflying