[01:43] <roaksoax> bigjools: maas-dhcp should be on the cluster right?
[01:43] <bigjools> roaksoax: yes
[01:43] <bigjools> you *just* caught me, about to go to lunch
[01:44] <roaksoax> bigjools: enjoy
[01:44] <bigjools> :)
[03:05] <roaksoax> bigjools:
[03:05] <roaksoax> Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
[03:05] <roaksoax> 2013-09-20 12:03:12+0900 [Uninitialized] Stopping factory <HTTPClientFactory: http:///MAAS/api/1.0/pxeconfig/?cluster_uuid=5c11d0fa-6b41-4cd6-b278
[03:12] <bigjools> roaksoax: I have never seen that
[03:13] <roaksoax> bigjools: we are seeing this right now ;/
[03:13] <roaksoax> how does pserv get that address?
[03:13] <bigjools> what address?
[03:13] <roaksoax> bigjools: the address of the region
[03:14] <bigjools> DEFAULT_MAAS_URL
[03:14] <bigjools> has to be contactable by the nodes
[03:14] <bigjools> and clusters
[03:14] <roaksoax> bigjools: it is
[03:14] <roaksoax> but pserv is trying to contact without address:"  http:///MAAS/api/1.0/pxeconfig/?cluster_uuid=5c11d0fa-6b41-4cd6-b278
[03:14] <bigjools> oh I see now!
[03:14] <bigjools> jtv1: ^
[03:14] <bigjools> halp
[03:15] <roaksoax> bigjools: this is kind of critical btw
[03:16] <bigjools> roaksoax: what is in /etc/maas/maas_cluster.conf
[03:16] <bigjools> the MAAS_URL in there should be the region's
[03:16] <roaksoax> it is
[03:17] <bigjools> so that Connection refused error is in the pserv log?
[03:18] <bigjools> roaksoax: ^
[03:18] <freeflying> roaksoax, seems the same
[03:20] <bigjools> roaksoax: can you sniff its tcp and see what it's trying to connect to
[03:20] <bigjools> it's just a bad config somewhere
[03:20] <bigjools> or perhaps a proxy getting in the way
[03:20]  * jtv1 reads backscroll
[03:22] <roaksoax> yes p serv.log
[03:24] <roaksoax> bigjools: this is multinode maas btw
[03:24] <bigjools> roaksoax: you mean multi cluster?
[03:25] <roaksoax> ok hold on
[03:25] <roaksoax> it seems squid homehow messing things up
[03:25] <roaksoax> but it shouldn't
[03:28] <jtv> Might be worth checking whether the start-cluster-controller command line has the right server URL.
[03:28] <bigjools> that's what I was juuuust about to say
[03:29] <bigjools> jtv: it comes from MAAS_URL in the maas_cluster.conf file right?
[03:29] <jtv> If it doesn't, then either /etc/maas/maas_cluster.conf is messed up, or alternatively I'd say something changed in how upstart scripts work.
[03:29] <bigjools> I think that's what the packaging uses
[03:29] <jtv> Yes.
[03:29] <jtv> The upstart script sources that config file, then passes $MAAS_URL on the command line.
[03:30] <jtv> Oh.
[03:30]  * jtv checks something...
[03:30] <bigjools> roaksoax: /etc/init/maas-cluster-celery.conf starts it remember
[03:31] <roaksoax> yeah looking at it
[03:31] <bigjools> but you said that config was already ok
[03:31] <bigjools> so squid getting in the way?
[03:31] <jtv> How could squid get in this particular way?
[03:31] <roaksoax> bigjools: yeah, otherwise it would not have registered the cluster controller in the region
[03:31] <bigjools> roaksoax: right
[03:31] <bigjools> jtv: it could be down :)
[03:32] <jtv> Down, sure.  But mangling URLs..?
[03:33] <bigjools> who knows what its config us
[03:33] <bigjools> is
[03:34] <roaksoax> maybe a dirty pyc file?
[03:35] <bigjools> unlikely
[03:35] <jtv> Could happen with permissions problems and an upgrade, I suppose.
[03:36] <roaksoax> jtv: clean system
[03:36] <bigjools> roaksoax: have you traced the tcp?
[03:36] <bigjools> ethereal or something
[03:36] <bigjools> tcpdump
[03:37] <bigjools> take it one step at a time and divide the problem into possible areas
[03:38] <roaksoax> bigjools: https://pastebin.canonical.com/97785/
[03:40] <bigjools> roaksoax: stops short ...
[03:40] <roaksoax> bigjools: repeats itself
[03:40] <bigjools> roaksoax: is it connecting to the right place?
[03:41] <roaksoax> bigjools: yeah
[03:41] <roaksoax> again, otherwise the cluster controller would not have registered itself
[03:41] <bigjools> roaksoax: ok if you trace on the region controller do you see traffic?
[03:41] <roaksoax> bigjools: hold on, i rebooted the cluster
[04:10] <bigjools> jtv: roaksoax: so the config actually comes from /etc/maas/pserv.yaml
[04:10] <bigjools> tftp:generator:
[04:11] <bigjools> roaksoax: so I suspect the bug is in the packaging and it doesn't set that file up properly sometimes
[04:11] <bigjools> maybe a race condition with another package
[04:11] <bigjools> and the wrong one got installed first
[06:04] <freeflying> where can I watch commissioning log?
[06:41] <freeflying> bigjools, now I can  enlist node, but have problem to commission it, have apt_proxy in enlist_data and commission, any clue?
[06:43] <freeflying> jtv, ^^
[07:01] <jtv> Hi freeflying
[07:01] <jtv> I think a commissioning node will direct its syslog to either the region controller or the cluster controller...
[07:02] <jtv> Look for an "rsyslog" log.
[07:03] <freeflying> jtv, no log there
[07:04] <freeflying> jtv, it gives connect to 169.xxx fails, after it get ip address
[07:05] <jtv> The node says that on its console?
[07:05] <freeflying> yes
[07:05] <jtv> Any idea what the 169.xxx address is?
[07:06] <freeflying> no, all address we're using within 10.x.x.x
[07:07] <freeflying> looks that is epheremal image?
[07:08] <jtv> Yes, commissioning runs an ephemeral image.
[07:08] <jtv> But I don't think the node should be talking to the internet at that point.
[07:09] <jtv> Doesn't look as if it's the Ubuntu archive either, although I suppose it might be your local mirror.
[07:10] <freeflying> why its calling its calling 169.254.169.254/2009-04-04/meta
[07:11] <jtv> So that's where it's trying to find its metadata service.
[07:11] <jtv> Strange address...
[07:12] <freeflying> jtv, where shall I configure it, or is it because it can't access to maas-regional contoller
[07:12] <jtv> That's a zeroconf address.  Any chance there's a wifi interface being mistaken for your server?
[07:12] <freeflying> jtv, no
[07:13] <freeflying> thre are 8 ports on the server, 1 of them are 10 gig, for of them are 1 gig;s
[07:14] <jtv> At least, when I use "whois" on it, it says computers use 169.154.*.* (note 154, not 254!) when they don't have an IP address and don't get one from the network.
[07:14] <freeflying> from weui, the metatada_url was set to maas regionl's
[07:14] <jtv> And that's a 10.*.*.* address, right?
[07:15] <jtv> And no 169.*.*.* networks at all?
[07:15] <rvba> 169.254.169.254 is used in Amazon EC2 and other cloud computing platforms to distribute metadata to cloud instances.
[07:15] <jtv> !
[07:15] <rvba> So cloud-init is acting up.
[07:16] <jtv> Thinking this is EC2...
[07:17] <rvba> cloudinit/sources/DataSourceEc2.py:DEF_MD_URL = "http://169.254.169.254"
[07:17] <freeflying> jtv, yes, in the preseed generated for commissioning is 10.209.13.204/MAAS
[07:17] <rvba> (this is in the cloud-init source code)
[07:19] <jtv> I'm looking at the cloud-init source now...
[07:23] <rvba> I suspect the node can't reach the region and thus cloud-init falls back to using the EC2 metadata address.
[07:23] <jtv> Ah, I was thinking it might not have received the right configuration...
[07:25] <rvba> Well, I don't see how the EC2 IP could originate from MAAS.
[07:27] <jtv> Neither do I...  I was thinking that cloud-init might not have received the right configuration, and decided to try things the EC2 way.
[07:27] <jtv> I'm trying to figure out how cloud-init chooses the DataSource to use.
[07:27] <rvba> That's possible indeed.  But from what freeflying was saying, it seems the configuration is right.
[07:28] <rvba> freeflying: if cloud-init errors somehow (and uses the EC2 address as a — somewhat crezy — fallback), you will see errors on the node console while it is commissioning… do you have access to it?
[07:29] <freeflying> rvba, you mean the kvm?
[07:30] <jtv> The screen, yes.
[07:32] <rvba> freeflying: well, the node's screen if it is a physical machine.
[07:33] <freeflying> let me post a screenshot
[07:34] <jtv> ("kvm" can be either the mouse/keyboard switch on a physical machine, or a commonly used type of virtual machine)
[07:35] <freeflying> in this case, it mean mouse/keyboard :)
[07:37] <rvba> Well, it's the V in KVM that I'm interested in :)
[07:41] <freeflying> people.canonical.com/~zhengpenghou/20130920_162508.jpg
[07:42] <freeflying> uploading
[07:42] <jtv> Thanks
[07:42] <jtv> Yup, that's cloud-init running out of DataSource candidates.
[07:43] <rvba> Yep
[07:43] <jtv> AFAICS cloud-init tries to download data from various sources, until it finds one that works.
[07:44] <jtv> It's not finding any.
[07:44] <freeflying> any suggestion?
[07:45] <jtv> We'll have to find the root cause first...  Do you have a way of verifying that the nodes can reach the given IP address?
[07:45] <rvba> When I encountered that problem (cloud-init was using the EC2 IP), it was because the node could not reach the MAAS IP address.
[07:46] <rvba> So it's worth checking, as jtv said.
[07:46] <jtv> It would be ideal if you could access the full URL on http...  if you get a 404, it's likely to be a problem in the URL configuration.
[07:46] <jtv> If it's a permissions error, then it should just have worked and we have a mystery.
[07:46] <jtv> If it's a networking error, then there's our problem.  :)
[07:46] <freeflying> jtv, no, the node never fails to be commissioned
[07:47] <jtv> I thought it failed during commissioning..?
[07:47] <freeflying> jtv, funny thing is I do have 1 node commissioned :)
[07:47] <freeflying> jtv, yes
[07:48] <jtv> I don't understand... if it fails during commissining, then the node fails to be commissioned, right?
[07:48] <freeflying> before cloud-init gives the error info, I did see the commissioning node get ip
[07:49] <jtv> So DHCP is working...  did it get its IP from the right server?
[07:49] <freeflying> jtv, I enlist 3 machines, 1 succeeded, not the other 2
[07:49] <freeflying> jtv, yes
[07:50] <rvba> Is there anything special about these two machines network-wise?
[07:50] <freeflying> rvba, this could be a issue, but not sure
[07:50] <jtv> Are all the nodes visible in the web UI, and similarly (correctly) configured?  If something went wrong there, they might fail to get to the metadata service.
[07:51] <freeflying> jtv, all of them are listed in webui
[07:52] <jtv> If we're very lucky, the metadata server log will show the nodes' requests...
[07:52] <rvba> So one of them is "ready" and two of them stuck "commissioning", correct?
[07:52] <freeflying> rvba, exactly
[07:53] <freeflying> rvba, and have proxy set up in both commissioning/enlist_data
[07:54] <freeflying> jtv, rvba would like t have a check?
[07:55] <jtv> Always worth a check...  if you have a way to simulate an http request to that URL from the same node, that might tell us something too.
[08:12] <jtv> (I should say: by "that URL" I mean the metadata URL)
[08:13] <rvba> I see requests to the metadata service from 10.209.13.1{0,1,2,3}.
[08:14] <jtv> freeflying: which node is the successful one?
[08:15] <rvba> jtv: the errors in maas/maas.log are concerning
[08:16] <rvba> The "PermissionDenied: Not authenticated as a known node." errors.
[08:16] <jtv> And first, a problem registering the node group..?
[08:17] <rvba> The two problems might be linked…
[08:17] <jtv> Yup.
[08:18] <jtv> Looks like the NodeGroupWithInterfacesForm.
[08:18] <freeflying> jtv, working node called xggt6
[08:18] <rvba> freeflying: we need its IP or it's uuid.
[08:18] <jtv> freeflying: do you know the last number of its IP address?  We see 4 machines making requests to the metadata service.
[08:19] <freeflying> rvba, 10.209.13.10
[08:19] <jtv> Thanks!
[08:19] <rvba> freeflying: you said you had a config with 2 clusters right?
[08:19] <rvba> s/had/have/
[08:19] <freeflying> rvba, 1 cluster 1 regional
[08:20] <rvba> freeflying: when you go to the settings page, do you see one or two cluster controllers?  (because there is one cluster alongside the region by default)
[08:21] <freeflying> rvba, only 1
[08:21] <rvba> Is it called 'master'?
[08:21] <freeflying> rvba, we don't have maas-cluster-controller installed on regional
[08:22] <freeflying> rvba, the one I can from webui is cluster-xxxxx
[08:23] <rvba> freeflying: okay, I see.  Not having the maas-cluster-controller installed on the region is an untested setup.
[08:24] <freeflying> rvba, hehe
[08:31] <rvba> freeflying: the MAAS region has IP 10.208.11.203 right?
[08:31] <rvba> Or is that the cluster machine?
[08:32] <freeflying> that is cluster
[08:32] <freeflying> regional is 204
[08:50] <jtv> The 403 errors match the "Not authenticated as a known node" errors in the maas log.
[08:56] <jtv> A kernel download failed at one point.  But that should either break commissioning completely, or leave it unaffected.
[08:56] <rvba> freeflying: could you please run this in 'sudo maas shell': http://paste.ubuntu.com/6131745/ ?
[08:56] <freeflying> jtv, when idid that error happen
[08:57] <rvba> freeflying: it will just print information about the nodes and the clusters, to help us debug the problem.
[08:58] <jtv> freeflying: that failure happened at 11:13:54 +0900
[08:59] <freeflying> rvba, on it, give me secs
[09:02] <freeflying> rvba, 1 nodegroup there, and 2 node
[09:03] <rvba> freeflying: only two nodes total?  I though you said you had 3?
[09:03] <rvba> thought*
[09:03] <freeflying> jtv, during that time, we're still trying to configure them, and after that, roaksoax has reconfigure cluster
[09:03] <jtv> Yeah, I think the 404 is harmless here or we'd have seen different failures.
[09:03] <jtv> What IP addresses did you get from rvba's script?
[09:04] <jtv> A paste of the output would be ideal.
[09:04] <freeflying> rvba, sorry, forgot to say I deleted others stucked in commissioning
[09:05] <rvba> freeflying: okay, can you try re-enlisting then re-commissioning the problematic nodes, then run that script again?
[09:07] <freeflying> rvba, ok, give me mins :)
[09:08] <freeflying> thanks
[09:09] <rvba> freeflying: sorry but I'm a bit confused, you said you had 3 nodes total, but I can see 4 different IP adresses requesting the preseed…
[09:10] <freeflying> rvba, we actually have 31 machines, guess some one else powered on it
[09:10] <rvba> Okay.
[09:14] <freeflying> rvba, http://paste.ubuntu.com/6131822/
[09:17] <rvba> freeflying: can you run [(n.ip_addresses(), n.system_id, n.nodegroup, n.architecture, n.status, n.hardware_details) for n in Node.objects.all()]
[09:17] <rvba> freeflying: the output will be large :)
[09:22] <freeflying> rvba, any module I shall import?
[09:23] <rvba> freeflying: just 'from maasserver.models import Node'
[09:23] <rvba> (nothing new compared to the previous commands)
[09:24] <freeflying> Traceback (most recent call last):                                                                                                                 │·····················
[09:24] <freeflying>   File "<console>", line 1, in <module>                                                                                                            │·····················
[09:24] <freeflying> AttributeError: 'Node' object has no attribute 'ip_addresses'
[09:25] <rvba> freeflying: ah, right, you're using the precise package.
[09:26] <rvba> freeflying: could you run this: [(n.system_id, n.nodegroup, n.architecture, n.status, n.hardware_details) for n in Node.objects.all()]
[09:31] <AskUbuntu> Should MaaS and Juju get installed on one of my servers or on a client system? | http://askubuntu.com/q/347866
[09:32] <jtv> freeflying: I also have a bit of python I'd like to see the output to.
[09:32] <jtv> from metadataserver.models import NodeKey
[09:32] <jtv> for nk in NodeKey.objects.all():
[09:32] <jtv>     print(nk.node.nodegroup.uuid, nk.node.system_id, len(nk.key))
[09:32] <jtv>  <- that should tell us a bit (not the actual keys of course) about which oauth keys are being sent to which nodes.
[09:32] <jtv> Because we're seeing those nodes fail to authenticate with those keys.
[09:34] <freeflying> rvba, http://paste.ubuntu.com/6131873/
[09:35] <freeflying> jtv, how can I redirect from python console to stdout
[09:36] <jtv> Try:
[09:36] <jtv> python <<EOF
[09:36] <jtv> # Script code goes here
[09:36] <jtv> EOF
[09:36] <jtv> Ahem.  Not python, of course, but "sudo maas shell <<EOF"
[09:37] <jtv> The <<EOF tells it to feed what comes next (until the EOF line) into stdin.
[09:37] <jtv> To redirect: sudo maas shell >/tmp/my-output <<EOF
[09:41] <rvba> freeflying: okay, so everything seems fine so far, you're got one allocated node and 3 commissioning nodes… did they get out of commissioning or are they stuck, same as before?
[09:43] <freeflying> rvba, same
[09:44] <rvba> freeflying: can you try removing the working node and re-enlist, re-commission it?
[09:45] <rvba> freeflying: if that works, then there is definitely a difference between this node and the others (I suspect related to the network config).
[09:45] <freeflying> jtv, no output
[09:47] <freeflying> rvba, error: gomaasapi: got error back from server: 409 CONFLICT (Node cannot be released in its current state ('Commissioning').
[09:47] <jtv> No output!?
[09:47] <rvba> freeflying: wait, the node in question should not be commissioning.
[09:48] <freeflying> jtv, no
[09:49] <jtv> That would explain why the nodes fail to authenticate with the metadata service, but...
[09:53] <freeflying> jtv, any configure I need change to get it fix?
[09:53]  * freeflying gonna go, will be back late, need grab some food 
[09:54] <jtv> freeflying: I think somehow either the input or the output must have been lost.  It's got to print at least one entry for the working node, and if all is normal, one for each of the other ones as well.
[09:55] <rvba> freeflying: you can manually get rid of the juju environment and all the nodes using this: http://paste.ubuntu.com/6131943/
[09:56] <rvba> freeflying: again, if you get one node commissioned, this means you need to investigate how the other nodes (the one stuck commissioning) differ from this one.
[09:57] <rvba> freeflying: can these nodes download stuff from the internet for instance?
[10:21] <rvba> freeflying_away: could you also show us how the cluster controller is configured? (the network config on the cluster page)
[10:22] <jtv> At 14:02:22 there's a POST from the _successful_ node to a broken URL: /MAAS/api/1.0/nodes//MAAS/api/1.0/nodes/
[10:22] <jtv> Oh, not broken apparently -- I'm told there's a workaround on the MAAS side for that.
[10:28] <jtv> The other nodes are signaling to the metadata service...  their individual Node pages in the UI may show useful output.
[10:31] <jtv> Later attempts to do that hit 403.  But the attempts around 14:02:27--14:02:45 got OK responses.
[10:55] <freeflying> rvba, ok, so I'll delete all nodes, and re-enlist
[10:56] <rvba> Yep, let's see if what you've seen before can reproduced.
[10:56] <freeflying> besides this, anything else I shall try
[10:56] <rvba> jtv: ^ ?
[10:56]  * jtv can't think of anything
[10:56] <freeflying> rvba, I left there office, so might not be able to watch the screen
[10:57] <rvba> freeflying: if you can still reach the MAAS machine, then that will be enough.
[10:58] <rvba> Like I said, I want to be sure that the odd behavior you've seen (one node fine, two nodes stuck) can be reproduced.
[11:29] <freeflying> rvba, http://paste.ubuntu.com/6132269/
[11:30] <freeflying> rvba, we use maas to manager another network, which is a 10 gig, will have all later on traffic go through this one
[11:39] <rvba> freeflying: there is a problem right there, the network defined here is 10.208.11.203/24 and the nodes connect to the region using IPs like 10.209.13.10.
[11:39] <rvba> freeflying: did you get one node commissioned, same as before?
[11:40] <freeflying> 10.209.13.0/24 we use for pxe boot, because of hw limitation, 10 gig can't do pxe boot
[11:43] <rvba> We're hitting the same problem we talked about before, MAAS can only deal with one interface right now, used for pxe booting and later on, the IP attached to that interface is the one juju services deployed on the node will use.
[11:43] <rvba> Now I really wonder how you got one node commissioned successfully :).
[11:44] <freeflying> rvba, for commissioning should be fine, we have tested it last week, difference is it was using single server for maas
[11:45] <freeflying> rvba, we have no problem to enlist/commission node
[11:46] <rvba> freeflying: hang on, the problem we've been trying to figure out today is that the nodes never get out of the commissioning phase.
[11:47] <freeflying> rvba, yes,  despite the magical one :)
[11:51] <rvba> freeflying: the whole problem revolves around the network setup.  The nodes need to connect to the region using an IP in the network defined on a cluster controller.
[11:51] <rvba> freeflying: that's how MAAS figures out to which cluster a node belongs.
[11:52] <rvba> freeflying: now, why did it work with one node, that's what needs to be investigated (did you change the cluster configuration half-way through?).
[12:23] <freeflying> rvba, no, the only things has done is set up proxy in preseed
[12:23] <freeflying> rvba, btw, the second network(10 gig's 10.208.11.0/24, managed by maas) never been used so far, no item in that dhcp leases file
[12:25] <rvba> freeflying: the config of the cluster is what defined DHCP config.  The onlyl option I can see is that the config changed when MAAS was running.
[12:26] <freeflying> rvba, no idea
[12:27] <rvba> freeflying: what's in the DHCP config?  cluster machine, file /etc/dhcp/dhcpd.conf.
[12:30] <freeflying> rvba, http://paste.ubuntu.com/6132485/
[12:31] <rvba> freeflying: it defines 10.208.11.0/24.  How come the nodes have ips in 10.209.11.0/24?
[12:32] <freeflying> rvba, because we use an external dhcp to provide pxe boot, so during this stage, its only use pxe boot network, which is 10.209.11.0/24
[12:35] <rvba> freeflying: so the problem is there, the cluster is configured to manage DNS and DHCP (cf http://paste.ubuntu.com/6132269/).  So it believes the nodes will have IP addresses in the range defined here ie. [10.208.11.10 - 10.208.11.250].
[12:36] <freeflying> rvba, as long as the node bootup, it will have the ip :) (after deploying)
[12:37] <rvba> freeflying: yes, but like I said, MAAS uses the IP the node uses to connect to the region to figure out to which nodegroup it should belong.
[12:38] <freeflying> rvba, during our testing last week, it worked :)
[12:38] <freeflying> rvba, thats the thing confusing me now
[12:38] <rvba> freeflying: what changed then?
[12:39] <freeflying> rvba, split regional and cluster onto two servers
[12:39]  * freeflying really tired, need some relax and rest
[12:40] <freeflying> rvba, anyway, I can't continue on it today, thanks for you guys
[12:41] <rvba> freeflying: I understand, it's pretty late for you.  This is indeed a networking problem, with the region and the cluster on different machines, the nodes use, to connect to the region, an IP adress which is not recognized as belonging to the nodegroup.
[12:41] <rvba> nn freeflying
[12:43] <allenap> freeflying: Would you be able to summarise what's happened today and email it with us in Red?
[12:44] <freeflying> allenap, individually? or do you have a list?
[19:09] <marlinc> Where are the MAAS power templates stored?
[19:15] <kentb> While on the subject of power, maas seems to pre-populate the IPMI power parameters with a maas username and random password.  Is that even usable?  I've been going in and changing those to a known-working username/password without trying the one maas gives me.
[19:29] <roaksoax> marlinc: depends on the version
[19:29] <roaksoax> kentb: they work
[19:29] <marlinc> I found out already :)
[19:29] <roaksoax> kentb: maas creates them intentionally
[19:29] <roaksoax> kentb: and doesn't prepopulate
[19:29] <roaksoax> kentb: it access the BMC and adds them
[19:29] <roaksoax> marlinc: cool :)
[19:48] <kentb> roaksoax: ok. thanks for clarifying