=== freeflying_away is now known as freeflying [01:25] bigjools: yeah I saw that.. i'm not sure how to fix it because locally it builds just fine [01:26] roaksoax: oh really? I didn't try I just assumed, how naive. So what's the difference locally? [01:27] bigjools: i build locally against a pbuilder [01:27] bigjools: so it might be the build environment that's broken.. or maybe the recipe? [01:28] bigjools: the reason might be that it is not detecting the source format? [01:28] roaksoax: I was told that it's bzr's fault [01:28] bzr-builder will convert from 3.0 (quilt) to 3.0 (native) if you don't have pristine-tar information [01:29] and previously dpkg didn't care until a recent trusty upload [01:29] bigjools: where is the recipe that builds the package? [01:30] hello, guy, just a quick question, how can i force maas to update the dns record based on the its dhcp lease file? [01:31] if there is a new record in dhcpd.leases, any reason why hasn't maas has it update into dns? I have restart celery on both region and cluster controller [01:31] in my case , new node added, but failed to add dns record in the zone file [01:31] is the interface set to manage dhcp/dns? [01:31] bigjools, yes [01:32] can you see errors in either celery log? [01:32] bigjools, you mean celery logs on region controller? [01:32] and cluster [01:34] bigjools: so maybe bzr is not creating a pristine tarballs then [01:34] roaksoax: https://code.launchpad.net/~maas-maintainers/+recipe/maas-daily-trusty [01:34] roaksoax: correct, I think [01:35] bigjools, i don't think i have see the error in celery logs on both nodes, but i can see 0AuthUnauthorized error in maas.log [01:35] has this ever worked? [01:36] OAuthUnauthorized [01:37] have you ever had a DNS entry updated correctly? [01:38] bigjools, yes [01:38] rawang: ok so what has changed since then? [01:38] bigjools, nothing changes [01:38] do you keep a log of anything you configure? [01:38] bigjools: ok it seems it might be the versioning being used [01:39] roaksoax: yes, exactly right. it needs a pristine tar for that style [01:39] bigjools, we added a node in maas, enlist, commission, but juju add-unit, when the unit is started, we're trying to ssh ,and the hostname is able to be resolved. that's about it [01:39] s/but/and/ [01:39] rawang: ok one moment [01:40] ok :) [01:40] bigjools: nope, I think it might be a bug somewhere..., the error asks for a native package 3.0 (native), when in reality it is a 3.0 (quilt) [01:40] it is trying to build the package as native, and antive packages do not have the debian revisions [01:40] roaksoax: because bzr-builder converts it to native [01:41] bigjools: should it do that thoguh? the package is a 3.0 (quilt) not native [01:41] the problem might be related to having a debian revision i.e. 1.4-0ubuntu1 [01:41] the debien revision is '-0' [01:41] ok [01:42] which is intended [01:42] rawang: ok can you run "maas shell" and then type these lines: [01:43] ok [01:43] from maasserver.models import DHCPLease [01:43] DHCPLease.objects.all() [01:43] and tell me if you see your node's IP [01:45] bigjools, the output is truncated, how do i see the full output? [01:45] how many nodes do you have?! [01:46] bigjools, currently 33 [01:46] rawang: try: [01:46] for l in DHCPLease.objects.all(): [01:46] print l.mac,l.ip [01:47] k [01:47] bigjools: can't we tell bzr to not create a native package? [01:47] roaksoax: no idea! [01:47] don't think you can [01:47] roaksoax: go to #launchpad-dev and ask stevek or wgrant [01:48] stevenk even [01:48] bigjools, well, I don't think I have seen my new node's IP [01:48] bigjools: k, i;'ll have to check on tahat tomorrow [01:48] i'm off now [01:48] ttyl [01:48] rawang: ok then there is the problem. [01:48] roaksoax: ok cheers, good night [01:49] rawang: the celery job on the cluster controller parses the dhcp leases file. Can you check to se if it threw any errors, and can you manually check the actual leases file for your mac [01:49] see if there is a host{} entry for it as well [01:51] bigjools, yes, there is a host() entry for new node in /var/lib/maas/dhcp/dhcp.lease file, this node was added/removed a few days ago [01:51] rawang: is there a current lease? [01:51] roaksoax: It will fall back to a native package if thrre's no pristine-tar metadata to allow it to recreate the orig.tar.*. Since you can't really create a non-native package without the orig tarball. [01:54] bigjools, well the node's IP is 10.214.25.175 (we can ssh into via this IP), and we have seen this record in dhcp.leases, the only problem is the record was recorded on 11.20 -> [01:54] https://pastebin.canonical.com/101021/ [01:55] rawang: is that the most recent lease entry? It has expired, which is why maas is not writing the dns entry. [01:55] wgrant: so what's the recommended way to fix this? [01:56] bigjools, that's the most recent lease entry [01:56] rawang: ok, on the node, can you look at syslog to see if there's any dhclient errors? [01:56] is dhclient running? [01:56] let me check [01:58] bigjools, seems like there is no error for dhclient on client side. [01:58] Nov 26 10:55:15 74ctp dhclient: DHCPRELEASE on bond0.203 to 10.214.0.4 port 67 [01:58] 10.214.0.4 is the maas cluster controller's ip [01:59] Nov 26 10:55:24 74ctp dhclient: bound to 10.214.25.175 -- renewal in 17171 seconds. [02:00] bigjools, more dhclient logs https://pastebin.canonical.com/101022/ [02:01] bigjools, so client get the ip, but it does not update the maas lease file with latest date, and maas celery failed to add the new dhcp entry to dns record [02:01] bigjools, is there any command/option to reload maas dns record from dhcp lease [02:01] rawang: so can you find a matching entry in the maas dhcp server log? [02:01] is there a rogue dhcp server? [02:02] bigjools, :) I don't get what do you mean by rogue dhcp server? [02:02] roaksoax: You need to include pristine-tar metadata in the branch [02:02] Using 'bzr import-upstream', for example. [02:02] wgrant: ok, thanks! [02:03] rawang: is there another dhcp server apart from the maas-controlled one? [02:03] you need to find out where the node got that lease from [02:03] bigjools, there are two dhcp services running on maas cluster controller, one for maas managed interface for dns/dhcp , and another dhcpd daemon for the rest of the interfaces. [02:04] rawang: and they are *definitely* serving different interfaces? [02:04] can you find the lease in the other dhcp daemon's lease file? [02:05] bigjools, yes, they are definitely server different interface without any problem for the current nodes [02:06] bigjools, yes, there are two dhcp.lease, one in /var/lib/maas/dhcp/dhcp.lease, the other one is in /var/lib/dhcp/dhcp.leases [02:06] rawang: ok you need to find that lease in a server daemon log or lease file [02:06] I need to know where it came from [02:06] bigjools, the one I paste is from /var/lib/maas/dhcp/dhcp.leases [02:07] ok [02:13] rawang: so is it in /var/lib/dhcp/dhcp.leases? [02:13] yes [02:13] bigjools, so for short, i think new node entry is in /var/lib/maas/dhcp/dhcp.leases file, but maas didn't update it for dns record :) [02:13] rawang: wait [02:13] which leases file is it *current* in? [02:14] it is not in the maas one AFAICT because it expired [02:14] /var/lib/maas/dhcp/dhcp.leases [02:14] which is why it's not in DNS [02:14] rawang: well no, it's not in there [02:14] ok [02:14] can you grep it in the /var/lib/dhcp/dhcp.leases file? [02:14] ie the non-maas one [02:15] bigjools, to grep the different mac from same node from /var/lib/dhcp/dhcp.leases? [02:16] rawang: grep the IP address [02:16] ok [02:17] bigjools, yes i can [02:19] rawang: and this is the IP for the interface on the maas-dhcp side? [02:19] you presumably have two IPs [02:20] rawang: the basic problem is that there is no active lease for the interface that is connected to the maas dhcp. You need to fix this, I don't know why it is not working. There is a dhclient problem or other networking problem you're not telling me about yet. [02:21] bigjools, in short, I can get that node's ip from both dhcp leases files [02:22] rawang: then you have a fundamental network setup problem [02:23] somehow your node has got an IP from the wrong dhcp server [02:23] maas cannot assign a dns entry unless its own dhcp server is used [02:25] sorry, keep busy on looking at other problem, will be back soon [02:56] bigjools, problem solved, it's because we have another workaround for solved dns problem, which leads to this one :( [02:56] bigjools, thanks a lot :) [02:57] rawang: ok === Spideyman_afk is now known as Spideyman === freeflying is now known as freeflying_away [04:34] jtv: yo. So, I need to add something to periodically look for existing dhcp servers. [04:34] I was thinking of adding it to the lease parser job, what do you think? [04:34] Yes. [04:34] Combining async jobs sounds hazardous. [04:35] well I thought of doing it in pserv but it's a bit of a job [04:35] since there's no credential storage there [04:35] Ah. Credentials. [04:35] yes credentials [04:36] but with the leases job I can just call one more api endpoint and we're off [04:37] What if we decoupled the probing and the reporting? [04:37] in what way? [04:37] I'm open [04:37] Where one periodic job says "start probing for DHCP servers, boys" and then another (such as the lease-parser job) reports any DHCP servers that have been seen. [04:37] Eliminates the wait, if you see what I mean. [04:38] Although maybe I'm optimising prematurely there. [04:45] bigjools: still here? [04:45] jtv: sorry, waylaid by something, back a in sec [04:45] in a* [04:45] ok === freeflying_away is now known as freeflying [04:59] jtv: so remind me how often we parse leases? [04:59] 30 seconds wasn't it? [05:00] * jtv looks [05:00] bigjools: once a minute [05:01] Guilty thought: "send DHCP probe; parse DHCP leases; send DHCP leases to API; sleep for remainder of DHCP wait period; send DHCP results; done." [05:11] jtv: that's exactly what I was thinking :) [05:12] Then we're both guilty. [05:12] jtv: however, we'd need to do non-blocking IO [05:12] Not this way, AFAICT. [05:12] so I think sequential is fine [05:12] *Sending* the packet should be instantaneous. [05:12] We can start *receiving* once the leases parser is done. [05:13] we can [05:13] (Never mind the "sleep," I now see: the blocking recv() will do that) [05:13] in the example code I did I used a settimeout [05:13] so it waits ~3 seconds and then returns [05:17] Yeah. For us, the latency of parsing and reporting the DHCP leases would probably hide the latency of probing for a DHCP server, _if_ there is one. [05:18] Otherwise, well, at least we could subtract the elapsed time from the timeout. [05:19] there's very very little latency. [05:19] you usually get a response subsecond [05:31] Shouldn't we cater for the situation where there is no DHCP server? [05:35] of course, we'll update the interface every scan [05:53] jtv: I'll submit a branch now that has the api side changes so you can see my intended solution to that [05:58] jtv: https://code.launchpad.net/~julian-edwards/maas/dhcp-detect-pserv/+merge/196659 [06:27] I got an error while PXE booting : http://picpaste.com/20131126_122922-oSwGs8An.jpg [06:27] any clue for this? [06:29] anyone awake? I'd like to see the agent_name for nodes in my MAAS server; what maas-cli invocation do I need to make? [06:31] bigjools, jtv: ^^ is agent_name rendered in output ? I see it in models/node.py and api.py, but not explicitly in any of the templates [06:31] no it's not part of templates [06:32] what problem do you have? [06:32] jam: your idea works - I can just nodes list agent_name= [06:32] bigjools: if we acquire using one, we wanted to ensure we did that correctly [06:32] bigjools: I started a new instance in MAAS, just wanted to ensure it had the same agent_name as the others [06:32] we were hoping to see it in "nodes list" but you just have to guess the agent_name and filter yourself [06:32] well "guess" in that we have it written elsewhere [06:32] yay rushed fixes [06:33] ok so axw did you answer your own question? :) [06:33] bigjools: jam answered it for me, thanks :) [06:33] ok [06:34] may as well stay here while I learn how to use maas ;) [07:05] Morning jtv. In your review here https://code.launchpad.net/~gmb/maas-test/warn-on-virtual-hardware/+merge/196489 you say "we should be okay w.r.t the change..." (unicode vs bytes again). Is "should" indicating an obligation or is it meant to indicate likelihood? === CyberJacob|Away is now known as CyberJacob [08:46] Hi gmb — I meant likelihood. My apologies for the ambiguity. [09:12] gmb: about that pidfile... see the two pastebin links on rvba's MP: https://code.launchpad.net/~gmb/maas-test/accept-proxy-config-option/+merge/196664 [09:14] jtv: Yep, looking, thanks. [09:15] Oh wait, it's your MP. :) [09:16] Yep :) [09:16] On this massive screen I'm in front of, the name at the top is just too far out of the way. Raphaël's was right in front of me. [09:19] jtv: did you make any changes to the dhcp detection stuff yet? I need to start doing things with it and we might tread on each other's toes here [09:21] bigjools: yes. How about I push it now? [09:21] jtv: sure [09:21] I also have a script that I created for experimentation... Might as well include that I suppose. [09:22] bigjools: pushing to lp:~jtv/maas/probe-dhcp [09:22] And... done. [09:23] pwobe [09:23] bigjools: adds a scripts/maas-probe-dhcp.py for easy experimentation. [09:23] jtv: make a WIP MP so we get a working diff [09:23] OK [09:52] Can global kernel options be set through the cli ? maas-cli maas maas get-config 'kernel_opts' is returning "u'kernel_opts' is not a name=value pair" [09:52] s/set/displayed/ [09:58] gnuoy: name=kernel_opts IIRC [09:58] bigjools, perfect, thank you === freeflying is now known as freeflying_away [10:39] gmb: oops, I just voted Disapprove on your branch but it looks like it's already merged. [10:39] jtv: HOW DARE YOU. [10:39] I mean [10:39] Why, what was wrong with it? [10:40] I am sorry for being so negative, really I am. Basically, you were trying to stuff two fixtures into one body. [10:40] The thing has become wholly schizophrenic. Two personalities, no shared code. [10:40] jtv: Nah, it's fine; I'd rather have your negative reviews than most people's positive ones. And you're absolutely right, too. I'll put a branch together to take care of your points now. [10:43] Thanks. Very sporting of you. [10:45] jtv: since you seems to be in reviewing mode, could you have a look at https://code.launchpad.net/~rvb/maas-test/customize-series-arch/+merge/196685 please? [10:45] seem* [10:46] * jtv was trying to get _out_ of reviewer mode for a change === freeflying_away is now known as freeflying === CyberJacob is now known as CyberJacob|Away [11:08] jtv: all right, no worries, I'll ask someone else to pick it up then. [11:10] Thanks. [11:11] allenap: time for a review? https://code.launchpad.net/~rvb/maas-test/customize-series-arch/+merge/196685 === CyberJacob|Away is now known as CyberJacob [12:01] jtv, rvba: A branch to deal with the concerns about the overloading of ProxyFixture: https://code.launchpad.net/~gmb/maas-test/fix-buggery/+merge/196697 [12:01] * gmb -> lunch [12:01] Looking. [12:02] jtv: Might want to refresh; just pushed another revision. [12:03] OK [12:04] I was wondering about that class you just removed. :-) [12:07] gmb: done. [12:25] rvba: I'm beginning to think that we need a separate version of packages.txt for packages that we expect to be installed for development. [12:29] rvba: Sure. [12:54] Does anyone remember what to do about this one? [12:54] Getting distribution for 'psycopg2>=2.4.4'. [12:54] Error: Picked: psycopg2 = 2.5.1 [12:55] allenap..? [13:00] jtv: I tried to help someone with this last week but it was futile. Let me see if I can recreate it. [13:00] jtv: Is this in maas or maas-test? [13:01] allenap: maas [13:01] saucy [13:01] gmb: I'm getting a failing when running the tests on trunk… http://paste.ubuntu.com/6478833/ [13:01] rvba: Fuckmonkeys. [13:01] This seems related to landing of your most recent branch. [13:01] Yeah, that was my bad, I've missed a test run. [13:01] Fixing it now. [13:02] I think it's trying to import RemoteProxyFixture which does not seem to exist. [13:02] Yep [13:03] jtv: make install-dependencies? [13:03] jtv: I can reproduce by removing python-psycopg2. [13:04] allenap: I ran that of course. [13:04] rvba: https://code.launchpad.net/~gmb/maas-test/agiantpileofarse/+merge/196709 [13:04] I started hitthing the problem without python-psycopg2 installed. Installing it didn't help. [13:04] jtv: Ah, I can also reproduce now I've installed python-psycopg2! Dammit buildout. [13:04] "can we have CI now? Thanks." says gmb. Seconded :) [13:04] :) [13:05] allenap might have a secret script to spin up a juju-deployed tarmac. [13:05] jtv: rm -r ~/.buildout/eggs/psycopg2-*.egg [13:05] Actually, without python-psycopg2 it fails even earlier. :( [13:05] allenap: no ~/.buildout. [13:05] Or wherever your buildout cache is. [13:06] Removing the eg... [13:06] egg... [13:06] Nope, back to the same error. [13:06] This is a freshly installed machine, so no old eggs. [13:07] jtv: After `make distclean`, does `bzr ignored` report anything? [13:08] allenap: bin ./bin [13:08] etc. [13:08] Same for eggs, develop-eggs, lib, local, and parts. [13:08] rvba: Um, packages.txt seems to be broken: http://paste.ubuntu.com/6478860/ [13:08] jtv: It should have reported nothing. [13:08] (Just removing the comment fixes it, but still) [13:08] allenap: what does that mean? [13:09] gmb: uh-oh, my bad [13:09] jtv: make distclean should have removed all non-versioned files from the tree. [13:09] gmb: I'll clean it up… [13:09] I'm still waiting for the diff to show up on your MP… [13:09] rvba: Okay. And another +1 in the CI column... [13:09] rvba: Yeah, LP seems to be on a go-slow today. [13:10] allenap: ah! "make distclean" fails because "svok" can't be found. [13:10] jtv: Interesting! daemontools is probably missing from the package list. [13:10] Mmm, it's there. [13:11] rvba: To save you waiting: http://paste.ubuntu.com/6478874/ [13:11] Woah, hang on [13:11] rvba: Scratch that, I appear to have done something dumb. [13:11] One sec. [13:11] allenap: I also had to install libpq-dev and libpython-dev [13:12] jtv: Something weird is going on: libpq-dev is in required-packages/dev. libpython-dev is missing, granted. [13:12] allenap: make distclean works now, "bzr ignore" says nothing, but still no psycopg2. [13:12] Wha!? "make install-requirements" gives me some additional packages now! [13:12] rvba: http://paste.ubuntu.com/6478881/ [13:13] Ta. [13:13] allenap: whoops... lots and lots of additional dependencies in fact. [13:14] gmb: approved, and I fixed the packages.txt fuckup. [13:14] Cool. [13:14] rvba: Thanks. [13:28] allenap: success! Installation must simply have failed somewhere. [13:30] allenap: thanks for the review… I'm afraid my change will add more conflicts to the 2 branches you have in progress but solving them will be mostly mechanical… [13:32] rvba: Don't worry about it. [13:32] jtv: \o/ [13:33] That *finally* got to the point where I can test what I wanted to, and the news is bad. :( [13:33] Can't bind to the dhcp client socket when dhclient is running... even on another interface. [13:39] jtv: http://freeprogrammersblog.vhex.net/post/linux-39-introduced-new-way-of-writing-socket-servers/2 might help. [13:40] gmb: fwiw, MAASFixture.configure_default_series does exactly the sort of config you have to do to make the documentation about --http-proxy be true (i.e. for the proxy to be actually used by the nodes) :). [13:40] Thanks allenap. Will look at that tomorrow. [13:43] gmb: also, the branch I just landed is the cleanup you and I talked about a few days ago. The arch and the series for which the images are imported is not hardcoded anymore. [14:11] rvba: Is there any need to keep TestMAAS and TestOneNode separate? [14:13] rvba: Sweeeeeet. Thanks for the tip. [14:13] rvba: I guess TestOneNode has the actual cases, whereas TestMAAS is just utility-like code. [14:13] I'm getting TypeErrors... WTF? I'm going to make tea in anger. [14:25] allenap: yes, something like a mixin… I decided to split them up because at some point, it look as if we were about to have multiple test cases… turns out we only needed one so far. [14:25] rvba: I won't change it, but I will document it. [14:25] Sounds good. [14:51] gmb: I see something fishy in maastest/main.py: we pass cls.proxy to MAASFixture() instead of proxy_url. Looks like a problem to me… [14:52] rvba: Yeah, that's a bug — we don't actually have tests for the TestCase infrastructure, which doesn't help — I've got a fix for that in a branch now. [14:52] All right. [14:53] Also, I don't see cls.proxy.setUp() being called. [14:56] rvba: Yep [14:57] gmb: where will polipo put its cache? It's not obvious from the code… [14:57] (I'm asking because I can't have it put large files in ~/, I need to create a symlink to a place where there is space) [14:58] rvba: It goes in /var/cache/polipo by default. [14:58] It can write there even if it's running with my id? [14:59] I doubt it :) [15:00] The log is full of: [15:00] Couldn't create disk file /var/cache/polipo/archive.ubuntu.com/Uh9N+CMWid7p9wF5hBk53Q==: Permission denied [15:00] Couldn't create disk file /var/cache/polipo/archive.ubuntu.com/Q3DSIlpgswiWnUOrv3cdXQ==: Permission denied [15:00] rvba: Hmm, good point, I doubt it too... What an excellent feature! [15:00] :) [15:00] Okay, so we should fix _that_ then. [15:01] gmb: I'll see if I can change the config to cache things in .maas-test/cache [15:01] ok. [15:01] On the bright side, it does not cache anything but it does not break :) [15:02] Yes... that's the kind of failure we like: agreeable uselessness. [15:03] rvba: LP's being slow again; does this look good to you: http://paste.ubuntu.com/6479326/ [15:03] gmb: that's precisely what I've done to test this thing. [15:04] Excellent. I'll merge that now. === teknico_ is now known as teknico [15:31] any activity here ? [15:33] blueking: Plenty. What kind of activity were you looking for? [15:36] I've tested sphirewall/debian on new built pc but seems to be hard to configure nat not working properly, might be due new nic I210AT intel ... so looking into ubuntu server to be used on router/nat box/firwall and torrent client [15:36] have used ubuntu desktop version before [15:38] hardware on router are [15:38] mobo http://www.supermicro.nl/products/motherboard/Xeon/C220/X10SLM_-F.cfm [15:38] blueking: I suspect that maas is more than you need for what you're trying to do. You might be better off asking in #ubuntu or #ubuntu-server [15:39] ok [15:39] same irc server ? [15:40] blueking: Yes. === CyberJacob is now known as CyberJacob|Away [15:47] allenap: would you be available for a review? === CyberJacob|Away is now known as CyberJacob [15:52] allenap: here is the MP, if you have time: https://code.launchpad.net/~rvb/maas-test/fix-cache-config/+merge/196737 [16:16] gmb: did you start working on setting MAAS' proxy? [16:27] gmb: I'll take that as a no :) [16:27] gmb: can you have a look? https://code.launchpad.net/~rvb/maas-test/fix-proxy-usage/+merge/196745 [16:27] allenap, maybe ^ [16:46] rvba: Sure. === CyberJacob is now known as CyberJacob|Away === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying