=== lifeless_ is now known as lifeless [01:18] smoser: around ? [01:19] smoser: I'm curious, in http://bazaar.launchpad.net/~smoser/maas/maas-pkg-test/view/head:/maas-ephemeral-test-quantal.txt , why you're using libvirt, which will run dnsmasq, when you want maas to be managing dhcp and dns [06:39] jam, jelmer: you're not experts on “bzr builddeb” by any chance, are you? [06:40] heh [06:40] jtv: jelmer is, I am not. I don't think he's online for another hour, though [06:40] Any idea who might be? [06:41] (And don't say William, because he said to ask former bzr people!) [06:42] This failure looks similar: http://www.mail-archive.com/openstack-ubuntu-testing-notifications@lists.launchpad.net/msg00290.html [06:44] Hi rvba [06:44] Hi jtv. [06:45] rvba: could you help me with an experiment? [06:45] jtv: sure [06:45] The package build seems to have broken. [06:45] So please, _don't_ update your packaging branch just yet, [06:45] go into your packaging branch, [06:45] and type: [06:46] bzr bd -- -k [06:46] Or first, install your build-deps: [06:46] sudo apt-get builddep maas [06:46] *build-dep [06:47] That may install some packages that might be missing. [06:47] And after that, the “bzr bd” command line _should_ build a package — but currently doesn't. [06:47] I don't like installing the maas package on my dev machine but I guess I'll just purge it after the experiment :) [06:48] No need to install it. [06:48] Just the build dependencies. [06:48] Ah right. [06:49] jtv: http://paste.ubuntu.com/1214273/ [06:52] rvba: then I guess that's a failed experiment... means you don't have the required bzr plugin. :( [06:52] Nobody seems able to help with this at the moment, so I'll just start bisecting. [06:52] Well, I _have_ started bisecting. [06:52] Thanks for trying though — it was worth a shot. [06:52] I can install the right plugin :) [06:53] It's called builddeb. [06:57] jtv: looks like I need to install quilt (http://paste.ubuntu.com/1214283/) [06:58] Ah yes, that too. [07:03] jtv: no error it seems: (that's the end of the output) http://paste.ubuntu.com/1214288/ [07:03] ! [07:04] What revision is your packaging branch on? [07:04] 971? [07:04] No, that's got to be the trunk revision. [07:05] More like 78. [07:05] That one's failing for me, I think. [07:05] What's the error? [07:06] No wait, I was looking at 87. :) [07:06] The patching of the network settings maas_local_settings.py fails. [07:06] Ok, let me try with that revision. [07:06] Quilt calls patch, patch says it can't find the file it should patch. [07:07] Then it gives you the impression that it continues regardless, but the return value is 3, not 0. [07:07] bigjools was having that [07:07] I think [07:07] Yes. [07:07] yes [07:07] I take it the taxi didn't take too long then lifeless [07:07] bigjools: apparently not [07:07] bigjools: was at the airport at 25 to [07:08] nic [07:08] e [07:08] bigjools: thanks again for the power adapter. SHEESE. [07:08] did you go in the tunnel? [07:08] lifeless: I saw it an thought I'd better dig up your number, yes :) [07:08] there was a tunnel yes [07:08] Dunno if its 'the' tunnel [07:09] * jtv forgets what power plugs NZ uses… not the Australian kind then I take it? [07:09] Same [07:09] just wouldn't be useful to me if I was in NZ and it was in Julians house. [07:09] I was lucky to find a converter for that in Buenos Aires… just prior to getting stuck at the airport for 5 hours or so with only Australian-style sockets for some reason. [07:09] Ah! [07:10] Now, _why_ would dh say “unknown sequence get-packaged-orig-source”? [07:10] lifeless, any ideas? [07:11] jtv: http://en.wikipedia.org/wiki/AC_power_plugs_and_sockets#Australian_standard_AS.2FNZS_3112_.28Australasian_10.C2.A0A.2F240.C2.A0V.29 specifically :P [07:11] jtv: same problem here: http://paste.ubuntu.com/1214295/ [07:11] jtv: I've no idea, I haven't, and don't plan on, used/using bzr-builddeb that way :) [07:11] rvba: thanks — that puts it somewhere between r78 and now. [07:11] jtv: I'm fairly sure thats not a standard target [07:12] Clearly it isn't. [07:12] But then who invokes it and why? [07:14] Looks like r81 is OK. [07:14] check your rules file [07:15] There is a target of that name there. [07:15] What am I looking for? [07:15] r81 is OK, r85 is broken. [07:16] jtv: so - http://www.debian.org/doc/debian-policy/ch-source.html - no mention of your custom target. [07:17] Is it a custom target? [07:17] to find what invokes it, you probably need to grep around and follow every possible lead [07:17] yes, its not part of the stock debian toolchain [07:17] it's a builddeb target [07:18] That's strange. William/Steve mentioned the question of whether the package supported get-packaged-orig-source, and having the target in the rules file was more or less the definition of support. [07:18] it looks like a builddeb special [07:18] https://code.launchpad.net/~jelmer/bzr-builddeb/get-packaged-orig-source/+merge/83980 [07:22] Looks as if the addition of tests broke things: http://bazaar.launchpad.net/~maas-maintainers/maas/packaging/revision/82 [07:24] I need to afk for a bit. But r82 of the packaging branch seems to be the problem. It doesn't help that I reviewed it. :) [07:25] bigjools: so do we need to teach the launchpad email parser about pirate speak? merge: be approved [07:27] rejected could be: "merge: to davy Jones' Locker wit ye" [07:54] jam: your MP is going to walk the plank, arrrr [08:13] jam: what bigjools said. And likewise if you extend bzr's revision specs to accept “arrrr931” [08:25] :) [08:27] the "bring out your dead" van is going down the street [10:52] jtv: there is a rule for 'get-orig-source' vs 'get-orig-packaged-source' I wonder if they could be combined somehow? [10:53] jam: I have absolutely no idea. [10:53] I wonder if builddeb is thinking it can pass a rule down to 'dh' and it will run the equivalent of make -f debian/rules $RULE [10:53] I would guess offhand that builddeb is ignoring that failure as not supported [10:54] unfortunately our master jelmer is still feeling quite ill today [10:56] jtv: I do see this in the builddeb code: http://paste.ubuntu.com/1214596/ [10:56] Yes, I get that in the output. [10:57] Maybe it's a simple matter of disabling get-orig-source? [10:57] however, I see no such thing in http://www.debian.org/doc/debian-policy/ch-source.html [10:57] which seems to say 'get-orig-source' is the right thing to do. [10:57] jtv: looking at the bzr-builddeb code, it would be as simple as doing: [10:58] get-packaged-orig-source: get-orig-source [10:58] so the 'make -f debian/rules' will be happy [10:58] Unfortunately I'm running out of time for experimenting with this… do you get the failure too? [10:59] jtv: I'm not sure how to 'build the package' where are you running 'bzr builddeb' ? [10:59] and not (bzr builddeb -S or whatever those things are) [11:00] jtv: I don't see the dh failures, I see 'Trying to run get-packaged-orig-source' which then cleanly falls back to 'get-orig-source'. [11:00] cd into a packaging branch and run “bzr builddeb -- -k” [11:00] I'll see if I can trigger the failure you're seeing [11:00] right now it is still exporting [11:00] and doing it from lp:maas even though I have a local copy :( [11:00] Be sure to try this on r82 or higher. [11:01] Yeah, it sure likes slow downloads. [11:01] I'm on 87 [11:01] yeah, it is re-downloading all of maas [11:01] because that is what 'make get-orig-source' says to do [12:04] It looks like I broke Jenkins, but I can't get the 10.189.74.2 to load [12:04] to find out any actual details [12:04] and the public one just says "No Changes" === matsubara-afk is now known as matsubara [12:33] lifeless, i do not have libvirt running dnsmasq in that. at least if i did then i had meant to disable that. [12:34] i chose to use libvirt becaues it fairly simply a.) sets up nat b.) sets up bridges c.) starts the stuff on boot [12:34] smoser: it seems highly likely that your emulated network is the cause of the fault, is all. [12:34] d.) will eventually allow me to plug in virsh power control for a set of systems (similar to how vdenv does it) and let maas power on and off systems. [12:35] lifeless, why would you say that? [12:35] IIRC the symptoms you had were being unable to reach the tftp server, which is a networking issue, if the tftp server service is running. [12:35] well, no [12:36] see the last comment there [12:36] https://bugs.launchpad.net/maas/+bug/1051626 [12:36] Ubuntu bug 1051626 in MAAS "next-server written wrong" [Critical,Confirmed] [12:36] its strange. the tftp *does* work. it *does* get pxelinux.0 in both cases. [12:37] the only thing i could guess was that pxelinux was making a decision based on the netmask of its IP adn the address of the server. [12:37] and chaning its behavior. [12:40] lifeless, does that make any sense to you? [12:40] because it didn't make any sense to me. [12:40] and all i could really think was, same as you, it didn't really get the pxelinux.0 file [12:40] but it says it did. [12:43] smoser: sorry, its nearly 1am and I just got off a plane, I'll follow up tomorrow [12:46] sure. [12:46] i notcie that i am running a dnsmasq for dns [12:46] but it is not running dhcp [12:46] so that is not the issue [13:24] anyone want to pick this off really quick: [13:24] https://code.launchpad.net/~smoser/maas/trunk.enlist_minor/+merge/125199 [13:24] jtv: maybe it is becuase you issue bzr debbuild instead ob bzr builddeb (or bzr bd) [13:24] roaksoax, [13:25] smoser: lookin [13:25] looking* [13:26] * roaksoax has a crappy netework connection today :( [13:27] smoser: done [14:28] roaksoax: Hi there. jam approved your branches but some tests are still missing. I'm working on it right now. [14:32] rvba: yeah I saw and got them approved and they got merged [14:34] roaksoax: all right. That's you unblocked then. Gavin and I will work on adding a few tests. [14:34] rvba: awesome thanks [14:35] rvba: now, I've been thinking on getting the available releases vs dynamically adding new ones. I just realized that enums.js is created on packagebuild so if any release is not available during build time, it won't be shown on the UI at all if it becomes available later on, right? [14:36] roaksoax: indeed. [14:37] roaksoax: we could add an API method to get the available release and use that method in the UI. [14:37] releases* [14:39] rvba: yeah I think something like that would be necessary. I have a few improvements ideas too related to maas-import-pxe-files too so i'm trying to write them down and i'll send an email [14:39] roaksoax: cool [15:26] roaksoax: now it's my turn, I need your help with a change I need to make in the packaging. It's related to bug 1050492. I've created the following MP: https://code.launchpad.net/~rvb/maas/packaging.set-rabbitmq-creds/+merge/125248. [15:26] Launchpad bug 1050492 in maas (Ubuntu) "MAAS uses the 'guest' account to communicate with RabbitMQ" [Critical,Triaged] https://launchpad.net/bugs/1050492 [15:26] * roaksoax looks [15:27] rvba: so that password is only on maas_local_settings.py? [15:28] roaksoax: yes. And then every cluster that will be accepted in the region controller will get these credentials. [15:29] rvba: ok so you want me to generate a password and set it on maas_local_settings.py? [15:29] roaksoax: that's what I tried to do in that branch yes. Generate a password, create the rabbitmq user, and then set BROKER_URL in maas_local_settings.py. [15:30] roaksoax: configure_maas_workers_rabbitmq_user is very similar to configure_maas_txlongpoll_rabbitmq_user. [15:31] rvba: where's the branch? [15:31] roaksoax: see the MP address I pasted above. [15:31] https://code.launchpad.net/~rvb/maas/packaging.set-rabbitmq-creds/+merge/125248 [15:33] rvba: alright, i'll review it and merge it right after I finish something :) [15:33] roaksoax: cool, thanks a lot. I've got a few questions but let me know when you'll be available. [15:34] rvba: give me say 20 mins [15:34] Sure, no problem. [15:40] roaksoax: I just realized that if we land this now it will break the package because the cluster controller still relies on the fact that we're using the default credentials to connect to RabbitMQ. My change needs to be landed when we will have a separate maas-cluster package. [15:41] rvba: ok, so that just made me think something. So cluster controller is the master MAAS, and what are you calling the client MAAS? [15:42] how will the client MAAS know the password to connecto to RabbitMQ? [15:42] The region controller is the master MAAS. [15:42] So the plan is this: [15:43] When a cluster controller is installed, a debconf question asks for the address of the Region controller. [15:43] Then a script connect to the API of the region controller and says: I'm the cluster controller with UUID=1234. [15:44] The on the region controller, an admin has to accept that cluster controller. [15:44] The cluster controller then polls the region controller, waiting for an admin to accept the cluster controller. [15:45] So the answer of the region controller to the cluster controller is: "hold on, an admin need to accept you" [15:45] When an admin accepts the cluster controller, then the answer changes (remember the cluster controller is still polling). [15:46] The answer now contains the credentials so that the cluster controller can now connect to RabbitMQ. [15:46] rvba: right, the anwer that contains the credentials, need to be written on maas_local_settings? [15:47] The cluster controller will do that each time the service is started, but, in my example, next time the cluster controller starts, it will already be accepted in the region controller so the credentials will be sent right away. [15:47] rvba: right, but when it has the response with the credentials, does it have to write them somewhere? i.e. maas_local_settings? [15:48] roaksoax: well, yes. For two reasons: a) the region controller need to be able to send tasks via RabbitMQ and b) the region controller need to be able to send the credentials to the cluster controllers. [15:48] rvba: ok, so that process or writing into maas_local_settings will be done wher? upstart? [15:48] or within the django code? [15:48] cause note that packaging cannot do that in that particular case [15:48] roaksoax: no, the cluster controller will use the credentials to start celeryd: celeryd -borker-url=123 [15:49] broker-url* [15:50] right [15:50] so the package will only install the cluster controller [15:50] and after installation has finished [15:50] then the cluster controller will modify maas_local_settings.py [15:51] There is no maas_local_settings.py on the cluster side. [15:51] oh ok, but where is this information stored then? [15:52] It does not need to be stored: the upstart script of the cluster will contact the API of the region controller to get the credentials every time it starts. [15:52] ah!! I see [15:59] roaksoax: ok, I've added a card on our kb board. This MP is on hold for now. We need a separate cluster controller package with all the polling logic in place before we can land this. [16:04] rvba: ok, cool. So now, we need to start figuring out what part of the software is that we want on the region, and what on the cluster [16:04] rvba: so, what does the region does not do in comparison to the current package we have. What does the cluster controller do? === matsubara is now known as matsubara-lunch [16:05] roaksoax: indeed. But that's pretty well defined already. The cluster contains pserv, the dhcp stuff and celery. [16:05] roaksoax: Julian started working on that. He wants to touch base with you guys about it. [16:06] rvba: righjt, but we need to figure out what's common code, and what pieces of the code we need. then we need to split up postinst [16:07] rvba: so for example, maas package should now pull both region and cluster controllers and have everything running in one machine [16:07] roaksoax: right. [16:07] the maas-region only the region code [16:07] and etc et [16:07] rvba: which means that, do they all have to install src/maasserver? do they all have to install src/provisioningserver? etc etc [16:08] roaksoax: the region needs all the code. The cluster needs src/provisioningserver. [16:08] rvba: so region *also* needs src/provisioning server then? [16:08] roaksoax: yes. [16:09] roaksoax: that's because of celery. The signature of the tasks are defined in src/provisioningserver. [16:09] roaksoax: but maybe we could have a maas-common package with all the code and then maas-region and maas-cluster could contain only the upstart scripts. [16:09] That's just an idea for now. [16:09] Julian is the packaging expert on our side :). [16:12] rvba: yeah, well let's see what julian says [16:13] roaksoax: yep. I hope he will be able to make progress on this front soon as this is a fairly large change which will require a lot of testing. Plus it's blocking some work that needs to be done. [16:14] right, this is not gonna be an easy thing though :) [16:14] It should. [16:14] roaksoax: you don't overlap much with Julian do you? [16:15] rvba: no I don't unfortunately [16:15] rvba: i don't at all TBH [16:15] rvba: do we also need metadaserver on the cluster I guess? [16:16] roaksoax: no, that's on the region too. [16:17] rvba: so machines deploying will contact the region instead of the cluster controller to get the meta-data they need, say commissioning? [16:17] roaksoax: yes. [16:18] rvba: ok, but what if you put the cluster to deploy nmodes on a particular network that you dont want to access the region? [16:19] roaksoax: I guess we will need to add some sort of proxy in that case. [16:19] rvba: would provisioningserver still be considered django code? [16:19] no right? [16:20] That's celery code really. [16:20] It simply contains celery tasks. [16:20] rvba: pr im othjer words, what's considered django code? maasserver metadaserver? [16:20] Yes, these two. [16:21] And it's not considered django code, it *is* django code :). [16:21] Well, python code based on Django. [16:21] what will the clusters use for storage initially, just the filesystem? [16:22] rvba: so the cluster controller does not need src/maas right? [16:22] roaksoax: no. [16:22] I'm not sure what things are going to look like after the first pass of multi-cluster support either really... [16:22] mgz: I guess so for now, since mongodb has been ruled out. [16:23] but they run some kind of server that exposes an api? [16:23] No. [16:23] They are just celery nodes. [16:23] so nodes on commissioning will post directly to the region controller? [16:24] Yes. [16:24] There is no way around that right now. [16:24] okay, thanks, that clarifies the scope of what I need to do right now quite a bit. [16:25] rvba: so this is what I'm thinking: http://paste.ubuntu.com/1215097/ [16:27] * roaksoax bbl [16:28] roaksoax: looks good to me. There is something else: we need a special controller called the master which is in charge of dealing with master tasks like setting up the DNS. [16:29] roaksoax: Julian mentioned we could use a virtual package for that. That package would basically be a cluster controller package but configured slightly differently (instead of generating a UUID to send to the region controller, that cluster would be called… wait for it… 'master'). [16:30] right [16:30] yeah that' [16:30] s easy to add [16:30] I think i'd need to catch up with him [16:30] anyways [16:30] i'll be back later [16:53] roaksoax: Hello! [16:53] roaksoax: How do you feel about creating a new binary package for the cli? [16:54] roaksoax: I see you're away. I'll email you about it. [16:56] allenap will be available in 10 mins [16:56] roaksoax: I'll be away in 10 minutes, and there's enough detail that it's worth emailing anyway. [17:06] allenap: k === matsubara-lunch is now known as matsubara [17:13] allenap: if you happen to see this, tell me where's your CLI and what does it need and I'll create a new package. it is simple enough to do [17:37] smoser: around? [17:38] roaksoax, here. [17:38] smoser: could you please approve this: https://code.launchpad.net/~andreserl/maas/maas_correct_paths/+merge/125286 [17:41] roaksoax, this grades on me. [17:41] the right fix to that problem is NOT TO SPECIFY FULL PATHS! [17:42] why do people do that? [17:42] why do they want their code to break because a program changes path? [17:43] smoser: i agree we should not need to specify paths, but well... we shoudl raise that upstream [17:43] why do they not want me to be able to put preferred versions of a program in /opt/bin and do the right thing? [17:45] heh [17:53] https://bugs.launchpad.net/maas/+bug/1053025 [17:53] Ubuntu bug 1053025 in MAAS "do not hard code paths to executables" [Undecided,New] === matsubara is now known as matsubara-afk [19:03] smoser: i think you are gonna like this: [19:03] smoser: https://code.launchpad.net/~andreserl/maas/maas_packaging_fixe_bzr1030/+merge/125313 [19:23] smoser: thanks!! === matsubara-afk is now known as matsubara [20:47] * roaksoax lunch === matsubara is now known as matsubara-afk [23:09] juju issue after successful maas install: [23:09] 2012-09-19 18:08:22,490 DEBUG Initializing juju status runtime [23:09] 2012-09-19 18:08:22,503 INFO Connecting to environment... [23:09] 2012-09-19 18:08:22,783 DEBUG Connecting to environment using maas07... [23:09] 2012-09-19 18:08:22,784 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="maas07" remote_port="2181" local_port="59813". [23:09] 2012-09-19 18:08:23,334:3276(0x7f39ca8ec700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5 [23:09] 2012-09-19 18:08:23,335:3276(0x7f39ca8ec700):ZOO_INFO@log_env@662: Client environment:host.name=maasctrl [23:09] 2012-09-19 18:08:23,335:3276(0x7f39ca8ec700):ZOO_INFO@log_env@669: Client environment:os.name=Linux [23:09] 2012-09-19 18:08:23,335:3276(0x7f39ca8ec700):ZOO_INFO@log_env@670: Client environment:os.arch=3.2.0-30-generic [23:09] 2012-09-19 18:08:23,336:3276(0x7f39ca8ec700):ZOO_INFO@log_env@671: Client environment:os.version=#48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 [23:09] 2012-09-19 18:08:23,338:3276(0x7f39ca8ec700):ZOO_INFO@log_env@679: Client environment:user.name=ubuntu [23:09] 2012-09-19 18:08:23,341:3276(0x7f39ca8ec700):ZOO_INFO@log_env@687: Client environment:user.home=/home/ubuntu [23:09] 2012-09-19 18:08:23,341:3276(0x7f39ca8ec700):ZOO_INFO@log_env@699: Client environment:user.dir=/var/lib/maas/media/storage [23:09] 2012-09-19 18:08:23,341:3276(0x7f39ca8ec700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:59813 sessionTimeout=10000 watcher=0x7f39c88906b0 sessionId=0 sessionPasswd= context=0x30f5910 flags=0 [23:09] 2012-09-19 18:08:23,343:3276(0x7f39c55ea700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:59813] [23:09] 2012-09-19 18:08:24,250:3276(0x7f39c55ea700):ZOO_ERROR@handle_socket_error_msg@1603: Socket [127.0.0.1:59813] zk retcode=-4, errno=112(Host is down): failed while receiving a server response [23:09] 2012-09-19 18:08:27,587:3276(0x7f39c55ea700):ZOO_WARN@zookeeper_interest@1461: Exceeded deadline by 911ms [23:09] 2012-09-19 18:08:27,588:3276(0x7f39c55ea700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:59813] zk retcode=-4, errno=111(Connection refused): server refused to accept the client [23:09] oops, that's from juju -vvv status after an uneventful juju bootstrap