[03:39] bigjools: hey [03:39] bigjools: we basically run the installer on every machine today, right ? [03:39] wotcha lifeless [03:39] lifeless: yes [03:39] cool, thanks. [03:39] Just checking my facts. [03:40] I think Daviey was looking at using an image instead, don't know how far he got [03:46] nova bare metal does images [03:46] it writes them using iscsi straight to the bare metal host [03:46] which is an interesting approach [03:50] lifeless: can iscsi be proxied? [03:51] yes [03:51] there are proxy initiators around according to a quick google [03:51] but failing that, its routable [03:52] proxying would be useful if you need to support action at a distance or handle 3rd party actor scenarios [03:54] I thought we were doing images by now [03:55] only difficulty there is storage configuration. [03:56] lifeless: well I am thinking whether to send it via the clusters or just get everything to download from the region [03:56] will need to do the latter eventually [03:56] err former I mean [05:38] "make test" broken on quantal: buildout complains about selecting amqplib 1.0.2. :( [05:38] bigjools: are you getting that? [05:38] jtv: I use "make check" successfully [05:39] Broken for me, in trunk. :( [05:39] * bigjools tries again [05:39] deleting all my eggs first [05:39] Looks as if it's that don't-pick-your-own-version setting [05:40] well amqp is not in buildout any more [05:40] we're using system packages [05:41] but I just cleaned everything out, building from scratch now [05:42] FWIW it's for building maas-test. [05:42] seems ok here [05:43] have you got latest trunk and installed all package deps? [05:43] Ah, I hadn't installed dependencies! [05:43] * bigjools rolls eyes [05:43] * jtv cowers [05:50] Still no luck with the dependencies updated though. [05:53] Ah, _also_ had to delete the egg. [05:56] you forgot one thing... [05:56] http://www.youtube.com/watch?v=25q3hxlgvw4 [05:56] you forgot.. to hook up.. the doll [05:59] I'll try that, but better if I do it on Friday afternoon, right? [06:09] certainly better for attendance at the party [07:25] Good morning rvba [07:26] "do-release-upgrade -d" on precise is telling me there's no upgrade ... [07:27] \o jtv. [08:02] jam: question about your tag-list-api branch. Why is "nodes" a POST and not a GET? [08:11] jtv: because if I make it a GET, it overrides the default GET, and I can't get the regular tag back [08:11] jtv: if you know of how to add a GET with an op and preserve the get without an op [08:11] I would love to hear it. [08:12] Maybe rvba knows? [08:12] jtv: I agree it should be a plain GET [08:12] I suppose I could make it "/tags//nodes" or some other URL [08:15] jam: I forgot about that problem. I know of no solution. [08:15] jam: you're right, that's not possible with what we have in place right now. [08:16] rvba: k, do you know of a reasonable compromise? POST/nodes path/something else? [08:16] (I went with POST) [08:18] jam: I think this can be properly fixed by improving how the api_operation thing works. otp right now, let me think about it some more. [08:18] k [09:15] jam: the problem you're having is related to bug 1049933 (perform_api_operation() should always use the default CRUD method if op is not defined.) [09:15] Launchpad bug 1049933 in MAAS "The public-facing API code confusingly conflates CRUD, HTTP methods, and piggybacked operations" [Medium,Triaged] https://launchpad.net/bugs/1049933 [09:15] jam: I suggest you XXX that code (and keep it a POST method for now). I think the fix can be simple, I'll look into it. [09:16] rvba: thanks for the heads up. Will do. === jam2 is now known as jam [09:44] bigjools: I guess we don't have the bit of packaging yet that asks the cluster admin where the region controller is? [09:45] jtv: we don't [09:45] that will come last when the upstream code is finished [09:45] I'll have to use some kind of placeholder and file a bug/card. [09:45] you just need to look in a config [09:45] yeah we need to decide on a common config [09:46] I think I did /etc/maas/maas_api.conf [09:46] MAAS_URL= [09:46] inside it [09:46] and I'll make the package write thjat [09:46] I already did some code to do that if you look in my WIP branch [09:47] jtv: why do you want another card? [09:48] If I understand you correctly, it's something I can't finish with the rest of the work. In which case we need one card to track the work I'm doing now, and another to track that capstone piece of config. [09:48] jtv: I think it's ok to work with this minimal spec as above [09:49] just read the file and use that config format [09:49] OK. I didn't know you were already working on that packaging change. [09:49] yeah I had forgotten I had started it :) [09:51] If you're reading that file, ideally I'd reuse your code for doing so. [09:51] I am just writing it [09:51] here's my code [09:52] echo "MAAS_URL=$RET" > /etc/maas/maas_api.conf [09:52] :) [09:53] I sometimes get my technical terms confused, but I could have sworn that was called "writing" the file. :) [09:53] I know -- I'll make the URL a command-line parameter. [09:53] [19:51:31] If you're reading that file, ideally I'd reuse your code for doing so. [09:54] "reading" [09:54] :) [09:54] Yes, and what you showed me was "writing" the file. [09:54] like I said! [09:54] I was being facaetious [09:55] I'm still confused. I said I'd like to reuse your code for reading the file, and you showed me the code for writing the file... [09:55] But never mind. [09:55] I'll make this a command-line parameter and that's an easy way out of the temporary problem. [09:56] jtv: my point was that I am not reading it [09:56] just writing [09:57] Ah. [09:57] jtv: make it default to that file name, with an optional override to another [09:57] Wouldn't it be easier and cleaner to source that file in the shell? [09:58] And pass it explicitly to the maas-provision command I'm writing? [09:58] Then the python code doesn't need to know about the file as an extra level of indirection. [09:58] what shell? [09:59] you're not writing any shell [10:00] allenap: that might be the problem, because 0.6.1 is not mentioned anywhere explicitely. [10:00] explicitly* [10:00] jtv: your new command will need to be started from an upstart conf [10:01] jtv: there's no shell as such, it needs to read the config [10:01] making sense? [10:01] ok it's 8pm here, I am ducking out [10:01] good night all [10:01] The upstart conf is where you normally run shell code. [10:01] Cheerio bigjools. [10:02] allenap: 'make check' works on the jenkins box so I'm a bit confused… [10:02] \o bigjools. [10:02] jtv: not at all, you just tell it what executable to run [10:02] Can do, I guess. I'm used to it running script. [10:02] Okay, I'll read that file. [10:02] jtv: look at the files in /etc/init/ [10:03] they are config files [10:03] you're thinking of /etc/init.d [10:03] No [10:03] they are shell [10:03] I'm thinking of upstart scripts. [10:03] They can have script sections. [10:03] rvba: rabbitfixture depends on amqplib >= 0.6.1. [10:03] rvba: Maybe we can turn off allow-picked-versions... [10:03] Or on even. [10:04] jtv: ok I'll leave it to you to decide how best to do it [10:04] bigjools: I just picked one at random, for resolvconf, and it contains a script section that runs the daemon. [10:04] OK. Good night! [10:04] jtv: it does, can't say I like it, but it does :) [10:04] :-P [10:05] jtv: but open(file,"r").splitlines().split("=") is not too complicated :) [10:05] * jtv reels [10:05] with a read() somewhere in there [10:06] * bigjools exits, bye! [10:06] bye [10:06] I'm stepping out for a bite too. [10:07] allenap: changing allow-picked-versions seems to fix the problem indeed. At least locally on my machine :) [10:15] rvba: Cool. Can you check that it's using the local lxml rather than something from a random egg? [10:17] rvba: And, if so, are you happy to propose those changes, and I'll review? [10:17] allenap: it's still downloading lxml (lxml-2.3.5-py2.7-linux-x86_64.egg) and amqplib-1.0.2-py2.7.egg :/ [10:17] Balls. [10:17] At this point we may need to ditch buildout and just use virtualenv+pip. [10:19] allenap: are you sure it's a good idea to do this right now. Yo're the buildout expert here so I really rely on your expertise for all this ;) [10:20] rvba: I'm not sure it's a good idea to do this right now, but unless we can coax buildout to stop being an idiot, we may have to do it. Can you try one more thing: [10:20] make distclean && rm -rf ~/.buildout/eggs && make [10:20] then see where lxml and co. are coming from? [10:21] allenap: you're not seeing the problem I'm having? [10:21] allenap: running it right now… [10:21] rvba: My machine is still on precise and I've been too busy/lazy to fire up my laptop ;) [10:22] allenap: I'm still on precise too :). I use canonistack to get quantal machines. [10:22] rvba: I've been to busy/lazy to fire up any canonistack instances ;) [10:23] allenap: ok :). But all of this is on precise. [10:23] allenap: and the jenkins box is on precise too. [10:24] rvba: Gah, we've got to get that fixed then. [10:28] rvba: With just http://paste.ubuntu.com/1224227/ against trunk, trunk builds with the packaged amqplib and lxml. [10:28] On precise. [10:30] allenap: I just realized that for some reason, buildout added amqplib = 1.0.2 in versions.cfg :/ [10:30] rvba: Huh? Is that the auto stuff kicking in? [10:30] I've never seen that before. [10:31] allenap: I guess it's the auto stuff. http://paste.ubuntu.com/1224232/ [10:33] rvba: How annoying :-/ Anyway, with that earlier patch to remove lxml from buildout.cfg, I'm getting a clean and, I think, correct build on both precise and quantal. [10:34] allenap: if that's enough then that's great. Now we're back with our Jenkins problem. [10:36] rvba: And the Jenkins machine that keeps kicking me out every 30 seconds. [10:36] rvba: Shall I propose that patch then? [10:37] allenap: please do. I'll investigate the Jenkins problem. So far, I've been unable to reproduce but I'll try again… [10:50] rvba: https://code.launchpad.net/~allenap/maas/lxml-not-a-test-dep/+merge/125969 [11:00] allenap: approved. The test suite runs fine on the Jenkins box when I run it manually. I think I'll wait for Diogo to come online and ask him to help me out with this. This seems to be Jenkins-related somehow. [11:01] rvba: Ta. How &^%$%^& frustrating. [11:15] rvba: Which is the DNS card you were thinking of? [11:17] rvba: Also, do we land to 12.04-nocobbler (precise trunk) by hand? [11:18] allenap: the high priority card in story 2. [11:18] allenap: yes. [11:19] rvba: Okay, I'm going to get some lunch now then I'll ping you about a pre-imp. [11:19] allenap: cool. [11:27] allenap: so, I'm trying to land some of mgz's code because he has poor access, but lp.net is currently taking about 30+s for every action I do. Is it terrible to just land on trunk directly? (given that tarmac doesn't run the test suite anyway.) [11:30] jam: I'll land them if you want, LP's working fine for me. [11:31] allenap: that is very strange. https://code.launchpad.net/~jameinel/maas/arch_constraint/+merge/125979 needs to be marked approved [11:31] jam: But I don't think it's terrible to land directly either. [11:31] but my 'review' vote hasn't happened in 2 min [11:31] I don't know why I'm not getting timeouts, it is possible it is on my end, but a simple HTTPS request shouldn't be particularly slow. [11:32] jam: The Great Firewall Of $Location? [11:32] allenap: well, it is strange, as 'bzr up' happens quickly over ssh, but https submit is being slow. I really wonder what is happening. [11:34] jam: I've approved that mp. I'm going for lunch now, but I'll be back in ~1h. [11:34] allenap: enjoy your food. catch you later. [11:37] allenap: +1 on your lxml change [11:38] we want to use it, too [13:03] allenap: time for a call? [13:04] jam: by the way don't forget my comments on your tag-list-api MP [13:05] allenap: need to discuss a technicality of starting celeryd with you. [13:05] jtv: well, since nobody has approved it, I don't have a choice :). But I'm stil working through the prereqs. I will certainly address them. [13:05] OK [13:06] jam: Once those notes are addressed I think the branch should be good to land -- I just didn't want to jump the gun before you'd had a chance to discuss the GET/POST issue with Raphael. [13:06] Any outcome for that? [13:07] jtv: it is a current limitation of the infrastructure, and we should just land it with POST until we can update things. [13:07] OK [13:07] Then I'm ready to approve, as long as you address those notes before landing. [13:13] jtv: Yeah, in 5; I'm defuckerating buildout with rbasak right now. [13:13] ok [13:31] jtv: Sorry for the delay. Hangout? [13:31] Yup. I'll set one up. [13:32] Done. [14:13] rvba: do you happen to know what (if anything) we need to pass to celeryd besides the broker URL? Doesn't it also need to know its queue name? [14:14] jtv: yes, we need to pass the queue names. But if we don't, it defaults to using 'celery'. [14:14] jtv: I'm currently working on adding proper routing for the tasks. [14:15] I'm working on starting up celeryd from a maas-provision command. [14:15] The queue name is the cluster's uuid, right? [14:15] yes [14:15] For a cluster, the param should be: -Q $uuid,common [14:16] (because there is also a 'common' queue for broadcast messages) [14:16] For the master, the param should: -Q celery,common [14:16] Thanks! Do we store the uuid anywhere on the cluster controller yet? [14:17] (We could s/celery/master but since 'celery' is currently used to route tasks, this will allow you (and I) to land stuff without breaking anything) [14:17] jtv: not yet, that needs to be done by the packaging I guess: generate the UUID and store it somewhere in a config file. [14:18] I guess I'll have to use "celery" for now then. [14:18] And file an XXX for the follow-up. [14:19] Yes, sounds good to me. [14:20] Thanks. [14:20] np [14:31] jtv: I just noticed that src/provisioningserver/tasks.py:remove_dhcp_host_map isn't used anywhere (but in tests). Do we still want to keep that task? [14:31] I've been assuming that we would still want it. [14:31] I think Julian wrote it. [14:32] When we decommission a node, we ought to free up the host map so that dhcpd can reclaim the address eventually, if appropriate. [14:50] allenap: I'm afraid the recent changes made to buildout.cfg et al are causing lots of timeouts :/ https://jenkins.qa.ubuntu.com/job/maas-trunk/836/console [14:53] rvba: Why on earth is it pulling those things? buildout has about as much common sense as a bag full of hungry squirrels. [14:53] I need a cup of tea before I wage war with this again. [14:54] allenap: can you see why https://twitter.com/lmukadam/statuses/249471182510882816? hasn't landed? [14:54] er [14:54] https://code.launchpad.net/~allenap/maas/buildfallout/+merge/126009 === dpb_ is now known as Guest75974 [15:05] jam: I suspect you have a lock in lp:maas. [15:07] jam: I also suspect you no longer have it. [15:09] jam: I now suspect that you've been a victim of a vicious slur. [15:10] rvba, build passed [15:10] rbasak: Tarmac is falling over when getting a lock on lp:~maas-maintainers/maas/packaging and not proceeding any further. [15:11] rbasak: Seems to be working again now. [15:11] thanks! [15:11] matsubara: What did you do? [15:11] allenap, I just forced the build, rvba fixed it :-) [15:12] allenap: I… well… installed python-django-piston :) [15:13] rvba: I thought it was installed!? [15:13] How was it passing beforehand? [15:13] It was not, for some reason. I was on the wrong machine earlier. [15:13] allenap, we have two VMs running tests now, one for quantal and one for precise [15:14] matsubara: Ah ha! [15:14] allenap: my guess is that it was using an egg. Then the change to the buildout broke that. [15:14] buildout config* [15:15] allenap: yep, there is django_piston-0.2.3-py2.7.egg in the cache. [15:52] allenap, ping. [15:53] smoser: Hi. [15:53] a while ago, when trying to do ipmi setting/gathering.... [15:53] we came across the issue that the commissioning envirionment did not have access to credentials that could modify the node that it was running on [15:53] has this been resolved in any way? [15:56] smoser: Not that I'm aware of. roaksoax and I were just talking about similar things. We'll just provide an anonymous API method for now. There's no good way of getting credentials on a machine right now that's less likely to leak than a chocolate teapot. [15:57] allenap, sure there is [15:57] the provisioing network is assumed safe [15:58] if its not, you're hosed. [15:58] smoser: what I was suggesting was simply ensure that the commissioning node updating the power settings should send all the MAC addresses of its NIC's and verify that are the same with the ones stored in MAAS during enlistment [15:58] simply to make sure that the node what's updating itself [15:59] smoser: Which provisioning network? There is no separate provisioning network. [16:01] there is a provisioing network. [16:01] its the one that is utilized for pxeboot and such. [16:01] and installation. [16:01] that may happen to be another network also. [16:01] but we are assuming it is secure. [16:01] otherwise, we're hosed already. [16:02] i thikn you should just assume that if you passed credentials via commissioning user-data, that they are safe. [16:03] afaik, it has been accepted that http is good enough for that. [16:03] smoser: It was made explicit in the IoM that there is no separate provisioning network at all. [16:04] allenap, i'm not suggesting that it is separate. i'm suggesting that if you have 1 network, and you're doing pxeboot and http and dhcp on that network, then it has to be secure. [16:04] and if that network is not secured some way, then you cannot solve your problem. [16:04] * roaksoax brb [16:05] smoser: What do you mean by secure in that sense? [16:05] non-hostile i guess. [16:06] smoser: Okay, but there's nothing in maas that helps ensure that. [16:06] as if a deployed node has a hostile agent inside it, and that node can listen to traffic due for other nodes, you're kind of screwed. [16:06] allenap, you're right. [16:06] anwyay... [16:06] smoser: We have to assume that everyone on the network is nice. But, if we assume that, we don't need to pass credentials to the node. [16:06] we're using http for metadata [16:06] and we accepted that [16:07] because we accepted that the network was secure [16:07] hm.. [16:07] just pass credentials. [16:07] assume they're not intercepted. [16:08] then you only need to ensure that the transmission is safe. [16:08] not that everyone is nice. [16:08] (niether is ensurable at this point, but ensuring safe transmission is easier than ensuring that all people are good) [16:08] smoser: How can we do that? Can we do something on the switch/router to prevent eavesdropping? [16:08] allenap, we cannot, really. [16:09] but high end switches would be able to do that. [16:09] i really dont have a lot of experience here. but i've been told that basically people virtually plug a provisioning network in (via switch apis), then provision a node, then un-plug the provisioning netowrk. [16:10] smoser: Okay, we could pass a token on the kernel command line. Does that sound about right? [16:10] so that when the node is deployed it is not "physically" connected to the provisioning network. [16:10] allenap, dont pass it on the command line. [16:10] pass it in the user-data [16:10] for the commssioning. [16:11] the user-data is general purpose, and could easily be made to be done over https [16:11] smoser: That sounds fine. It can even be encoded in the API URL that might already be in there. http://token@... [16:11] well yo'ure already passing to the commissioning environment a set of metadata tokens [16:11] those tokens then get the user-data [16:12] and the user-data says "here are some maas api tokesn" [16:12] does that make sense? [16:12] its a little less direct, but it uses infrastructure we already have in place. [16:12] and is intended to be general purpose. [16:13] smoser: Yeah, makes sense, sounds good. I don't know enough to see all the plumbing, but I guess you do. [16:18] matsubara: Can you review https://code.launchpad.net/~allenap/maas/offline-more/+merge/126045 please? [16:24] allenap, so we won't have the make build-offline target anymore? [16:25] allenap, you change looks good. I just need to update the target to run 'make offline=true build' then? [16:25] matsubara: Nope, instead you can `export offline=true` or call `make offline=true {build,...}` [16:25] cool [16:26] matsubara: I personally prefer the latter; it's more explicit when reading the log. [16:26] ok [16:26] allenap, are you going to backport that to the precise branch? [16:26] matsubara: Thanks for the review. Erm, yes, I guess I will. [16:27] let me know when it's done so I can update the jenkins job [16:29] matsubara: Done. [16:32] allenap, ok, job updated, a build kicked [16:32] allenap, hmm it still tried to contact pypi [16:34] matsubara: !*^£ buildout's days are numbered. Small numbers. [16:35] matsubara: I'm off now. Speak to you tomorrow, have a good evening! [16:35] allenap, you too! [17:37] allenap: bug #1055662 [17:37] Launchpad bug 1055662 in MAAS "Anonymous API access to update power_type, power_settings" [Critical,Confirmed] https://launchpad.net/bugs/1055662 [17:37] .wub 18 [17:55] hmmm https://launchpad.net/~maas-maintainers [17:55] how can i get an archive created there. [17:55] maybe i need to be an admin? [17:56] or maybe rvba or allenap or matsubara could do i tfo rme ? [17:56] do it for me [17:56] or let me be an admin i suppose. [17:57] allenap, ^ [17:57] smoser, I can create a new one, what's the name of the ppa? [17:57] maas-ephemeral-images [17:58] smoser, https://launchpad.net/~maas-maintainers/+archive/maas-ephemeral-images [17:58] thanks [17:58] np [18:00] smoser: what did you change to get SOL of the ephemeral images to tty0? [18:02] roaksoax: wget http://ttys0.daviey.com/ > /etc/init/ttyS0.conf [18:02] ugh, that upstart job can be cleaned up now. [18:04] roaksoax, provisioningserver/kernel_opts.py [18:04] look for console= there [18:05] roaksoax, https://bugs.launchpad.net/maas/+bug/1044503 [18:05] Ubuntu bug 1044503 in MAAS "kernel command line is not easily customizable" [High,Triaged] [18:05] smoser: we should probably put a getty on ttyS0 and ttyS1 in ephemeral [18:13] i think thats a generic (very generic) ubuntu bug [18:16] andresrodriguieza [18:16] err [18:16] smoser: thanks [18:20] Daviey: so you were doing the wget http://ttys0.daviey.com/ > /etc/init/ttyS0.conf to get ttyS0 output? [19:03] roaksoax_: that is how you create a tty on a serial device [19:04] well -O rather than > i guess [19:04] Daviey: heh ok. anyways, SOL is working again === roaksoax_ is now known as roaksoax [19:05] Daviey: just had to do this: http://paste.ubuntu.com/1225168/ [19:09] roaksoax: that doesn't give you shell access, tho does it? [19:10] (or grub access) [19:11] [SOL session stolen] :) [19:12] Daviey: Daviey hehe, waiting for it to finish installing [19:12] i'll see the rest after it finished === matsubara__ is now known as matsubara === wgrant_ is now known as wgrant === hazmat` is now known as 5JTAACNC2 === 5JTAACNC2 is now known as hazmat === dannf` is now known as dannf