[05:03] bigjools: are juju/openstack questions best answered here or on a different channel? [05:05] burnbrighter: #juju [05:05] or #juju-dev [05:05] jtv: http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html [05:10] bigjools: https://github.com/KayEss/django-async [05:15] jtv: https://github.com/chrisdoble/django-celery-transactions [05:48] bigjools: what I see is subsequent nodes to the first admin node created by juju don't seem to bootsrap and pull over things like the ssh keys and such. [05:48] do you have any ideas? [05:49] burnbrighter: we're just otp, hang on [05:49] sure [05:52] burnbrighter: we should be almost done… but the standard thing to look out for is clock skew between the node and the server. That breaks things. [05:54] yes. I've been paying very close attention to that. Make note that the nodes have already successfully commissioned via maas, but its the juju side that's having problems [05:55] I can only ssh with key successfully to the first node bootstrapped by juju [05:57] burnbrighter: thats expected, you need 'juju ssh $unit' for other nodes [05:57] burnbrighter: or have you been trying that ? [05:57] that doesn't work either - yes "juju ssh 1" [05:58] I discovered this because my agents don't show up as running - then I started digging [05:58] only the agent on the very first node [05:58] in my case, it was my maas07 node [06:01] all of the nodes however show status "Allocated to Admin" [06:02] At that point you should be able to ssh in, with an ssh key registered for the Admin user. [06:02] "should" is the operative word :) [06:02] (Of course the ssh key must have been registered in maas before allocating the node, but I'm guessing it was) [06:03] yes indeedy [06:03] at what point and how does the ssh key get to each node? [06:03] is there a second bootstrap that happens from juju? [06:04] my assumption is yes [06:04] I don't think so… juju hands maas some metadata that maas then makes available to the booting node. But the ssh keys are sent over by maas, I forget how. [06:04] juju provides the key you configure in your environment to cloud-init. [06:04] jtv: ^ [06:05] only for bootstrapping [06:05] its like these nodes aren't picking up this metadata [06:05] I thought it passed the key via its own means on the units [06:05] but I dunno [06:06] I know it sounds like a classic case of oauth (or whatever its called) and utc time clock skew, but its not [06:07] I also configured a password in to maas.preseed in hopes of accessing the node to do further digging, but that isn't working either [06:11] Does the ssh-key need to be in place prior to maas-commission, or only before juju bootstrap? [06:12] my assumption is the latter [06:14] the latter [06:15] Do I need to destroy my environment then and start over? Is that the best way to move forward? [06:15] ie juju destroy, not maas re-do [06:28] I don't think so [06:28] but I'd ask the juju people as it doesn't seem like a maas problem to me [06:29] this is one of those weird issues that seems to sit in the in between ground somewhere between maas and juju. [06:30] it's possibly a bug in the maas provider code in juju, who knows [06:31] any suggestions on who would be the expert in this area? [06:35] *cough* :) [06:35] :) [06:35] I'd ask in the juju channel first and they will offer broader pointers. Although I wrote the provider, it was slightly blind. [06:35] I dont know juju very well and I was the only one available to do it :) [06:36] were you able to get this to successfully work in the lab? [06:42] yes [06:43] and locally in my virtual machines [06:43] that's what I'm using is VMs [06:44] I'd start with the juju people, they'll ask better questions than me [06:44] I've got the question posted [06:44] cool [06:45] are most of those people on your side of the globe? [06:45] most are in the Americas [06:45] IIRC [06:46] ok, cool [06:46] cause I'm tired [06:46] :) [06:46] going to bed... [06:46] yeah late for you [06:46] near midnight [06:47] cheez0r and I have been bouncing ideas off each other [06:47] we actually work for the same company we found out [06:47] :) [06:48] haha [06:48] which company? [06:49] ah, one of those big american telcos [06:49] think of the biggest gsm carrier in the US and you'll be right === matsubara-afk is now known as matsubara [07:32] Ok, think I figured it out. I had to before I went to bed. You need to start with all nodes off (that is key), otherwise the nodes can't bootstrap [07:33] I've been doing them one at a time, and it appears the key is being installed along with the new image [07:33] not necessarily in that order - I do see cloud-init install the key at the end === matsubara is now known as matsubara-lunch [11:21] How does MAAS currently determine what releases are available? Is it using distro-info, or is this hardcoded in some way? === matsubara-lunch is now known as matsubara [11:27] rvba: [11:27] rvba: howdy! [11:44] jtv: http://pastebin.ubuntu.com/1062430/ [11:44] jtv: maas-import-pxe-files failing ^^ [11:44] Thanks. [11:44] jtv: this is from a new package i'm preparing [11:45] Any idea where those templates _do_ get installed? [11:45] I guess the path only works from the source tree. [11:45] jtv: yeah they do get installed, see at the end of the pastebin [11:46] jtv: I had to patch MANIFEST.in though [11:46] So it ended up in /usr/share/pyshared instead of /usr/share/maas [11:46] Ah no [11:46] it's more wrong than that :( [11:47] jtv: if I add the file to MANIFEST.in it is going to get installed with the python modules under /usr/share/pyshared/maasprovisioning/etc/etc/ [11:48] Doesn't seem reasonable to expect it under “src,” does it? [11:48] jtv: if I don't add it there, then it won't get installed at all [11:48] jtv: and yes it doesn't [11:48] Is there a standard way to resolve this in python packages? [11:49] jtv: you mean where it is being installed? [11:49] that particular file? [11:49] No, I assume that's standardized. [11:49] I mean, generating paths to artifacts that will work both in a source tree and on an installed version of the program. [11:49] jtv: well, isn't that what PYTHONPATH is for? [11:50] I'm not thinking of python modules, but files such as the one that we're failing to find here. [11:50] “Other” files that need to be installed, somewhere under /usr/share [11:50] jtv: right, but I mean, the file is being installed in $PYTHONPATH/maasprovisioning/etc/etc [11:51] so you should be able to use it from there [11:51] I thought PYTHONPATH could have multiple entries. [11:51] jtv: yes, it is a list of path's [11:52] jtv: in this case we install python modules in /usr/share/pyshared which is part of PYTHONPATH [11:52] Okay, but is there a standard pattern for writing paths that will pick up these files both when running in the source tree and when running from an installed package? [11:53] I'm guessing, not using quite so much “..” [11:53] jtv: I don't know TBH [11:54] jtv: i guess you'd have to evaluate whether the file is in src for running from trunk, if not, search it in pythonpath [11:54] And of course the one guy who knows that particular piece of code best just left and won't be back for a while. :) [11:54] lol [11:54] Searching for it… what a pain. [11:55] jtv: i guess using something like this: os.path.join(os.path.dirname(__file__), "templates"), [11:56] Yeah — without the “..” would be easier. [11:57] self.path = os.path.join(basedir, power_type + ".template") [11:57] jtv: so you can do it similarlay to how it is being done with the power templates ig uess [11:57] power/poweraction.py: self.path = os.path.join(basedir, power_type + ".template") [12:00] Okay, several distracting phone conversations later… [12:00] Let's just have a look [12:01] Right after I deal with a difficult toddler [12:02] Ow argh [12:03] That's not the toddler. It's the code. [12:03] The path to those templates is configurable, which is good. [12:03] But it's set in etc/celeryconfig.py [12:03] —which means it's hard to get back to src/provisioningserver/pxe/ (where the path is actually needed) without the ..! [12:04] I think there's an easier way. [12:05] Have a built-in default: os.path.join( os.path.dirname(__file__), 'templates' ) [12:05] Give etc/celeryconfig.py the ability to override that [12:05] But have it default to None, which means use the built-in default. [12:07] roaksoax: I don't suppose we could just stall src/provisioningserver/pxe/pxeconfig.py and src/provisioningserver/pxe/templates/ into the same location? [12:08] jtv: but isn't it easier to join path of pxeconfig.py + templates [12:08] ? [12:08] That won't work if the two ain't in the same location, will it? [12:08] By “stall” I meant “install.” [12:08] Typo. [12:11] So: can we install things such that os.path.join( os.path.dirname(__file__), 'templates' ) will actually work? [12:11] And yes, that is the same thing as joining the path of pxeconfig.py + templates. [12:11] * roaksoax looks [12:18] jtv: ok so it seems that there's a few things to improve [12:19] jtv: 1. celeryconfig.py should be taken out of "etc/" and should be placed somewhere in the PYTHONPATH as this file is not meant to be modified. [12:19] Isn't it? I have no idea. [12:19] jtv: check at the source, it is being shipped in etc/celeryconfig.py [12:19] jtv: which makes me understand that this is meant to be installed into /etc/maas/ [12:20] however, this should be treated as how we are treating settings.py and maas_local_settings.py [12:22] Probably, yes. [12:22] How does the packaging treat those? [12:25] jtv: src/maas/* is installed in /usr/share/maas/maas/* which contains the settings.py. Then maas_local_settings.py is installed in /etc/maas and symlinked into /usr/share/maas. So, since celeryconfig.py is meant to not be modified, should be installed in a location like /usr/share/maas/maas or /usr/share/pyshared/provisioningserver/ (and need to change the location in of it in trunk to not be under the 'etc/' dir), and then we ship a user_maasceleryconfi [12:26] Ah, so /usr/share/* can symlink to files in /etc? [12:27] jtv: we are symlinking from /etc/maas/ to /usr/share/maas, not from /usr/share to /etc [12:27] So the actual file is in /usr/share/maas? [12:27] jtv: the actual file is in /etc/maas/maas_local_settings.py [12:28] jtv: and it gets symlinked in /usr/share/maas/ [12:28] roaksoax: hey. [12:28] rvba: o/ good timing! :) [12:29] rvba: can you explain jtv and me how can we effectively resolve http://pastebin.ubuntu.com/1062430/ [12:29] * rvba looks. [12:29] since you've dealt with issues like these [12:29] Now, as I was asking, can src/provisioningserver/pxe/pxeconfig.py and src/provisioningserver/pxe/templates/ be installed to the same location? Or at least can we have a symlink there? [12:31] OMG [12:32] Yeah. It's actually quite easy to simplify though. [12:32] I don't think relative paths should be defined in etc/celeryconfig.py [12:32] jtv: if you ship those in the same location in trunk, I can ship those in the same location in packaging. Otherwise, symlinking is adding to much risk [12:32] They should be defined inside the modules themselves and relative to, say, src/provisioningserver [12:32] Either is fine by me. [12:32] rvba: yes, that's what I outlined above. [12:33] jtv: Then I agree :) [12:33] Nice. :) [12:33] In fact we may not need a setting at all. [12:33] I also don't see why it's a setting. [12:34] Looks like module-level constant to me. [12:34] I'm guessing it may become useful for node-group workers, _if_ we go with distributed TFTP. [12:34] It's not used anywhere else, so yeah, we can shortcut the whole thing now. [12:34] Well, each and every worker will ship its own copy of the templates. [12:35] At that point I guess the maasserver won't need them at all anyway. [12:36] As a rule of thumb, nothing in the code/config should reference 'src'. [12:36] The way we're planning to do things now, for now only the maasserver needs them. And then later, only the celery worker on the maas server will need them. [12:36] Duh. :) [12:36] :) [12:36] rvba: btw.. we have a segfaulting apache now :) [12:36] roaksoax: uh oh [12:36] Congratulations! [12:37] rvba: I have yet to finish the celeryd stuff, but as of now, for some reason, it segfaults [12:37] roaksoax: maybe that's related to the changes to the wsgi file… but I was able to test that in a prod environment so I'm surprised. [12:38] roaksoax: do you have a half-finisehd package I could play with? [12:38] rvba: yeah I tested those, and removed them, and it shows a different error [12:38] rvba: i had to add this though: WSGIImportScript /usr/share/maas/wsgi.py process-group=maas application-group=maas [12:38] the application-group part [12:38] Ok. [12:38] I think I need to read the wsgi doc again. [12:38] rvba: I need to do one more thing. Since I lost my notes, what was needed to start celeryd? [12:39] roaksoax: nothing really, just make sure celeryconfig.py in on the path and also the provisioning server module. [12:39] rvba: right, but what was the command? [12:39] roaksoax: celeryd [12:39] just that, no args? [12:40] roaksoax: "celeryd -l INFO" with the required modules on the PYTHONPATH. [12:40] And that's it. [12:41] roaksoax: oh, one more thing: "-f logfile" [12:41] [Wed Jun 27 08:40:35 2012] [error] [client 127.0.0.1] Premature end of script headers: wsgi.py [12:42] [Wed Jun 27 08:40:35 2012] [notice] child pid 31318 exit signal Segmentation fault (11) [12:42] rvba: (i removed the above line you added to maas-http.conf [12:42] rvba: ok so let me add the upstart job and you will have a package [12:42] roaksoax: ok, I'll then give you a hand with that segfault. [12:45] rvba: exec /usr/bin/celeryd --pidfile=/run/maas-celeryd.pid --logfile=/var/log/maas/celeryd.log --loglevel=INFO [12:46] that should be enough then [12:49] rvba: there's no --app instance of celery to use right? [12:49] roaksoax: "--app"? [12:50] rvba: --app=APP Name of the app instance to use. [12:50] celeryd --help [12:51] roaksoax: no [13:10] Daviey: I'd like to talk to you briefly about DNS in MAAS. Would you be available sometime later today or tomorrow? [13:14] rvba: 2hrs from now? [13:16] Daviey: that would be prefect, please ping me when you'll be available. [13:17] roaksoax: rvba asked me if I could let you cut your teeth on this review… https://code.launchpad.net/~jtv/maas/import-ephemerals-tftp/+merge/112353 [13:18] frankban: A review for you ^ [13:18] roaksoax: I think jtv meant frankban. [13:18] Oh, frankban. [13:18] Got my names confused I guess. [13:18] Guess so :) [13:18] jtv :-) ok will take a look. [13:18] Thanks. [13:23] rvba: uhmmm [13:23] rvba: when I try to run celeryd as user/group 'maas' [13:23] /usr/lib/python2.7/dist-packages/celery/loaders/default.py:45: NotConfigured: No 'celeryconfig' module found! Please make sure it exists and is available to Python. "is available to Python." % (configname, ))) [13:29] roaksoax: as you probably have guessed, this means that celeryconfig.py is not on the path. [13:29] rvba: figured it out :) [13:35] rvba: ok, got a package for you. Are you running quantal by any chance? [13:35] roaksoax: I'm not but I'll spin up a canonistack instance. [13:35] rvba: ok, I'll upload to the testing PPA then [13:36] rvba: don't upgrade though otherwise it might be broken :) [13:36] k [14:06] roaksoax: I see the package has been uploaded… but 'apt-get install maas' wants to install the old version 0.1+bzr482+dfsg-0ubuntu1). Any idea why that might be? [14:08] rvba: it is still waiting to be published [14:08] roaksoax: ah… also, "apt-get install raphael" says "Unable to locate package raphael" [14:09] rvba: libjs-raphael [14:09] rarg, silly me. [14:11] :) [14:13] rvba: maas should be installable now, (sudo apt-get update && sudo apt-get install maas) [14:14] roaksoax: cool, I'm installing the last version right now. [14:16] awesome [14:16] roaksoax: I get that error when installing the package http://paste.ubuntu.com/1062592/ [14:17] let me test [14:17] This happens in one of the migration scripts. I'll investigate but I wanted to make sure you were seeing that error too. [14:17] i wwasn't actually [14:23] roaksoax: looks like it's an incompatibility in our code with south version 0.7.5 [14:23] rvba: ah, so maybe just download python-django-maas and maas deb packages and install them in precise [14:23] Also, I'm surprised to see that Django is still version 1.3 in quantal. [14:24] Django 1.4 has been released in March 2012. [14:25] roaksoax: I'll try that. But I'll fix the code too. [14:25] rvba: it's probably because it hasn't yet been merged into ubuntu from debian [14:27] rvba: python-django is 1.4 [14:27] rvba: https://launchpad.net/ubuntu/+source/python-django/1.4-1 [14:28] roaksoax: arg, you're right, wrong terminal, sorry ;) [14:28] hehe [14:31] Yigh. What am I doing wrong!? My test does: self.patch(celeryconfig, 'PXE_TEMPLATES_DIR', path) but it never seems to affect celeryconfig.PXE_TEMPLATES_DIR. [14:31] O_o [14:32] jtv: care to paste the whole test? [14:32] I'll give you the relevant bits. [14:34] rvba: example of failing test: http://paste.ubuntu.com/1062616/ [14:34] Excerpt of the affected code follows. [14:35] Code being tested: http://paste.ubuntu.com/1062618/ [14:35] In etc/celeryconfig.py, I changed the default value for PXE_TEMPLATES_DIR to None. [14:35] jtv: when exactly is template_basedir computed? Maybe before patch(...) is called. [14:36] It's not cached AFAICS [14:36] How is PXE_TEMPLATES_DIR defined? [14:36] If it's done at module level then that happens before anything is patched. [14:37] It's initialized to None at the module level, yes. I thought you could patch at the module level. [14:37] I mean, sure it gets defined, but the point of the patch is to override its value, right? [14:38] Right. [14:38] So PXE_TEMPLATES_DIR should already be imported and initialized at this point, and AFAICS the patch should override its value. [14:39] Maybe import_settings messes with it somehow? I can't really imagine it. [14:41] Maybe the patch() does not impact the value imported and used by PXEConfig. [14:42] jtv: cuold you simply try to use celeryconfig.PXE_TEMPLATES_DIR in def template_basedir ? [14:42] Sure, I'll try that [14:44] Nope. That's still always None, even when I patch it. [14:51] roaksoax: it's a struggle to install the mew maas package on precise. [14:52] roaksoax: I'll fix the incompatibility instead. [14:52] rvba: sudo dpkg -i python-django-maas*.deb maas*.deb [14:52] rvba: and then: sudo apt-get -f install [14:52] and that should be all you need [14:53] rvba: no new insights on why my patch isn't working. :( I'm pretty sure it's supposed to work for modules. [14:54] roaksoax: temporary glitch in the archives? http://paste.ubuntu.com/1062648/ [14:54] jtv: pretty sure too… really a weird problem. [14:54] rvba: sudo apt-get update && sudo apt-get -f install [14:55] rvba: I had something similar with django.config, but at least there we knew that weird stuff was going on. [14:55] roaksoax: thanks. [14:55] Maybe I should leave it for ye morrow. [14:56] * rvba checks the world clock. [14:56] Indeed :) [14:56] Yawn. nn folks! [14:56] \o jtv. [16:01] Daviey: roaksoax there is actually a bug in piston that prevents it from working with Django 1.4. That bug seems fixed in piston upsteam https://bitbucket.org/jespern/django-piston/changeset/7c90898072ce#chg-piston/resource.py [16:01] But it's not yet packaged. [16:02] I'll test if that fix solves the problem I'm having with MAAS and Django 1.4. [16:07] rvba: if you want to validate that, i'll get it uploaded [16:09] rvba: I should also have a new yui3 package soonish [16:09] Daviey: test done, I have other problems with MAAS and Django 1.4 that I need to investigate but all the piston related problems are gone with the trunk version. Also, from what I understand of the problem, the fix in piston completely makes sense. [16:10] Our 173 API tests pass with piston trunk version. [16:11] rvba: to confirm, you used quantal piston and that patch applied? [16:12] Daviey: no, I changed the Django version in versions.cfg to 1.4 and then I used piston trunk version. [16:13] rvba: ah, i hoped you validated just that changeset :) [16:13] Daviey: I can do that. [16:21] rvba: btw were you able to find out why the errors with wsgi? [16:23] roaksoax: no, I was stopped by this Django 1.4 incompatibility problem. [16:24] Daviey: ok, so we need these changesets: https://bitbucket.org/jespern/django-piston/changeset/3a0d021dd042#files https://bitbucket.org/jespern/django-piston/changeset/7c90898072ce [16:24] Daviey: with these two patches all the API test pass with Django 1.4. [16:24] tests* [16:26] roaksoax: I need to go now but tomorrow I'll fix the problem which prevent the package from being installed. Then with the new package we will be able to investigate that wsgi problem. How does that sound? [16:26] prevents* [16:28] rvba: sounds good to me [16:28] I'll have a new yui3 package on PPA too which MAAS will be depending on [16:29] roaksoax: Do you want to cherry pick those two commits that rvba outlined? [16:29] Best to upload to -proposed as it's seeded. [16:30] Daviey: sure, I'll cherry pick those and upload python-piston to -proposed [16:30] ta [16:31] roaksoax: ta [16:57] Daviey: btw.. is the installer already with squashfs? [16:58] roaksoax: no [16:58] roaksoax: I hit a blocker trying to do it... needs some cjwatson love. === matsubara is now known as matsubara-afk [16:58] roaksoax: i do have a squashfs if you wanted to play with it? [16:59] but the d-i part seemed to wedge, when the squash is co-located on cd [16:59] (cd/usb) [16:59] Daviey: sure, point me to your squashfs image and I'll play with it [17:03] Daviey: python-django-piston has just been uploaded [17:06] roaksoax: thanks [17:07] roaksoax: I'll create a new squashfs tomorrow, and send it along [17:07] Daviey: awesome!