/srv/irclogs.ubuntu.com/2012/06/27/#maas.txt

burnbrighterbigjools: are juju/openstack questions best answered here or on a different channel?05:03
bigjoolsburnbrighter: #juju05:05
bigjoolsor #juju-dev05:05
bigjoolsjtv: http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html05:05
jtvbigjools: https://github.com/KayEss/django-async05:10
bigjoolsjtv: https://github.com/chrisdoble/django-celery-transactions05:15
burnbrighterbigjools: what I see is subsequent nodes to the first admin node created by juju don't seem to bootsrap and pull over things like the ssh keys and such.05:48
burnbrighterdo you have any ideas?05:48
jtvburnbrighter: we're just otp, hang on05:49
burnbrightersure05:49
jtvburnbrighter: we should be almost done… but the standard thing to look out for is clock skew between the node and the server.  That breaks things.05:52
burnbrighteryes.  I've been paying very close attention to that.  Make note that the nodes have already successfully commissioned via maas, but its the juju side that's having problems05:54
burnbrighterI can only ssh with key successfully to the first node bootstrapped by juju05:55
lifelessburnbrighter: thats expected, you need 'juju ssh $unit' for other nodes05:57
lifelessburnbrighter: or have you been trying that ?05:57
burnbrighterthat doesn't work either - yes  "juju ssh 1"05:57
burnbrighterI discovered this because my agents don't show up as running - then I started digging05:58
burnbrighteronly the agent on the very first node05:58
burnbrighterin my case, it was my maas07 node05:58
burnbrighterall of the nodes however show status "Allocated to Admin"06:01
jtvAt that point you should be able to ssh in, with an ssh key registered for the Admin user.06:02
burnbrighter"should" is the operative word :)06:02
jtv(Of course the ssh key must have been registered in maas before allocating the node, but I'm guessing it was)06:02
burnbrighteryes indeedy06:03
burnbrighterat what point and how does the ssh key get to each node?06:03
burnbrighteris there a second bootstrap that happens from juju?06:03
burnbrightermy assumption is yes06:04
jtvI don't think so…  juju hands maas some metadata that maas then makes available to the booting node.  But the ssh keys are sent over by maas, I forget how.06:04
lifelessjuju provides the key you configure in your environment to cloud-init.06:04
lifelessjtv: ^06:04
bigjoolsonly for bootstrapping06:05
burnbrighterits like these nodes aren't picking up this metadata06:05
bigjoolsI thought it passed the key via its own means on the units06:05
bigjoolsbut I dunno06:05
burnbrighterI know it sounds like a classic case of oauth (or whatever its called) and utc time clock skew, but its not06:06
burnbrighterI also configured a password in to maas.preseed in hopes of accessing the node to do further digging, but that isn't working either06:07
burnbrighterDoes the ssh-key need to be in place prior to maas-commission, or only before juju bootstrap?06:11
burnbrightermy assumption is the latter06:12
bigjoolsthe latter06:14
burnbrighterDo I need to destroy my environment then and start over? Is that the best way to move forward?06:15
burnbrighterie juju destroy, not maas re-do06:15
bigjoolsI don't think so06:28
bigjoolsbut I'd ask the juju people as it doesn't seem like a maas problem to me06:28
burnbrighterthis is one of those weird issues that seems to sit in the in between ground somewhere between maas and juju.06:29
bigjoolsit's possibly a bug in the maas provider code in juju, who knows06:30
burnbrighterany suggestions on who would be the expert in this area?06:31
bigjools*cough* :)06:35
burnbrighter:)06:35
bigjoolsI'd ask in the juju channel first and they will offer broader pointers.  Although I wrote the provider, it was slightly blind.06:35
bigjoolsI dont know juju very well and I was the only one available to do it :)06:35
burnbrighterwere you able to get this to successfully work in the lab?06:36
bigjoolsyes06:42
bigjoolsand locally in my virtual machines06:43
burnbrighterthat's what I'm using is VMs06:43
bigjoolsI'd start with the juju people, they'll ask better questions than me06:44
burnbrighterI've got the question posted06:44
bigjoolscool06:44
burnbrighterare most of those people on your side of the globe?06:45
bigjoolsmost are in the Americas06:45
bigjoolsIIRC06:45
burnbrighterok, cool06:46
burnbrightercause I'm tired06:46
burnbrighter:)06:46
burnbrightergoing to bed...06:46
bigjoolsyeah late for you06:46
burnbrighternear midnight06:46
burnbrightercheez0r and I have been bouncing ideas off each other06:47
burnbrighterwe actually work for the same company we found out06:47
burnbrighter:)06:47
bigjoolshaha06:48
bigjoolswhich company?06:48
burnbrighterah, one of those big american telcos06:49
burnbrighterthink of the biggest gsm carrier in the US and you'll be right06:49
=== matsubara-afk is now known as matsubara
burnbrighterOk, think I figured it out.  I had to before I went to bed.  You need to start with all nodes off (that is key), otherwise the nodes can't bootstrap07:32
burnbrighterI've been doing them one at a time, and it appears the key is being installed along with the new image07:33
burnbrighternot necessarily in that order - I do see cloud-init install the key at the end07:33
=== matsubara is now known as matsubara-lunch
rbasakHow does MAAS currently determine what releases are available? Is it using distro-info, or is this hardcoded in some way?11:21
=== matsubara-lunch is now known as matsubara
roaksoaxrvba:11:27
roaksoaxrvba: howdy!11:27
roaksoaxjtv: http://pastebin.ubuntu.com/1062430/11:44
roaksoaxjtv: maas-import-pxe-files failing ^^11:44
jtvThanks.11:44
roaksoaxjtv: this is from a new package i'm preparing11:44
jtvAny idea where those templates _do_ get installed?11:45
jtvI guess the path only works from the source tree.11:45
roaksoaxjtv: yeah they do get installed, see at the end of the pastebin11:45
roaksoaxjtv: I had to patch MANIFEST.in though11:46
jtvSo it ended up in /usr/share/pyshared instead of /usr/share/maas11:46
jtvAh no11:46
jtvit's more wrong than that :(11:46
roaksoaxjtv: if I add the file to MANIFEST.in it is going to get installed with the python modules under /usr/share/pyshared/maasprovisioning/etc/etc/11:47
jtvDoesn't seem reasonable to expect it under “src,” does it?11:48
roaksoaxjtv: if I don't add it there, then it won't get installed at all11:48
roaksoaxjtv: and yes it doesn't11:48
jtvIs there a standard way to resolve this in python packages?11:48
roaksoaxjtv: you mean where it is being installed?11:49
roaksoaxthat particular file?11:49
jtvNo, I assume that's standardized.11:49
jtvI mean, generating paths to artifacts that will work both in a source tree and on an installed version of the program.11:49
roaksoaxjtv: well, isn't that what PYTHONPATH is for?11:49
jtvI'm not thinking of python modules, but files such as the one that we're failing to find here.11:50
jtv“Other” files that need to be installed, somewhere under /usr/share11:50
roaksoaxjtv: right, but I mean, the file is being installed in $PYTHONPATH/maasprovisioning/etc/etc11:50
roaksoaxso you should be able to use it from there11:51
jtvI thought PYTHONPATH could have multiple entries.11:51
roaksoaxjtv: yes, it is a list of path's11:51
roaksoaxjtv: in this case we install python modules in /usr/share/pyshared which is part of PYTHONPATH11:52
jtvOkay, but is there a standard pattern for writing paths that will pick up these files both when running in the source tree and when running from an installed package?11:52
jtvI'm guessing, not using quite so much “..”11:53
roaksoaxjtv: I don't know TBH11:53
roaksoaxjtv: i guess you'd have to evaluate whether the file is in src for running from trunk, if not, search it in pythonpath11:54
jtvAnd of course the one guy who knows that particular piece of code best just left and won't be back for a while.  :)11:54
roaksoaxlol11:54
jtvSearching for it… what a pain.11:54
roaksoaxjtv: i guess using something like this:  os.path.join(os.path.dirname(__file__), "templates"),11:55
jtvYeah — without the “..” would be easier.11:56
roaksoaxself.path = os.path.join(basedir, power_type + ".template")11:57
roaksoaxjtv: so you can do it similarlay to how it is being done with the power templates ig uess11:57
roaksoaxpower/poweraction.py:        self.path = os.path.join(basedir, power_type + ".template")11:57
jtvOkay, several distracting phone conversations later…12:00
jtvLet's just have a look12:00
jtvRight after I deal with a difficult toddler12:01
jtvOw argh12:02
jtvThat's not the toddler.  It's the code.12:03
jtvThe path to those templates is configurable, which is good.12:03
jtvBut it's set in etc/celeryconfig.py12:03
jtv—which means it's hard to get back to src/provisioningserver/pxe/ (where the path is actually needed) without the ..!12:03
jtvI think there's an easier way.12:04
jtvHave a built-in default: os.path.join( os.path.dirname(__file__), 'templates' )12:05
jtvGive etc/celeryconfig.py the ability to override that12:05
jtvBut have it default to None, which means use the built-in default.12:05
jtvroaksoax: I don't suppose we could just stall src/provisioningserver/pxe/pxeconfig.py and src/provisioningserver/pxe/templates/ into the same location?12:07
roaksoaxjtv: but isn't it easier to join path of pxeconfig.py + templates12:08
roaksoax?12:08
jtvThat won't work if the two ain't in the same location, will it?12:08
jtvBy “stall” I meant “install.”12:08
jtvTypo.12:08
jtvSo: can we install things such that os.path.join( os.path.dirname(__file__), 'templates' ) will actually work?12:11
jtvAnd yes, that is the same thing as joining the path of pxeconfig.py + templates.12:11
* roaksoax looks12:11
roaksoaxjtv: ok so it seems that there's a few things to improve12:18
roaksoaxjtv: 1. celeryconfig.py should be taken out of "etc/" and should be placed somewhere in the PYTHONPATH as this file is not meant to be modified.12:19
jtvIsn't it?  I have no idea.12:19
roaksoaxjtv: check at the source, it is being shipped in etc/celeryconfig.py12:19
roaksoaxjtv: which makes me understand that this is meant to be installed into /etc/maas/12:19
roaksoaxhowever, this should be treated as how we are treating settings.py  and maas_local_settings.py12:20
jtvProbably, yes.12:22
jtvHow does the packaging treat those?12:22
roaksoaxjtv: src/maas/* is installed in /usr/share/maas/maas/* which contains the settings.py. Then maas_local_settings.py is installed in /etc/maas and symlinked into /usr/share/maas. So, since celeryconfig.py is meant to not be modified, should be installed in a location like /usr/share/maas/maas or /usr/share/pyshared/provisioningserver/ (and need to change the location in of it in trunk to not be under the 'etc/' dir), and then we ship a user_maasceleryconfi12:25
jtvAh, so /usr/share/* can symlink to files in /etc?12:26
roaksoaxjtv: we are symlinking from /etc/maas/ to /usr/share/maas, not from /usr/share to /etc12:27
jtvSo the actual file is in /usr/share/maas?12:27
roaksoaxjtv: the actual file is in /etc/maas/maas_local_settings.py12:27
roaksoaxjtv: and it gets symlinked in /usr/share/maas/12:28
rvbaroaksoax: hey.12:28
roaksoaxrvba: o/ good timing! :)12:28
roaksoaxrvba: can you explain jtv and me how can we effectively resolve http://pastebin.ubuntu.com/1062430/12:29
* rvba looks.12:29
roaksoaxsince you've dealt with issues like these12:29
jtvNow, as I was asking, can src/provisioningserver/pxe/pxeconfig.py and src/provisioningserver/pxe/templates/ be installed to the same location?  Or at least can we have a symlink there?12:29
rvbaOMG12:31
jtvYeah.  It's actually quite easy to simplify though.12:32
rvbaI don't think relative paths should be defined in etc/celeryconfig.py12:32
roaksoaxjtv: if you ship those in the same location in trunk, I can ship those in the same location in packaging. Otherwise, symlinking is adding to much risk12:32
rvbaThey should be defined inside the modules themselves and relative to, say, src/provisioningserver12:32
jtvEither is fine by me.12:32
jtvrvba: yes, that's what I outlined above.12:32
rvbajtv: Then I agree :)12:33
jtvNice.  :)12:33
jtvIn fact we may not need a setting at all.12:33
rvbaI also don't see why it's a setting.12:33
rvbaLooks like module-level constant to me.12:34
jtvI'm guessing it may become useful for node-group workers, _if_ we go with distributed TFTP.12:34
jtvIt's not used anywhere else, so yeah, we can shortcut the whole thing now.12:34
rvbaWell, each and every worker will ship its own copy of the templates.12:34
jtvAt that point I guess the maasserver won't need them at all anyway.12:35
rvbaAs a rule of thumb, nothing in the code/config should reference 'src'.12:36
jtvThe way we're planning to do things now, for now only the maasserver needs them.  And then later, only the celery worker on the maas server will need them.12:36
jtvDuh.  :)12:36
rvba:)12:36
roaksoaxrvba: btw.. we have a segfaulting apache now :)12:36
rvbaroaksoax: uh oh12:36
jtvCongratulations!12:36
roaksoaxrvba: I have yet to finish the celeryd stuff, but as of now, for some reason, it segfaults12:37
rvbaroaksoax: maybe that's related to the changes to the wsgi file… but I was able to test that in a prod environment so I'm surprised.12:37
rvbaroaksoax: do you have a half-finisehd package I could play with?12:38
roaksoaxrvba: yeah I tested those, and removed them, and it shows a different error12:38
roaksoaxrvba: i had to add this though: WSGIImportScript /usr/share/maas/wsgi.py process-group=maas application-group=maas12:38
roaksoaxthe application-group part12:38
rvbaOk.12:38
rvbaI think I need to read the wsgi doc again.12:38
roaksoaxrvba: I need to do one more thing. Since I lost my notes, what was needed to start celeryd?12:38
rvbaroaksoax: nothing really, just make sure celeryconfig.py in on the path and also the provisioning server module.12:39
roaksoaxrvba: right, but what was the command?12:39
rvbaroaksoax: celeryd12:39
roaksoaxjust that, no args?12:39
rvbaroaksoax: "celeryd -l INFO" with the required modules on the PYTHONPATH.12:40
rvbaAnd that's it.12:40
rvbaroaksoax: oh, one more thing: "-f logfile"12:41
roaksoax[Wed Jun 27 08:40:35 2012] [error] [client 127.0.0.1] Premature end of script headers: wsgi.py12:41
roaksoax[Wed Jun 27 08:40:35 2012] [notice] child pid 31318 exit signal Segmentation fault (11)12:42
roaksoaxrvba: (i removed the above line you added to maas-http.conf12:42
roaksoaxrvba: ok so let me add the upstart job and you will have a package12:42
rvbaroaksoax: ok, I'll then give you a hand with that segfault.12:42
roaksoaxrvba: exec /usr/bin/celeryd --pidfile=/run/maas-celeryd.pid --logfile=/var/log/maas/celeryd.log --loglevel=INFO12:45
roaksoaxthat should be enough then12:46
roaksoaxrvba: there's no --app instance of celery to use right?12:49
rvbaroaksoax: "--app"?12:49
roaksoaxrvba:   --app=APP             Name of the app instance to use.12:50
roaksoaxceleryd --help12:50
rvbaroaksoax: no12:51
rvbaDaviey: I'd like to talk to you briefly about DNS in MAAS.  Would you be available sometime later today or tomorrow?13:10
Davieyrvba: 2hrs from now?13:14
rvbaDaviey: that would be prefect, please ping me when you'll be available.13:16
jtvroaksoax: rvba asked me if I could let you cut your teeth on this review… https://code.launchpad.net/~jtv/maas/import-ephemerals-tftp/+merge/11235313:17
rvbafrankban: A review for you ^13:18
rvbaroaksoax: I think jtv meant frankban.13:18
jtvOh, frankban.13:18
jtvGot my names confused I guess.13:18
rvbaGuess so :)13:18
frankbanjtv :-) ok will take a look.13:18
jtvThanks.13:18
roaksoaxrvba: uhmmm13:23
roaksoaxrvba: when I try to run celeryd as user/group 'maas'13:23
roaksoax/usr/lib/python2.7/dist-packages/celery/loaders/default.py:45: NotConfigured: No 'celeryconfig' module found! Please make sure it exists and is available to Python. "is available to Python." % (configname, )))13:23
rvbaroaksoax: as you probably have guessed, this means that celeryconfig.py is not on the path.13:29
roaksoaxrvba: figured it out :)13:29
roaksoaxrvba: ok, got a package for you. Are you running quantal by any chance?13:35
rvbaroaksoax: I'm not but I'll spin up a canonistack instance.13:35
roaksoaxrvba: ok, I'll upload to the testing PPA then13:35
roaksoaxrvba: don't upgrade though otherwise it might be broken :)13:36
rvbak13:36
rvbaroaksoax: I see the package has been uploaded… but 'apt-get install maas' wants to install the old version 0.1+bzr482+dfsg-0ubuntu1).  Any idea why that might be?14:06
roaksoaxrvba: it is still waiting to be published14:08
rvbaroaksoax: ah… also, "apt-get install raphael" says "Unable to locate package raphael"14:08
roaksoaxrvba: libjs-raphael14:09
rvbararg, silly me.14:09
roaksoax:)14:11
roaksoaxrvba: maas should be installable now, (sudo apt-get update && sudo apt-get install maas)14:13
rvbaroaksoax: cool, I'm installing the last version right now.14:14
roaksoaxawesome14:16
rvbaroaksoax: I get that error when installing the package http://paste.ubuntu.com/1062592/14:16
roaksoaxlet me test14:17
rvbaThis happens in one of the migration scripts.  I'll investigate but I wanted to make sure you were seeing that error too.14:17
roaksoaxi wwasn't actually14:17
rvbaroaksoax: looks like it's an incompatibility in our code with south version 0.7.514:23
roaksoaxrvba: ah, so maybe just download python-django-maas and maas deb packages and install them in precise14:23
rvbaAlso, I'm surprised to see that Django is still version 1.3 in quantal.14:23
rvbaDjango 1.4 has been released in March 2012.14:24
rvbaroaksoax: I'll try that.  But I'll fix the code too.14:25
roaksoaxrvba: it's probably because it hasn't yet been merged into ubuntu from debian14:25
roaksoaxrvba: python-django is 1.414:27
roaksoaxrvba: https://launchpad.net/ubuntu/+source/python-django/1.4-114:27
rvbaroaksoax: arg, you're right, wrong terminal, sorry ;)14:28
roaksoaxhehe14:28
jtvYigh.  What am I doing wrong!?  My test does: self.patch(celeryconfig, 'PXE_TEMPLATES_DIR', path) but it never seems to affect celeryconfig.PXE_TEMPLATES_DIR.14:31
rvbaO_o14:31
rvbajtv: care to paste the whole test?14:32
jtvI'll give you the relevant bits.14:32
jtvrvba: example of failing test: http://paste.ubuntu.com/1062616/14:34
jtvExcerpt of the affected code follows.14:34
jtvCode being tested: http://paste.ubuntu.com/1062618/14:35
jtvIn etc/celeryconfig.py, I changed the default value for PXE_TEMPLATES_DIR to None.14:35
rvbajtv: when exactly is template_basedir computed?  Maybe before patch(...) is called.14:35
jtvIt's not cached AFAICS14:36
rvbaHow is PXE_TEMPLATES_DIR defined?14:36
rvbaIf it's done at module level then that happens before anything is patched.14:36
jtvIt's initialized to None at the module level, yes.  I thought you could patch at the module level.14:37
jtvI mean, sure it gets defined, but the point of the patch is to override its value, right?14:37
rvbaRight.14:38
jtvSo PXE_TEMPLATES_DIR should already be imported and initialized at this point, and AFAICS the patch should override its value.14:38
jtvMaybe import_settings messes with it somehow?  I can't really imagine it.14:39
rvbaMaybe the patch() does not impact the value imported and used by PXEConfig.14:41
rvbajtv: cuold you simply try to use celeryconfig.PXE_TEMPLATES_DIR in def template_basedir ?14:42
jtvSure, I'll try that14:42
jtvNope.  That's still always None, even when I patch it.14:44
rvbaroaksoax: it's a struggle to install the mew maas package on precise.14:51
rvbaroaksoax: I'll fix the incompatibility instead.14:52
roaksoaxrvba: sudo dpkg -i python-django-maas*.deb maas*.deb14:52
roaksoaxrvba: and then: sudo apt-get -f install14:52
roaksoaxand that should be all you need14:52
jtvrvba: no new insights on why my patch isn't working.  :(  I'm pretty sure it's supposed to work for modules.14:53
rvbaroaksoax: temporary glitch in the archives? http://paste.ubuntu.com/1062648/14:54
rvbajtv: pretty sure too… really a weird problem.14:54
roaksoaxrvba: sudo apt-get update && sudo apt-get -f install14:54
jtvrvba: I had something similar with django.config, but at least there we knew that weird stuff was going on.14:55
rvbaroaksoax: thanks.14:55
jtvMaybe I should leave it for ye morrow.14:55
* rvba checks the world clock.14:56
rvbaIndeed :)14:56
jtvYawn.  nn folks!14:56
rvba\o jtv.14:56
rvbaDaviey: roaksoax there is actually a bug in piston that prevents it from working with Django 1.4.  That bug seems fixed in piston upsteam https://bitbucket.org/jespern/django-piston/changeset/7c90898072ce#chg-piston/resource.py16:01
rvbaBut it's not yet packaged.16:01
rvbaI'll test if that fix solves the problem I'm having with MAAS and Django 1.4.16:02
Davieyrvba: if you want to validate that, i'll get it uploaded16:07
roaksoaxrvba: I should also have a new yui3 package soonish16:09
rvbaDaviey: test done, I have other problems with MAAS and Django 1.4 that I need to investigate but all the piston related problems are gone with the trunk version.  Also, from what I understand of the problem, the fix in piston completely makes sense.16:09
rvbaOur 173 API tests pass with piston trunk version.16:10
Davieyrvba: to confirm, you used quantal piston and that patch applied?16:11
rvbaDaviey: no, I changed the Django version in versions.cfg to 1.4 and then I used piston trunk version.16:12
Davieyrvba: ah, i hoped you validated just that changeset :)16:13
rvbaDaviey: I can do that.16:13
roaksoaxrvba: btw were you able to find out why the errors with wsgi?16:21
rvbaroaksoax: no, I was stopped by this Django 1.4 incompatibility problem.16:23
rvbaDaviey: ok, so we need these changesets: https://bitbucket.org/jespern/django-piston/changeset/3a0d021dd042#files https://bitbucket.org/jespern/django-piston/changeset/7c90898072ce16:24
rvbaDaviey: with these two patches all the API test pass with Django 1.4.16:24
rvbatests*16:24
rvbaroaksoax: I need to go now but tomorrow I'll fix the problem which prevent the package from being installed.  Then with the new package we will be able to investigate that wsgi problem.  How does that sound?16:26
rvbaprevents*16:26
roaksoaxrvba: sounds good to me16:28
roaksoaxI'll have a new yui3 package on PPA too which MAAS will be depending on16:28
Davieyroaksoax: Do you want to cherry pick those two commits that rvba outlined?16:29
DavieyBest to upload to -proposed as it's seeded.16:29
roaksoaxDaviey: sure, I'll cherry pick those and upload python-piston to -proposed16:30
Davieyta16:30
rvbaroaksoax: ta16:31
roaksoaxDaviey: btw.. is the installer already with squashfs?16:57
Davieyroaksoax: no16:58
Davieyroaksoax: I hit a blocker trying to do it... needs some cjwatson love.16:58
=== matsubara is now known as matsubara-afk
Davieyroaksoax: i do have a squashfs if you wanted to play with it?16:58
Davieybut the d-i part seemed to wedge, when the squash is co-located on cd16:59
Daviey(cd/usb)16:59
roaksoaxDaviey: sure, point me to your squashfs image and I'll play with it16:59
roaksoaxDaviey: python-django-piston has just been uploaded17:03
Davieyroaksoax: thanks17:06
Davieyroaksoax: I'll create a new squashfs tomorrow, and send it along17:07
roaksoaxDaviey: awesome!17:07

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!