=== matsubara is now known as matsubara-afk [02:12] Hi! I'm trying MaaS but I'm facing a lots of problems... Anyway, is this a good doc: http://people.canonical.com/~gavin/docs/lp%3Amaas/install.html ? [02:13] I'll try first using Ubuntu 12.10 [02:15] Also, I'm reading this: http://www.ubuntu.com/cloud/private-cloud/reference-architecture this: https://help.ubuntu.com/12.04/serverguide/ubuntucloud.html and this: https://help.ubuntu.com/community/UbuntuCloudInfrastructure ... plus some others... [02:16] I'm finishing maas-import-pxe-files now... Is maas-import-isos still necessary under 12.10 ? [02:18] I already tried MaaS more than 10 times by now... But I can't figure out how to put it to work smoothly... :-( [02:19] BTW, I'm the author of the XenLiveCD v2.0... I meant, I have a good experience with Debian, Ubuntu, Xen, KVM, etc... But, MaaS is still a bit dark for me... =P [02:20] I'll try again with Ubuntu 12.04... maas-dhcp/maas-dns package on 12.10 do nothing... No dhcp server running after a fresh install... :-/ [02:55] ThiagoCMC: the documentation is being revamped for 12.10, try http://evilnick.org/MAAS/ [02:56] (this will be published on maas.ubuntu.com for the release) [02:57] ThiagoCMC: actually, the ~gavin links also good - it contains the latest documentation generated from trunk [03:02] Thank you!! [03:08] Just to make sure... Is MaaS usefull without Juju?! I mean, can I play a bit with MaaS before learn and install Juju?! [03:09] Maybe something like add / install / remove nodes, manually deploy some services on this nodes, etc...? [03:15] I'm seeing that the Ubuntu Installer, after accept and commission, I think, it just PXE boot and install a Ubuntu Server on the "machine / destination MAC" without asking me about node's disk partitions, user password, etc... And I an unable to login into a fresh installed node, even after it is "ready"... Is there a default user/password?! Is this the expected behavior? [03:15] Sorry if I'm asking too many things at once... [03:16] I really need to fully understand MaaS... [04:27] Guys... MaaS installed and managing my DHCP + DNS (Ubuntu 12.10) but, the following message do not disappear: "The region controller does not know whether any boot images have been imported yet. If this message does not disappear in 5 minutes, there may be a communication problem between the region worker process and the region controller. Check the region worker's logs for signs that it was unable to report to the MAAS API." [04:27] Wha can I do?! [04:28] I already do this: http://evilnick.org/MAAS/ [04:28] did... [04:31] Where is the "region worker's logs"? [04:31] Is it the Apache's log? [04:44] ThiagoCMC: yes, sure, you can use maas without juju [04:45] IIRC the region worker logs would be in /var/log/celery* on your maas server. [04:46] The worker runs under celeryd, which is entirely separate from apache and the webapp. [04:58] Okay... Tks! [05:00] I saw some errors within the Apache log... Something related to /home/maas not being found, and some other errors... Just delete it and doing a new fresh install again... Now I'll use only 1 eth within MaaS server... [05:44] Well, fresh installed Ubuntu 12.10 with maas dhcp+dns and, I'm seeing a error under apache error.log: 1- "[client 127.0.0.1] File does not exist: /var/www/MAAS" 2- "[alert] (2)No such file or directory: mod_wsgi (pid=7694): Unable to change working directory to '/home/maas'."... [05:45] Also, the message "...If this message does not disappear in 5 minutes..." don't go away... Even after ~5 minutes of "sudo maas-import-pxe-files" being executed... [05:53] What can I do? Can I starting adding nodes? It will work?! =P [06:46] Is it possible to "enlist (add a node?)" into MaaS, a already-running fresh-installed Ubuntu 12.04 Server? I do not want use DHCP/DNS/PXE/AvahiBoot... I only want to install my ubuntu server, with my raid, my user/password and add it to MaaS, is it possible? If yes, how? [06:58] YAY! Two nodes deployed... But I'm seeing a lots of errors... On Apache, on dns resolving... [06:58] Anyway, thank you guys! [07:00] I'm still seeing " If this message does not disappear in 5 minutes"... But the MaaS seems to be working now... lol [07:02] I'll try with Ubuntu 12.04 now... 12.10 is great! ^_^ [07:06] rvba: hi, can you help me with this: when running make syncdb I get "initdb: Postgres-XC node name is mandatory" and "pg_ctl: database system initialization failed" [07:09] dimitern: Hi. I've never seen that error before… I assume you have all the dependencies installed and you are on Quantal right? [07:10] rvba: yes, but once I run make syncdb before it complained pg_ctl is missing, so I installed postgres-xc as was recommended, and now i'm getting this [07:11] dimitern: I don't have postgres-xc installed… [07:11] rvba: hmmm... so which package do I need for pg_ctl it's trying to run? [07:12] dimitern: can you paste the full error? The one that says you have a missing binary. [07:13] rvba: just a sec, I'm removing postgres-xc to get the original error [07:30] rvba: crap, so now it fails to install and I reverted the vm to a previous snapshot and need to do the release upgrade again, let's see if it'll work - when I get to the error I'll let you know, 10x [07:31] s/to install/to uninstall [07:31] k [08:16] rvba: hey, so I have some progress. After a fresh upgrade to quantal, make install-dependencies, make syncdb, I get this error: ImportError: No module named postgresfixture.main [08:17] dimitern: the postgresfixture should have been downloaded from pypi. [08:18] dimitern: did you configure a buildout cache in ~/.buildout ? [08:18] whoops no, that's what I missed, I'll do it now [08:19] dimitern: even if you did not do that, it should have worked, but you will have to re-download eggs from pypi each time you create a new branch in a new directory. [08:20] dimitern: you need to create that conf file http://paste.ubuntu.com/1282675/, plus the 3 directory mentioned in there. [08:21] rvba: ok, I did that and running syncdb fails, so I did make clean [08:22] dimitern: is ~/.buildout/eggs/ populated? [08:22] dimitern: does it contain postgresfixture-0.1.2-py2.7.egg ? [08:23] rvba: it's populated, but it's still installing so no postgres yet [08:23] (i did make again after make clean) [08:25] dimitern: Are you trying to develop maas, or do you just want to run it? [08:26] allenap: well, develop, if I can make it running :) [08:27] dimitern: Cool, right. Someone just whispered in my ear that you're the new on the Blue Squad. Welcome :) [08:27] allenap: yeah, 10q :) [08:27] allenap: I think you missed him yesterday, dimitern is the new guy on Blue squad. [08:28] hi dimitern, sorry I missed you around earlier. I think mgz ran into the same thing you are seeing, with pg_ctl conflicting with the postgres package itself. [08:28] jam: How did I manage that I wonder? [08:28] I think he said 'apt-get update' fixed it, but I don't quite remember. [08:28] well, initially i started dist upgrade from ssh and i think that screwed up the thing at some point, so i restored from a clean snapshot and now seems working better [08:55] Daviey: do we need any MAAS release notes? I don't see any in https://wiki.ubuntu.com/QuantalQuetzal/ReleaseNotes/UbuntuServer but I'd like to have one for bug 1064291 for ARM juju/MAAS users [08:55] Launchpad bug 1064291 in Release Notes for Ubuntu "Default constraints make no sense on MAAS" [Undecided,New] https://launchpad.net/bugs/1064291 [08:56] rvba: so now, running syncdb gets me this: [08:56] FATAL: could not open file "pg_xlog/000000010000000000000001" (log file 0, segment 1): No such file or directory [08:56] child process exited with exit code 1 [08:56] initdb: removing contents of data directory "/home/maas/work/maas/db" [08:56] pg_ctl: database system initialization failed [08:58] Hum… I think I need to summon our pg expert here… jtv ^ any idea on what might be going on? [08:58] Huh? [08:59] Hmmm [08:59] Sounds as if it might be an inconsistent database backup. [09:00] i did apt-get update/upgrade just in case - the same error [09:00] jtv: ok, so any clue how i might fix it? [09:00] I haven't read the full backscroll... this is in the db directory in your branch? [09:00] i did make syncdb in the main maas dir [09:01] If there's nothing valuable in that database, you can always "make distclean ; rm -rf db" [09:01] And then start again with your database. [09:01] The postgres-XC bit is surprising to me as well. [09:02] Is this a system that was upgraded from an older ubuntu? Or a fresh Quantal install? [09:02] i did rm -fr db, and run syncdb again - same error [09:02] it's an upgrade from P [09:03] So... upgrade from postgres 9.1 to postgres 9.1 I guess. [09:03] Maybe ls /etc/postgresql/ just to be sure? [09:03] it's 9.1 [09:04] I hope we didn't get any unexpected config changes in the package. [09:04] And you did "make install-dependencies," right? [09:05] yes [09:05] I'm did install --reinstall postgresql-9.1 now [09:05] make syncdb - same error [09:05] weirdweirdweird [09:06] mgz: didn't you run into postgres-XC issues? [09:06] He doesn't have postgres-XC installed, AIUI [09:07] But yes, there was an error about it. [09:07] dimitern: just as a sanity check, make sure you have enough disk space, and double check what your directory permissions are. [09:07] I expect those are fine, but it has tripped me up before. [09:07] no, I had -xc installed yesterday, after upgrading to 12.10 and running make complained missing pg_ctl, running it, command-not-found recommended postgres-xc, so i installed it [09:08] ah [09:08] but today i did a fresh upgrade from a snapshot and now pg_ctl is there, after install-deps [09:08] It should be there anyway, just not necessarily in your PATH. [09:08] not much info on that pg_xlog error [09:09] The _usual_ cause for that error is sysadmins thinking "oh, it's only a log so I can delete it" [09:09] But somehow I don't think that's the case here. :) [09:10] jam: it's a misleading error [09:10] the actual issue was the database was unhappy [09:11] I wonder if any replication code is still running... did you purge postgres-xc? [09:11] As in apt-get remove --purge? [09:11] actually, postgres-xc failed to stop and uninstall [09:11] that's why i reverted to a snapshot [09:12] the coordinator hangs while stopping [09:12] so, postgres was running okay, but the test, and running make syncdb etc failed trying to run pg_ctl, without a clear reason [09:12] yes, i did it with --purge [09:12] argh. [09:13] and if you just run pg_ctl as is, you're helpfully told to install the postgres-xc, which is not actually what you want, the command just needs to be run as the right user [09:13] If possible, I'd purge postgres/postgresql-9.1 as well. [09:14] And the command is hidden away somewhere in /usr/lib -- it's not going to be in your PATH. [09:14] ok, but doesn't install --reinstall do just that? [09:14] You shouldn't normally need to run it yourself. [09:14] I'm not sure exactly what --reinstall does, or how carefully it checks for errors. It sounds as if you're in a state you're not supposed to be in, so things will get confused. [09:14] right, you don't want to, apart from the test fails with "command returned 1" or something, rather than any indication of the actual problem [09:15] so, trying to run the same thing to see what the *actual* problem was seemed reasonable [09:15] but anyway, right response is to ignore all feedback, and just wipe and start again :D [09:15] so I did --purge, then install postgresql-9.1, make syncdb - same error :( [09:16] dimitern: and you said you did rm -rf db in the maas tree to? [09:17] mgz: yes [09:17] dimitern: another option, if you want to divert via setting up canonistack credentials, would be to use my script which gives you maas on an instance in canonical's cloud [09:17] mgz: but anyway db/ is empty [09:18] Best remove it altogether so that our db setup script won't think it has something to work with. [09:18] mgz: does it give maas me from trunk? [09:19] yes, it's a mini dev environment thing [09:20] so, in /var/lib/postgresql/9.1/main/pg_xlog/ there is the 0000..... file, i suspect the permissions are the problem [09:21] No, that file will be for the default postgres cluster. [09:21] You get your own private little cluster in the branch. [09:21] dimitern: I thought 'make syncdb' was supposed to only install locally in $PWD/db/ [09:21] What jam said. [09:21] It also means that permissions are never an issue: you are sole owner and master of your in-branch cluster. [09:21] You are root, you are admin, you are God. [09:22] jtv: ok, but is it supposed to create a structure in db/, similar to the one in /var/lib/postgresql/9.1/main ? [09:22] dimitern: db/ [09:23] 'make syncdb' should not touch /var/lib [09:23] dimitern: yes, it creates its own local version of what you see in /var/lib/... [09:23] With its own logs and everything. [09:23] :) ok, so I mkdir db/pg_xlog and now it fails with a different error: initdb: directory "/home/maas/work/maas/db" exists but is not empty [09:24] that sounds hopeful [09:24] You should not need to create pg_xlog. [09:24] And the db directory should be either functional, or not there at all. [09:27] well, I probably have a rather unusual setup - I have a vbox shared folder (for some reason mounted automatically in /media/sf_work), and then I did a mount --bind /media/sf_work /home/maas/work [09:27] and the permissions seem more and more to be the problem [09:27] Oh, your branch is on a "weird" filesystem? [09:28] :) [09:28] something like it [09:28] As far as the database is concerned, as long as you are consistently running as the owner of that home directory, there should be no permissions problem. [09:28] But running a database out of a weird filesystem is a problem. [09:28] You get things like locks not working properly. [09:28] ok i'll change this so it'll work on a local fs [09:29] you mean weird like nfs? or weird like fat? :) [09:29] Either is icky. [09:29] vbox shared folders is rather icky [09:30] Don't do that then. You may be able to link the db directory to something local, so that the rest of your branch is still remote-mounted but the database state is only on the machine that you're running on. [09:31] jtv: how do I do that inside a bound mounted dir? [09:32] You create a softlink to somewhere on your local filesystem. But I don't know off the top of my head whether that directory already existing, or being a softlink, might confuse our setup script. [09:33] I'd try this the simple way first. [09:33] We can worry about making it cool later. [09:34] I moved the whole repo locally, keeping the paths and removed the vbox sf, will see it'll work that way [09:34] Exactly. Walk before you run. [09:35] whooha! it worked :) 10x guys [09:35] \o/ [09:49] i should probably add bin/ and scripts/ to PATH, right? assuming a normal installation these should be available [09:50] but after doing that, maas-import-pxe-files tries to access /var/lib/maas and fails [09:50] I just sit in the tree and use bin/whatever [09:51] yeah, but if I do that, some scripts fail, since they expect (i.e. maas-provision) to be available globally [09:52] ./scripts/maas-import-pxe-files: line 148: maas-provision: command not found [09:52] see the 'Development MAAS server setup' section in HACKING [09:52] you need maas-provision installed [09:54] so from the repo, not the one in trunk? because I have bin/maas-provision [09:54] E: Package 'maas-provision' has no installation candidate [09:55] dimitern: Those scripts are exceptional, in that they ought to die before 13.04 (where die means burnt, ashes scattered to the winds) and replaced with something tested. tl;dr Don't draw too many conclusions from their behaviour. [09:56] mgz: maas-provision is no longer generated iirc; it was the old Cobbler package. [09:57] allenap: it's in hacking, and what I did to get the images I think, though probably on precise [10:01] allenap: ok, I have the webapp running on 5240, but it complains ths images are missing - how do i get them without running that import script? [10:02] dimitern: Don't worry about them unless you're doing something that involves them; you can safely ignore them for now. [10:03] ok, 10x [10:04] mgz: How odd. I'm not sure how that instruction got in there, but we should remove it. [10:22] Any reviewer in the house? https://code.launchpad.net/~rvb/maas/bug-1066938-rndc2/+merge/129841 [10:25] rvba: Sure. [10:26] allenap: ta [10:27] rvba: How do you test packaging branch changes? lxc-start-ephemeral appears to be completely broken in Quantal, so my scripts no longer work. [10:28] allenap: I create a package and then I test it on a canonistack instance. [10:33] rvba: How do you create the package? [10:34] rvba: The default controls stanza. What silently adds it? Does named see that there's not control statement and add it internally, or is it something the init scripts do? [10:34] allenap: you first have to change 2 things in the packaging branch to point to the right branch/revision [10:34] http://paste.ubuntu.com/1282812/ [10:35] allenap: then bzr bd -- -kmyemail@example.com [10:35] allenap: no, named does that without telling anyone apparently. [10:36] allenap: bzr bd -S -- -kmyemail@example.com if you just want to build the source package. To upload it to a ppa for instance. [10:36] And for the sake of everything you love, do not insert a space between the -k and the email address. [10:36] :) [10:36] Cool, thanks both of you. And FFS to the world. [10:37] allenap: forgot something, if you create the source package to upload to a ppa, you also need to s/UNRELEASED/quantal/ in debian/changelog. [10:37] As in 'maas (0.1+bzr1277+dfsg-0ubuntu2) UNRELEASED; urgency=low' -> 'maas (0.1+bzr1277+dfsg-0ubuntu2) quantal; urgency=low' [10:38] * allenap high-fives bureaucracy in software [10:38] Red tape binds the universe together. [10:38] I am feeling very cynical today. [10:38] Didn't know you had it in you. [10:38] Hah :) [10:38] this all makes sense in the context of debian, but I inevitably forget at least one of the packaging steps :) [10:39] generally changelog related... [11:37] allenap: you are aware that you have a long-approved MP apparently waiting for you to land it? [12:01] rvba: I'm getting "access_refused to vhost '/maas_workers' for user 'maas_workers'" after trying to get rabbit back up and running after the IP change. [12:02] Is there something obvious I'm missing? [12:02] dpkg-reconfigure does redo the maas_workers password, but it doesn't seem to add it to the vhost? [12:03] allenap: ^^ if you have an idea [12:04] jtv: Yes, but I haven't figured out how to test it yet, wrt jam's comment. I'm actually procrastinating in the hope that Django will be recalled worldwide and fired into the sun, thus avoiding the need to test it. [12:05] allenap: chances are slim. Think of the pollution it might cause in the sun. [12:05] jam: No idea I'm afraid :-/ My RabbitMQ knowledge is minimal. [12:05] allenap: well, another check sasy "rabbitmqctl list_vhosts" doesn't show a maas_workers vhost [12:16] jam: I don't know, sorry. roaksoax, can you help jam with this packaging problem? [12:28] allenap: so I think I sorted it out, it looks like 'dpkg-reconfigure' sees that there is already a 'mass_worker' and changes its password, but doesn't notice that there isn't a /maas_workers vhost, so it doesn't create it and set the permissions. [12:28] (it uses the 'is there a user' and assumes that if the user exists, then the vhost exists). [12:28] And I think the bug is that if your hostname changes, rabbit changes its working directory. [12:29] which apparently means it keeps the same users, but not vhosts... [12:29] not sure [12:29] rvba: ^^ [12:29] We get rid of Cobbler, and end up with Django and RabbitMQ instead. Groan. [12:32] jam: that's a packaging bug I was discussing with roaksoax yesterday evening actually. He wanted me to test something for him and I just noticed that problem (http://paste.ubuntu.com/1282969/). [12:33] rvba: right, it just needs to decouple the 'is there a vhost' from 'is there a user'. [12:33] Yep. [12:33] jam: can you please file a bug so that we don't loose track of this. [12:33] though I don't know what permissions happen if you have a user but not a vhost, or a vhost but not the user. [12:33] rvba: is there a tag for packaging issues? [12:33] roaksoax is currently refactoring that code. [12:33] and what ist he 'packaging' branch specifically? [12:33] the [12:34] jam: just assign the bug to the package in ubuntu. [12:37] rvba: bug #1067332 [12:37] Launchpad bug 1067332 in MAAS "'dkpg-reconfigure' conflates user existing for vhost existing" [High,Triaged] https://launchpad.net/bugs/1067332 [12:38] roaksoax: could you please have a look at the bug jam just filed? (1067332) [12:38] jam: ta [12:40] rvba: bug #1067336 is that something already known that I should be duping against? [12:40] Launchpad bug 1067336 in MAAS "maas startup can leave lock file" [High,Triaged] https://launchpad.net/bugs/1067336 [12:40] Essentially, I see a stale lock file in /run/locks [12:41] if startup crashes (say because rabbitmq is refusing connections :) [12:43] jam: there is a 'finally: lock.release()' in the startup code… so I'm surprised that the lock file does not get cleaned up. [12:43] rvba: well, it happens somehow. [12:45] rvba: maybe something is stuck in a loop in inner_start_up. [12:45] I do see your point about the f inally. [12:51] jam: the start_up method fires off tasks… maybe it's stuck in there because rabbitMQ cannot be reached. [12:51] rvba: yeah it is possible it is actually stuck during startup. I'm guessing we are also in one of those 'we aren't connected to a logger yet' issues. === matsubara-afk is now known as matsubara [14:31] allenap: jtv: bzr1280 is currently building in my ppa (ppa:rvb/maas), in case you guys want to test the latest package. [14:48] allenap: please add a debian/changelog entry for your change to the packaging branch [14:48] roaksoax: Okay. [14:49] allenap: refer to the bug too [14:56] roaksoax: I don't really know what I'm doing with the changelog. Does http://paste.ubuntu.com/1283217/ look reasonable? [14:57] roaksoax: Or should I just edit the 0ubuntu2 stanza? [14:57] allenap: no that was released yesterday [14:57] Okay. [14:57] hold on [14:58] * allenap holds [14:58] allenap: http://paste.ubuntu.com/1283222/ [14:59] roaksoax: Ah ha, okay cool. Thanks. === adam_g_ is now known as adam_g === matsubara is now known as matsubara-lunch [17:11] Anyone know why am I getting Internal Server Error with the following thing: ImproperlyConfigured: Middleware module "django.middleware.csrf" does not define a "CsrfResponseMiddleware" class [17:12] Anyone know why am I getting Internal Server Error with the following thing: ImproperlyConfigured: Middleware module "django.middleware.csrf" does not define a "CsrfResponseMiddleware" class ? === matsubara-lunch is now known as matsubara === hazmat is now known as kapilt === matsubara is now known as matsubara-dinner === kapilt is now known as hazmat