#maas 2012-10-15
<jam> morning all
<jam> rvba: morning
<rvba> jam: Hi
<jam> I don't know if you've seen this, but it appears that maas-region-controller gets *horribly* broken if you have your IP address changed.
<jam> I'm trying to setup a 'large' region on EC2, and I shut it down while I went away, so when I started it again, it came back up with a different IP.
<jam> lots of bits were broken, and so far, no luck actually getting it running.
<jam> So I'm trying to just purge everything and reinstall.
<rvba> jam: by horribly broken you mean that nothing tells you that the connection is broken?
<jam> dpkg-reconfigure maas-region-controller  doesn't actually seem to reset things to get them to the right IP addresses and URLs.
<jam> rvba: well, #1 I filed a bug on, which is that the cluster spins forever trying to connect to the region
<jam> but fails
<jam> and fails before it starts celery
<rvba> Right.
<jam> so it doesn't even have task_logger hooked up.
<jam> but the maas-region-controller itself sits spinning.
<jam> because we now set BROKER_URL
<jam> and that is no longer valid after the IP address changes.
<jam> also, rabbitmq-server is... unhappy at best.
<rvba> I see.
<rvba> Well, not really rabbitmq
<jam> like 'service rabbitmq-server stop' claims that it is stopping the broker
<jam> but 'beam.smp' is still running.
<rvba> Celery will be unhappy.
<jam> rvba: celery was at least logging that there was a problem.
<rvba> That looks like a problem in rabbitmq itself.
<jam> rvba: sure
<jam> I'm not saying it is Maas's fault that rabbit is broken, but it was a lot of stuff that was just 'not working', and I couldn't really figure out how to fix it.
<jam> note that squid was also ended up pointing to the wrong IP address, etc.
<rvba> Interesting.
<rvba> And scary :)
<jam> rvba: also, there are some bits where it wasn't clear whether the dpkg-reconfigure wanted the URL or the IP Address.
<rvba> jam: I'm surprised that restarting rabbitmq does not fix the problem (the problem with rabbitmq of course).
<jam> rvba: well 'service rabbitmq-server stop' doesn't actually stop it, so the restart doesn't really work.
<jam> rvba: the cluster controller also calls itself name='master', is there supposed to be a way to configure that?
<rvba> jam: when the cluster controller connects to the region, it changed the name of the nodegroup using its UUID.
<rvba> changes*
<jam> rvba: http://ec2-23-23-14-48.compute-1.amazonaws.com/MAAS/api/1.0/nodegroups/?op=list
<jam> not that I see
<jam> rvba: though... it is still failing to startup, so there may be something else going on.
<rvba> I can't get the page to load.  But I suppose you are still seing the cluster named 'master' ... this simply means that the cluster controller was unable to connect to the region.
<jam> on the plus side, EC2 c1.medium are a lot faster than my VM, and I can do 10,000 nodes in 3s, vs the 12s I was seeing locally.
<jam> rvba: ah, you aren't in the security group, I imagine.
<jam> rvba: so I see 2 nodegroups listed
<jam> so it is somehow getting a message through, but not enough to change its name.
<jam> rvba: http://paste.ubuntu.com/1280693/
<jam> (if you want to help, you can give me your ip address and I can add you to the security group)
<rvba> jam: I have access now.  ta
<rvba> jam: having 2 clusters named 'master' is really a weird situation.  Not really scary because the 'master' nodegroup is identified by picking the *first* nodegroup but still, the situation is really bizarre.
<jtv> mgz: my diff has updated.  Thanks for reviewing it.  :)
<mgz> jtv: I will now review for real :)
<jtv> Thanks!
<jtv> I'll have to go catch a train in a minute.
<jam> well, I got up to 10 cluster controllers all talking to 1 master (so 11 total) and 4k nodes on each, for 44,000 nodes total. Rebuilds tags in about 15s, but the 2-cpu region controller is definitely the bottleneck.
<jtv> jam: great to hear that you got that working... would be very interesting to know where the bottlenecks at finer granularities will be.
<rvba> jam: that's very good news.
<jam> jtv: well, in the immediate term I'm going to restart the central machine with c1.xlarge instead of a c1.medium, and see what I can get it to do.
<jtv> Heh -- you did 44K nodes without even scaling up the central server?  That is happy news indeed.  :)
<jam> jtv: well, there are certainly aspects of the system that aren't scaling well.
<jam> You can't really load "http://.../MAAS" or "http://.../MAAS/nodes/"
<rvba> The whole page is definitely one of them :)
<jtv> The main page in particular might be nice to have working......
<rvba> jtv: this is not completely trivial, txlongpoll is the component that should be improved/replaced.
<jtv> mgz: Raphael got in first
<jam> rvba: the big issue is that whatever lag is introduced into the system makes it *very* hard to play with it. 10s lag spike without interaction means I miss the whole time the rebuild is happening.
<mgz> jtv: I also posted, with a note you may want to look at
<jtv> Willdo
<mgz> well, the last sentence, the rest is just me thinking
<mgz> basically, expression should use // not / I think
<jam> on the plus side, I discovered that you can tell Amazon that everything in a given *security group* should be able to talk to eachother on a given set of ports. So when you add new nodes, they automatically can talk to eachother.
<jtv> mgz: I think you're right, but also I think it's a separate issue.  I haven't tried it out but I suspect the test picks numbers that happen to come out to an even amount in megs.
<mgz> jam: yup.
<jtv> FWIW I reviewed that branch and said please to document these little things...
 * jtv -> train
<jtv> See you tomorrow, folks!
<mgz> >>> n>>20<<20 == n
<mgz> False
<mgz> so, nope...
<mgz> I didn't understand the security group that permits itself idiom for ages, but it's actually quite handy.
<jam> mgz: the downside is that I don't have a way to 'ssh' to all the machines and poke at them... which is what Juju would be nice to have.
<mgz> hm, you do, but probably not in a handy way without writing a wrapper script
<mgz> you can always permit 22 in the same group, or add default and that group, and the api will give you the hostnames/ips
<jam> mgz: sure, but managing 10+ machines manually is certainly at the point of a bit of pain.
<jam> add to it the multi-second latency spikes
<mgz> hm, is trunk make broken currently?
<mgz> I probably just need to clean first I guess...
<mgz> clean not removing the stuff in bin/ and distclean removing a local egg cache is annoying
<mgz> maybe I need to add a global egg cache to my setup script, but it's really not needed apart from if you blow away the working tree files
<mgz> okay, this is fun
<mgz> today's machine can't run tests, because it can't start the db, because pg_ctl is not installed... because the packaging thinks that conflicts with postgres itself?!
<mgz> <http://pastebin.ubuntu.com/1280886/>
<mgz> probably the same as usual need to update my package lists locally...
<mgz> okay, this latest bit is derived from there being no python-selenium package
<jam> everyone say hello to our newest blue squad member, dimitern
<mgz> need to add multiverse... why did this not break earlier I wonder
<mgz> her dimitern
<mgz> *hey
<mgz> dammit, can't even greet without tyop
<dimitern> mgz: hey :)
<dimitern> glad to meet your guys!
<mgz> gah, what is borked with postgres...
<mgz> okay, rm -rf db && make sampledata worked
<mgz> the error reporting for this stuff is painful, CalledProcessError is not very helpful
<rvba> Hi dimitern, welcome aboard.
<dimitern> rvba: hey :) 10x
<jam> mgz, dimitern: so I'm currently load testing MAAS with 1 Region Controller, and 10 Cluster Controllers, each with 4000 node records (in EC2)
<jam> I made the Region controller a C1.xlarge, so it has 8 cores, and the individual nodes are c1.mediums with 2 cores.
<jam> If I just issue 1 tag to be rebuilt, it finishes in 9s.
<jam> (as measured by the time for all requests to complete, the first one completes in a lot less time)
<jam> If I rebuild 2 concurrently, it goes up to 12s.
<jam> Anything higher than that, and it just serializes (presumably because the cluster nodes only have parallel = 2)
<jam> I also think we are roughly at capacity for 1 MAAS controller, because it is going from 9s to 12s when running simultaneous requests.
<jam> Certainly all 8 Apache WSGI processes are 'active' at >25% CPU during this time
<jam> (there should be on the order of 20 concurrent requests.)
<jam> I wonder about over-committing for Apache, though.
<mgz> that sounds as expected
<jam> mgz: well setting wsgi parallel = 20, and I still only ever see ~8 apache processes consuming memory.
<jam> sorry, consuming cpu
<jam> mgz: I also only see postgres consume at most 4 CPUs and normally <3. Is there a good way to tell if postgres is bottlenecking, or do I need to move that onto another machine.
<jam> rvba: what is a reasonable way from python/api to decommision 4k nodes?
<jam> (I want to get rid of all nodes in the 'master' group)
<rvba> jam: I guess you can simply delete() them.  From python directly I mean.
<jam> rvba: Node.delete() ? Or NodeGroup.node_set.delete() ?
<jam> or is the latter only deleting the linkage?
<rvba> jam: the latter is fine.
<rvba> It will delete the nodes.
<jam> rvba: 'RelatedManager has no attribute delete'
<rvba> jam: nodegroup.node_set.all().delete()
<jam> rvba: right, vs nodegroup.node_set.clear() which would just remove the linkage.
<rvba> Yep.
<jam> rvba: interestingly, the bottleneck when deleting 4k nodes is actually' django-admin'.
<jam> Presumably lots of python hooks on the delete method?
<jam> (It has to delete mac addresses for each node it deletes.)
<jam> maybe we should have had a pre-join there :)
<rvba> Yeah, very probably.
<rvba> jam: deleting mac addresses should be handled at the db level (delete cascade).
<jam> mgz: definitely seems to be a 'thundering herd' issue here. Many of  the processes finish significantly before the rest. It appears that they all get their first request started on time, but they tend to get starved out for that first 'give me all the nodes to work on' request.
<jam> rvba: well it should, but we have custom code in Node.delete
<rvba> Yeah, to update the DNS config.
<rvba> I mean we have signals hooked up to node.delete() to update the DNS config.
<jam> ah
<dimitern> jam: how many VMs I need to setup to run maas locally? is 1 enough or I'll need more?
<rvba> That might be heavy.
<jam> dimitern: I use just 1
<dimitern> jam: 10x
<jam> if you actually want to test the PXE booting, etc. You'll want more VMs for that.
<jam> But just for running a server, and poking at the metadata, 1 is enough.
<dimitern> yeah, that's what i figured
<dimitern> i'll setup pxe later, once the webapp is running
<allenap> rvba: FORCE_SCRIPT_NAME and STATIC_URL are driving me up the wall. Do you have time this afternoon to talk about it?
<jam> rvba: so I might have been at fault for the first IP change mess-up. I did it again this time (rebooting to a larger Instance), and all I needed was to do the dpkg-reconfigure on all the machines.
<rvba> allenap: sure
<rvba> jam: all right.  Having to do that is normal given the state we are in.
<rvba> allenap: I can talk now if you're free.
<allenap> rvba: Yeah
<allenap> rvba: Fwiw, this is my solution: http://paste.ubuntu.com/1281102/
<rvba> allenap: looks good to me.  It even fixes using settings.STATIC_URL in src/maasserver/middleware.py which is wrong.  Never actually exercised but still wrong.
<rvba> allenap: even in debug mode on a prod instance, apache serves the static files btw, so this is actually never used in production.
<flacoste> roaksoax, rvba, allenap: anyone looked at bug 1066421 yet?
<ubot5> Launchpad bug 1066556 in debian-installer (Ubuntu Quantal) "duplicate for #1066421 MAAS installed via d-i/tasksel: fails when opening the browser to /MAAS" [Critical,Confirmed] https://launchpad.net/bugs/1066556
<roaksoax> flacoste: yeah, it is duplicate from bug 1065763
<ubot5> Launchpad bug 1065763 in maas (Ubuntu) "UI URL displays "200 Error" page after CD install" [High,Confirmed] https://launchpad.net/bugs/1065763
<roaksoax> rvba: so, is there any way we can use the same queue from txlongpoll?
<roaksoax> rvba: using say... a different vhost?
<rvba> roaksoax: well, a queue belongs to a vhost.
<rvba> roaksoax: can you remind me what you're trying to do again?
<roaksoax> rvba: so during the installtion from cd/preseed, we need to start rabbitmq server (in the chroot) to be able to create the queues (running a daemon in the installer is against policy).
<roaksoax> rvba: so the queue for txlongpoll is being created correctly, but the one for the celery wprkers is not
<roaksoax> so we need to find a workaround for that
<roaksoax> either by using same suser/password and different vhost
<roaksoax> or osmething
<rvba> roaksoax: AFAIK the queues are created dynamically.  We are only creating users and vhosts in the postinst scripts.
<rvba> Am I missing something?
<roaksoax> rvba: right, so that's what I meant
<roaksoax> rvba: the users/vhosts are created for txlongpoll
<roaksoax> rvba: but not for the workers
<roaksoax> rvba: and that's during the installer only
<rvba> roaksoax: how are the workers' user/vhost different from those of txlongpoll?
<roaksoax> rvba: it is not a problem of the postinst
<roaksoax> rvba: it is a problem from running rabbitmq daemon in the installer (chroot), which goes against policy
<roaksoax> but for some reason it does work for the first, (txloinpoll user/vhost create) but not for the second one (celery user worker/vhost creattion)
<roaksoax> rvba: so that's why I was wondering if we would be able to use the same username/password created for txlongpoll, with a different vhost
<rvba> roaksoax: It would be nice to understand why it works for the first and not for the second.
<rvba> roaksoax: there is no real reason why not.  But it's a bit ugly to use the same credentials for two completely unrelated stuff.
<roaksoax> rvba: that's what I'm trying to figure out, but it seems that the first part gets created because rabbitmq is running correctly, but then gets killed and the rest of the stuff doesn't get done
<roaksoax> i've tried to work around it without success
<roaksoax> rvba: i'll further investigate now as I have a couple of other ideas on how to fix it
<roaksoax> rvba: but the issue is basically rabbitmq not running during the installer
<allenap> roaksoax, rvba: Another (better? harder?) solution might be to get the region to configure all this stuff at runtime instead of install-time.
<rvba> allenap: it would be one step towards having less stuff in the packaging.  But obviously we don't really have the "framework" in place to do that properly.
<roaksoax> indeed
<roaksoax> i think ti would be to risky to do it at this point
<roaksoax> being so close to release
<allenap> Fair point.
<dimitern> guys, can you help with errors like this when running buildout on maas src root: Error: There is a version conflict.
<dimitern> We already have: distribute 0.6.28dev-r0
<allenap> dimitern: Are you on Quantal or Precise?
<dimitern> quantal
<dimitern> i did a vanilla install of 12.04 server, updated/upgraded, did release upgrade to 12.10 and called make install-dependencies, then i got this
<dimitern> I read some forums about this error, which recommended to list the package = version in versions.cfg explicitly
<roaksoax> rvba: http://pastebin.ubuntu.com/1281273/
<dimitern> I did that, but then it complains about other packages further on (like ampqlib)
<roaksoax> rvba: i found out what's the issue... i'm such a dumbass
<allenap> dimitern: Here, I have python-setuptools 0.6.28-1ubuntu2 installed, but virtualenv provides 0.6.24 when I create a new env. Can you try:
<rvba> roaksoax: what's the issue?
<allenap> dimitern: virtualenv foo && foo/bin/python -c 'import setuptools; print setuptools.__file__'
<roaksoax> rvba: i just remmebered that we work around it creating the rabbitmq stuff on the upstart job to address this issue
<dimitern> allenap: so I need to run buildout in a virtualenv?
<roaksoax> rvba: btw... the worker stuff is for maas-region-celery.upstart
<allenap> dimitern: Ah. Just type `make` and it'll sort it out for you.
<allenap> dimitern: The virtualenv was a hack to make buildout behave a little better. I suspect we'll actually get rid of buildout eventually, or scale back its role.
<dimitern> allenap: ok, now vbox issues... I'll dig into it some more (shared folders/read only filesystem error)
<dimitern> allenap: it seems to have worked after chown -R :) now it's installing, 10x
<allenap> Cool.
<roaksoax> rvba: the worker stuff is used when running maas-region-celery right?
<rvba> roaksoax: right.
<dimitern> any help on how to resolve this?: maas@maas1:~/work/maas$ make syncdb
<dimitern> bin/database --preserve run -- bin/maas syncdb --noinput
<dimitern> initdb: Postgres-XC node name is mandatory
<roaksoax_> rvba: ok so we are gonna need a wrapper to start maas-region-celery
<roaksoax_> rvba: http://paste.ubuntu.com/1281418/
<roaksoax> '
<roaksoax> rvba: still around?
<rvba> roaksoax: yeah, but I'll need to bugger off real soon.
<roaksoax> rvba: ciukd you please help me test this? http://paste.ubuntu.com/1281498/
<roaksoax> rvba: i'm at ODS and network is crap
<roaksoax> and can't really test it
<roaksoax> rvba: start-region-celery -> create one on /usr/sbin
<roaksoax> rvba: and just modify the upstart job accordingly
<rvba> roaksoax: why a custom script instead of the exec /usr/bin/celeryd ?
<roaksoax> rvba: we need to create rabbitmq credentials on upstart job if tey had not being created on the installer
<roaksoax> rvba: which is the actual workaround we made for txlongpoll
<rvba> roaksoax: yes, I understand that part, but why do you need another startup script?
<roaksoax> rvba: but the upstart job cannot do so because we set the uid,gid to maas
<roaksoax> rvba: cannot run rabbitmq
<roaksoax> rvba: so the best approach is tyo simply make a "wrapper" that runs celery d
<rvba> Ah ok.  I take it it's not possible to setuid maas/setgid maas just before executing exec /usr/bin/celerydâ¦
<roaksoax> rvba: nope unfortunately :(
<roaksoax> rvba: it applies to everything
<rvba> roaksoax: ok, I'll test that for you, but then I'll need to run.
<roaksoax> rvba: awesome thank you
<roaksoax> rvba: i'm tethering from my phone cause internet sucks here
<rvba> roaksoax: looks like it works ok, the celeryd daemon gets started all right.
<rvba> roaksoax: I've manually deleted the user maas_workers.
<rvba> In that case it fails because the pre-start script tried to create the vhost which was already there.
<rvba> roaksoax: if I also delete the vhost it is fine.
<rvba> roaksoax: note that this /usr/sbin/start-region-celery utility is very similar to a helper function we already have so it would be better to share the code.
#maas 2012-10-16
<ThiagoCMC> Hi! I'm trying MaaS but I'm facing a lots of problems... Anyway, is this a good doc: http://people.canonical.com/~gavin/docs/lp%3Amaas/install.html ?
<ThiagoCMC> I'll try first using Ubuntu 12.10
<ThiagoCMC> Also, I'm reading this: http://www.ubuntu.com/cloud/private-cloud/reference-architecture this: https://help.ubuntu.com/12.04/serverguide/ubuntucloud.html and this: https://help.ubuntu.com/community/UbuntuCloudInfrastructure ... plus some others...
<ThiagoCMC> I'm finishing maas-import-pxe-files now... Is maas-import-isos still necessary under 12.10 ?
<ThiagoCMC> I already tried MaaS more than 10 times by now... But I can't figure out how to put it to work smoothly...  :-(
<ThiagoCMC> BTW, I'm the author of the XenLiveCD v2.0... I meant, I have a good experience with Debian, Ubuntu, Xen, KVM, etc... But, MaaS is still a bit dark for me...   =P
<ThiagoCMC> I'll try again with Ubuntu 12.04...  maas-dhcp/maas-dns package on 12.10 do nothing... No dhcp server running after a fresh install...    :-/
<flacoste> ThiagoCMC: the documentation is being revamped for 12.10, try http://evilnick.org/MAAS/
<flacoste> (this will be published on maas.ubuntu.com for the release)
<flacoste> ThiagoCMC: actually, the ~gavin links also good - it contains the latest documentation generated from trunk
<ThiagoCMC> Thank you!!
<ThiagoCMC> Just to make sure... Is MaaS usefull without Juju?! I mean, can I play a bit with MaaS before learn and install Juju?!
<ThiagoCMC> Maybe something like add / install / remove nodes, manually deploy some services on this nodes, etc...?
<ThiagoCMC> I'm seeing that the Ubuntu Installer, after accept and commission, I think, it just PXE boot and install a Ubuntu Server on the "machine / destination MAC" without asking me about node's disk partitions, user password, etc... And I an unable to login into a fresh installed node, even after it is "ready"... Is there a default user/password?! Is this the expected behavior?
<ThiagoCMC> Sorry if I'm asking too many things at once...
<ThiagoCMC> I really need to fully understand MaaS...
<ThiagoCMC> Guys... MaaS installed and managing my DHCP + DNS (Ubuntu 12.10) but, the following message do not disappear: "The region controller does not know whether any boot images have been imported yet. If this message does not disappear in 5 minutes, there may be a communication problem between the region worker process and the region controller. Check the region worker's logs for signs that it was unable to report to the MAAS API."
<ThiagoCMC> Wha can I do?!
<ThiagoCMC> I already do this: http://evilnick.org/MAAS/
<ThiagoCMC> did...
<ThiagoCMC> Where is the "region worker's logs"?
<ThiagoCMC> Is it the Apache's log?
<jtv> ThiagoCMC: yes, sure, you can use maas without juju
<jtv> IIRC the region worker logs would be in /var/log/celery* on your maas server.
<jtv> The worker runs under celeryd, which is entirely separate from apache and the webapp.
<ThiagoCMC> Okay... Tks!
<ThiagoCMC> I saw some errors within the Apache log... Something related to /home/maas not being found, and some other errors... Just delete it and doing a new fresh install again... Now I'll use only 1 eth within MaaS server...
<ThiagoCMC> Well, fresh installed Ubuntu 12.10 with maas dhcp+dns and, I'm seeing a error under apache error.log: 1- "[client 127.0.0.1] File does not exist: /var/www/MAAS" 2- "[alert] (2)No such file or directory: mod_wsgi (pid=7694): Unable to change working directory to '/home/maas'."...
<ThiagoCMC> Also, the message "...If this message does not disappear in 5 minutes..." don't go away... Even after ~5 minutes of "sudo maas-import-pxe-files" being executed...
<ThiagoCMC> What can I do? Can I starting adding nodes? It will work?!  =P
<ThiagoCMC> Is it possible to "enlist (add a node?)" into MaaS, a already-running fresh-installed Ubuntu 12.04 Server? I do not want use DHCP/DNS/PXE/AvahiBoot... I only want to install my ubuntu server, with my raid, my user/password and add it to MaaS, is it possible? If yes, how?
<ThiagoCMC> YAY! Two nodes deployed... But I'm seeing a lots of errors... On Apache, on dns resolving...
<ThiagoCMC> Anyway, thank you guys!
<ThiagoCMC> I'm still seeing " If this message does not disappear in 5 minutes"... But the MaaS seems to be working now...    lol
<ThiagoCMC> I'll try with Ubuntu 12.04 now...  12.10 is great!   ^_^
<dimitern> rvba: hi, can you help me with this: when running make syncdb I get "initdb: Postgres-XC node name is mandatory" and "pg_ctl: database system initialization failed"
<rvba> dimitern: Hi.  I've never seen that error beforeâ¦ I assume you have all the dependencies installed and you are on Quantal right?
<dimitern> rvba: yes, but once I run make syncdb before it complained pg_ctl is missing, so I installed postgres-xc as was recommended, and now i'm getting this
<rvba> dimitern: I don't have postgres-xc installedâ¦
<dimitern> rvba: hmmm... so which package do I need for pg_ctl it's trying to run?
<rvba> dimitern: can you paste the full error?  The one that says you have a missing binary.
<dimitern> rvba: just a sec, I'm removing postgres-xc to get the original error
<dimitern> rvba: crap, so now it fails to install and I reverted the vm to a previous snapshot and need to do the release upgrade again, let's see if it'll work - when I get to the error I'll let you know, 10x
<dimitern> s/to install/to uninstall
<rvba> k
<dimitern> rvba: hey, so I have some progress. After a fresh upgrade to quantal, make install-dependencies, make syncdb, I get this error: ImportError: No module named postgresfixture.main
<rvba> dimitern: the postgresfixture should have been downloaded from pypi.
<rvba> dimitern: did you configure a buildout cache in ~/.buildout ?
<dimitern> whoops no, that's what I missed, I'll do it now
<rvba> dimitern: even if you did not do that, it should have worked, but you will have to re-download eggs from pypi each time you create a new branch in a new directory.
<rvba> dimitern: you need to create that conf file http://paste.ubuntu.com/1282675/, plus the 3 directory mentioned in there.
<dimitern> rvba: ok, I did that and running syncdb fails, so I did make clean
<rvba> dimitern: is ~/.buildout/eggs/ populated?
<rvba> dimitern: does it contain postgresfixture-0.1.2-py2.7.egg ?
<dimitern> rvba: it's populated, but it's still installing so no postgres yet
<dimitern> (i did make again after make clean)
<allenap> dimitern: Are you trying to develop maas, or do you just want to run it?
<dimitern> allenap: well, develop, if I can make it running :)
<allenap> dimitern: Cool, right. Someone just whispered in my ear that you're the new on the Blue Squad. Welcome :)
<dimitern> allenap: yeah, 10q :)
<jam> allenap: I think you missed him yesterday, dimitern is the new guy on Blue squad.
<jam> hi dimitern, sorry I missed you around earlier. I think mgz ran into the same thing you are seeing, with pg_ctl conflicting with the postgres package itself.
<allenap> jam: How did I manage that I wonder?
<jam> I think he said 'apt-get update' fixed it, but I don't quite remember.
<dimitern> well, initially i started dist upgrade from ssh and i think that screwed up the thing at some point, so i restored from a clean snapshot and now seems working better
<rbasak> Daviey: do we need any MAAS release notes? I don't see any in https://wiki.ubuntu.com/QuantalQuetzal/ReleaseNotes/UbuntuServer but I'd like to have one for bug 1064291 for ARM juju/MAAS users
<ubot5> Launchpad bug 1064291 in Release Notes for Ubuntu "Default constraints make no sense on MAAS" [Undecided,New] https://launchpad.net/bugs/1064291
<dimitern> rvba: so now, running syncdb gets me this:
<dimitern> FATAL:  could not open file "pg_xlog/000000010000000000000001" (log file 0, segment 1): No such file or directory
<dimitern> child process exited with exit code 1
<dimitern> initdb: removing contents of data directory "/home/maas/work/maas/db"
<dimitern> pg_ctl: database system initialization failed
<rvba> Humâ¦ I think I need to summon our pg expert hereâ¦ jtv ^ any idea on what might be going on?
<jtv> Huh?
<jtv> Hmmm
<jtv> Sounds as if it might be an inconsistent database backup.
<dimitern> i did apt-get update/upgrade just in case - the same error
<dimitern> jtv: ok, so any clue how i might fix it?
<jtv> I haven't read the full backscroll... this is in the db directory in your branch?
<dimitern> i did make syncdb in the main maas dir
<jtv> If there's nothing valuable in that database, you can always "make distclean ; rm -rf db"
<jtv> And then start again with your database.
<jtv> The postgres-XC bit is surprising to me as well.
<jtv> Is this a system that was upgraded from an older ubuntu?  Or a fresh Quantal install?
<dimitern> i did rm -fr db, and run syncdb again - same error
<dimitern> it's an upgrade from P
<jtv> So... upgrade from postgres 9.1 to postgres 9.1 I guess.
<jtv> Maybe ls /etc/postgresql/ just to be sure?
<dimitern> it's 9.1
<jtv> I hope we didn't get any unexpected config changes in the package.
<jtv> And you did "make install-dependencies," right?
<dimitern> yes
<dimitern> I'm did install --reinstall postgresql-9.1 now
<dimitern> make syncdb - same error
<jtv> weirdweirdweird
<jam> mgz: didn't you run into postgres-XC issues?
<jtv> He doesn't have postgres-XC installed, AIUI
<jtv> But yes, there was an error about it.
<jam> dimitern: just as a sanity check, make sure you have enough disk space, and double check what your directory permissions are.
<jam> I expect those are fine, but it has tripped me up before.
<dimitern> no, I had -xc installed yesterday, after upgrading to 12.10 and running make complained missing pg_ctl, running it, command-not-found recommended postgres-xc, so i installed it
<jam> ah
<dimitern> but today i did a fresh upgrade from a snapshot and now pg_ctl is there, after install-deps
<jtv> It should be there anyway, just not necessarily in your PATH.
<dimitern> not much info on that pg_xlog error
<jtv> The _usual_ cause for that error is sysadmins thinking "oh, it's only a log so I can delete it"
<jtv> But somehow I don't think that's the case here.  :)
<mgz> jam: it's a misleading error
<mgz> the actual issue was the database was unhappy
<jtv> I wonder if any replication code is still running... did you purge postgres-xc?
<jtv> As in apt-get remove --purge?
<dimitern> actually, postgres-xc failed to stop and uninstall
<dimitern> that's why i reverted to a snapshot
<dimitern> the coordinator hangs while stopping
<mgz> so, postgres was running okay, but the test, and running make syncdb etc failed trying to run pg_ctl, without a clear reason
<dimitern> yes, i did it with --purge
<jtv> argh.
<mgz> and if you just run pg_ctl as is, you're helpfully told to install the postgres-xc, which is not actually what you want, the command just needs to be run as the right user
<jtv> If possible, I'd purge postgres/postgresql-9.1 as well.
<jtv> And the command is hidden away somewhere in /usr/lib -- it's not going to be in your PATH.
<dimitern> ok, but doesn't install --reinstall do just that?
<jtv> You shouldn't normally need to run it yourself.
<jtv> I'm not sure exactly what --reinstall does, or how carefully it checks for errors.  It sounds as if you're in a state you're not supposed to be in, so things will get confused.
<mgz> right, you don't want to, apart from the test fails with "command returned 1" or something, rather than any indication of the actual problem
<mgz> so, trying to run the same thing to see what the *actual* problem was seemed reasonable
<mgz> but anyway, right response is to ignore all feedback, and just wipe and start again :D
<dimitern> so I did --purge, then install postgresql-9.1, make syncdb - same error :(
<mgz> dimitern: and you said you did rm -rf db in the maas tree to?
<dimitern> mgz: yes
<mgz> dimitern: another option, if you want to divert via setting up canonistack credentials, would be to use my script which gives you maas on an instance in canonical's cloud
<dimitern> mgz: but anyway db/ is empty
<jtv> Best remove it altogether so that our db setup script won't think it has something to work with.
<dimitern> mgz: does it give maas me from trunk?
<mgz> yes, it's a mini dev environment thing
<dimitern> so, in /var/lib/postgresql/9.1/main/pg_xlog/ there is the 0000..... file, i suspect the permissions are the problem
<jtv> No, that file will be for the default postgres cluster.
<jtv> You get your own private little cluster in the branch.
<jam> dimitern: I thought 'make syncdb' was supposed to only install locally in $PWD/db/
<jtv> What jam said.
<jtv> It also means that permissions are never an issue: you are sole owner and master of your in-branch cluster.
<jtv> You are root, you are admin, you are God.
<dimitern> jtv: ok, but is it supposed to create a structure in db/, similar to the one in /var/lib/postgresql/9.1/main ?
<jam> dimitern: db/
<jam> 'make syncdb' should not touch /var/lib
<jtv> dimitern: yes, it creates its own local version of what you see in /var/lib/...
<jtv> With its own logs and everything.
<dimitern> :) ok, so I mkdir db/pg_xlog and now it fails with a different error: initdb: directory "/home/maas/work/maas/db" exists but is not empty
<mgz> that sounds hopeful
<jtv> You should not need to create pg_xlog.
<jtv> And the db directory should be either functional, or not there at all.
<dimitern> well, I probably have a rather unusual setup - I have a vbox shared folder (for some reason mounted automatically in /media/sf_work), and then I did a mount --bind /media/sf_work /home/maas/work
<dimitern> and the permissions seem more and more to be the problem
<jtv> Oh, your branch is on a "weird" filesystem?
<dimitern> :)
<dimitern> something like it
<jtv> As far as the database is concerned, as long as you are consistently running as the owner of that home directory, there should be no permissions problem.
<jtv> But running a database out of a weird filesystem is a problem.
<jtv> You get things like locks not working properly.
<dimitern> ok i'll change this so it'll work on a local fs
<mgz> you mean weird like nfs? or weird like fat? :)
<jtv> Either is icky.
<dimitern> vbox shared folders is rather icky
<jtv> Don't do that then.  You may be able to link the db directory to something local, so that the rest of your branch is still remote-mounted but the database state is only on the machine that you're running on.
<dimitern> jtv: how do I do that inside a bound mounted dir?
<jtv> You create a softlink to somewhere on your local filesystem.  But I don't know off the top of my head whether that directory already existing, or being a softlink, might confuse our setup script.
<jtv> I'd try this the simple way first.
<jtv> We can worry about making it cool later.
<dimitern> I moved the whole repo locally, keeping the paths and removed the vbox sf, will see it'll work that way
<jtv> Exactly.  Walk before you run.
<dimitern> whooha! it worked :) 10x guys
<jtv> \o/
<dimitern> i should probably add bin/ and scripts/ to PATH, right? assuming a normal installation these should be available
<dimitern> but after doing that, maas-import-pxe-files tries to access /var/lib/maas and fails
<mgz> I just sit in the tree and use bin/whatever
<dimitern> yeah, but if I do that, some scripts fail, since they expect (i.e. maas-provision) to be available globally
<dimitern> ./scripts/maas-import-pxe-files: line 148: maas-provision: command not found
<mgz> see the 'Development MAAS server setup' section in HACKING
<mgz> you need maas-provision installed
<dimitern> so from the repo, not the one in trunk? because I have bin/maas-provision
<dimitern> E: Package 'maas-provision' has no installation candidate
<allenap> dimitern: Those scripts are exceptional, in that they ought to die before 13.04 (where die means burnt, ashes scattered to the winds) and replaced with something tested. tl;dr Don't draw too many conclusions from their behaviour.
<allenap> mgz: maas-provision is no longer generated iirc; it was the old Cobbler package.
<mgz> allenap: it's in hacking, and what I did to get the images I think, though probably on precise
<dimitern> allenap: ok, I have the webapp running on 5240, but it complains ths images are missing - how do i get them without running that import script?
<allenap> dimitern: Don't worry about them unless you're doing something that involves them; you can safely ignore them for now.
<dimitern> ok, 10x
<allenap> mgz: How odd. I'm not sure how that instruction got in there, but we should remove it.
<rvba> Any reviewer in the house? https://code.launchpad.net/~rvb/maas/bug-1066938-rndc2/+merge/129841
<allenap> rvba: Sure.
<rvba> allenap: ta
<allenap> rvba: How do you test packaging branch changes? lxc-start-ephemeral appears to be completely broken in Quantal, so my scripts no longer work.
<rvba> allenap: I create a package and then I test it on a canonistack instance.
<allenap> rvba: How do you create the package?
<allenap> rvba: The default controls stanza. What silently adds it? Does named see that there's not control statement and add it internally, or is it something the init scripts do?
<rvba> allenap: you first have to change 2 things in the packaging branch to point to the right branch/revision
<rvba> http://paste.ubuntu.com/1282812/
<rvba> allenap: then bzr bd -- -kmyemail@example.com
<rvba> allenap: no, named does that without telling anyone apparently.
<rvba> allenap: bzr bd -S -- -kmyemail@example.com  if you just want to build the source package.  To upload it to a ppa for instance.
<jtv> And for the sake of everything you love, do not insert a space between the -k and the email address.
<rvba> :)
<allenap> Cool, thanks both of you. And FFS to the world.
<rvba> allenap: forgot something, if you create the source package to upload to a ppa, you also need to s/UNRELEASED/quantal/ in debian/changelog.
<rvba> As in 'maas (0.1+bzr1277+dfsg-0ubuntu2) UNRELEASED; urgency=low' -> 'maas (0.1+bzr1277+dfsg-0ubuntu2) quantal; urgency=low'
 * allenap high-fives bureaucracy in software
<jtv> Red tape binds the universe together.
<allenap> I am feeling very cynical today.
<jtv> Didn't know you had it in you.
<allenap> Hah :)
<mgz> this all makes sense in the context of debian, but I inevitably forget at least one of the packaging steps :)
<mgz> generally changelog related...
<jtv> allenap: you are aware that you have a long-approved MP apparently waiting for you to land it?
<jam> rvba: I'm getting "access_refused to vhost '/maas_workers' for user 'maas_workers'" after trying to get rabbit back up and running after the IP change.
<jam> Is there something obvious I'm missing?
<jam> dpkg-reconfigure does redo the maas_workers password, but it doesn't seem to add it to the vhost?
<jam> allenap: ^^ if you have an idea
<allenap> jtv: Yes, but I haven't figured out how to test it yet, wrt jam's comment. I'm actually procrastinating in the hope that Django will be recalled worldwide and fired into the sun, thus avoiding the need to test it.
<jtv> allenap: chances are slim.  Think of the pollution it might cause in the sun.
<allenap> jam: No idea I'm afraid :-/ My RabbitMQ knowledge is minimal.
<jam> allenap: well, another check sasy "rabbitmqctl list_vhosts" doesn't show a maas_workers vhost
<allenap> jam: I don't know, sorry. roaksoax, can you help jam with this packaging problem?
<jam> allenap: so I think I sorted it out, it looks like 'dpkg-reconfigure' sees that there is already a 'mass_worker' and changes its password, but doesn't notice that there isn't a /maas_workers vhost, so it doesn't create it and set the permissions.
<jam> (it uses the 'is there a user' and assumes that if the user exists, then the vhost exists).
<jam> And I think the bug is that if your hostname changes, rabbit changes its working directory.
<jam> which apparently means it keeps the same users, but not vhosts...
<jam> not sure
<jam> rvba: ^^
<allenap> We get rid of Cobbler, and end up with Django and RabbitMQ instead. Groan.
<rvba> jam: that's a packaging bug I was discussing with roaksoax yesterday evening actually.  He wanted me to test something for him and I just noticed that problem (http://paste.ubuntu.com/1282969/).
<jam> rvba: right, it just needs to decouple the 'is there a vhost' from 'is there a user'.
<rvba> Yep.
<rvba> jam: can you please file a bug so that we don't loose track of this.
<jam> though I don't know what permissions happen if you have a user but not a vhost, or a vhost but not the user.
<jam> rvba: is there a tag for packaging issues?
<rvba> roaksoax is currently refactoring that code.
<jam> and what ist he 'packaging' branch specifically?
<jam> the
<rvba> jam: just assign the bug to the package in ubuntu.
<jam> rvba: bug #1067332
<ubot5> Launchpad bug 1067332 in MAAS "'dkpg-reconfigure' conflates user existing for vhost existing" [High,Triaged] https://launchpad.net/bugs/1067332
<rvba> roaksoax: could you please have a look at the bug jam just filed? (1067332)
<rvba> jam: ta
<jam> rvba: bug #1067336  is that something already known that I should be duping against?
<ubot5> Launchpad bug 1067336 in MAAS "maas startup can leave lock file" [High,Triaged] https://launchpad.net/bugs/1067336
<jam> Essentially, I see a stale lock file in /run/locks
<jam> if startup crashes (say because rabbitmq is refusing connections :)
<rvba> jam: there is a 'finally: lock.release()' in the startup codeâ¦ so I'm surprised that the lock file does not get cleaned up.
<jam> rvba: well, it happens somehow.
<jam> rvba: maybe something is stuck in a loop in inner_start_up.
<jam> I do see your point about the f inally.
<rvba> jam: the start_up method fires off tasksâ¦ maybe it's stuck in there because rabbitMQ cannot be reached.
<jam> rvba: yeah it is possible it is actually stuck during startup. I'm guessing we are also in one of those 'we aren't connected to a logger yet' issues.
<rvba> allenap: jtv: bzr1280 is currently building in my ppa (ppa:rvb/maas), in case you guys want to test the latest package.
<roaksoax> allenap: please add a debian/changelog entry for your change to the packaging branch
<allenap> roaksoax: Okay.
<roaksoax> allenap: refer to the bug too
<allenap> roaksoax: I don't really know what I'm doing with the changelog. Does http://paste.ubuntu.com/1283217/ look reasonable?
<allenap> roaksoax: Or should I just edit the 0ubuntu2 stanza?
<roaksoax> allenap: no that was released yesterday
<allenap> Okay.
<roaksoax> hold on
 * allenap holds
<roaksoax> allenap: http://paste.ubuntu.com/1283222/
<allenap> roaksoax: Ah ha, okay cool. Thanks.
<guimaluf> Anyone know why am I getting Internal Server Error with the following thing: ImproperlyConfigured: Middleware module "django.middleware.csrf" does not define a "CsrfResponseMiddleware" class
<guimaluf> Anyone know why am I getting Internal Server Error with the following thing: ImproperlyConfigured: Middleware module "django.middleware.csrf" does not define a "CsrfResponseMiddleware" class ?
#maas 2012-10-17
<ThiagoCMC> Guys, the MaaS server really needs two ethernet interfaces?! Even if it is not the default gateway of the nodes?!
<bigjools> why do you think it needs two interfaces?
<ThiagoCMC> Weel, I think I misunderstood this: https://help.ubuntu.com/community/UbuntuCloudInfrastructure -> "...has all nodes with two network interfaces..."... Sorry for the buzz...
 * bigjools reads
<bigjools> huh, I don't know why it says that, I run it with one interface on my server/nodes
<ThiagoCMC> Damn... I'm trouble to find a good network topology for MaaS -> Juju -> OpenStack... I'll try it again...
<ThiagoCMC> BTW, I also liked this: http://people.canonical.com/~gavin/docs/lp%3Amaas/orientation.html#setup
<ThiagoCMC> KISS
<ThiagoCMC> lol
<ThiagoCMC> All my nodes have two gigabit eth...
<ThiagoCMC> I'm thinking about enabling the BOND interfaces before deploying my MaaS nodes on top of it (BOND)... What do you think?! My network will be very busy when in production...
<bigjools> I fixed the wiki
 * bigjools has to run, back later
<ThiagoCMC> Awesome!   :-D
<ThiagoCMC> What the "Ready" status after commissioning means?! I think that "Ready" means "ready to use" but, it is the beginning of the Installation process... Is that normal?
<ThiagoCMC> Also, how can I manually edit my node's harddisks partitions?! The MaaS node installation just do "guided partiutioning" but I do not like it. Any tips?
<ThiagoCMC> Now my newest node is "allocated to root" but, still installing...
<ThiagoCMC> When I click "start node", MaaS starts the node automated-installation...    Have no idea if this is right...
<ThiagoCMC> I'll start from scratch again... Damn...     =(
<ThiagoCMC> This is hard...
<bigjools> ThiagoCMC: Ready means "in the pool of commissioned nodes and ready for users to allocate and deploy"
<bigjools> ThiagoCMC: generally, you want to use juju with maas
<bigjools> makes life much easier
<ThiagoCMC> Mmm... Okay... Gotcha!
<ThiagoCMC> And about node's harddisk partitioning? I want to do it manually.
<ThiagoCMC> Is that possible?
<ThiagoCMC> Or Juju/OpenStack expect it (HD partitions) the way MaaS deploy it? I'm familiarized with nodes that have: two HDs -> raid1 -> LVM -> lv_swap / lv_root / lv_vps1_disk0 / lv_etc...
<ThiagoCMC> Where can I find the IP of my fresh MaaS node?
<ThiagoCMC> I'm using managed DHCP/DNS and using my MaaS DNS server to resolve but, it does not resolve by names...  :-/
<ThiagoCMC> So, there is only one way to figure out the IP of my node: "grep -i MAC /var/log/syslog"  ???
<ThiagoCMC> on maas server?
<melmoth>  ThiagoCMC what about /var/lib/misc/dnsmasq.leases ?
<ThiagoCMC> I'm using Ubuntu 12.10... There is no dnsmasq installed...
<ThiagoCMC> Also, this isn't a expected solution, right?!
<ThiagoCMC> Seems more or less like a "workaround"... I would like to know the IP of a node by MaaS CLI or GUI interfaces... Not this...    =(
<melmoth> i havent used 12.10 nor do i know if there s a more convenient way.
<melmoth> hmmm, what about juju status ? It should list all the machine and their names
<melmoth> at least it does on 12.04
<ThiagoCMC> I still do not have touched Juju...
<ThiagoCMC> I'm learning about MaaS first...
<ThiagoCMC> But tks! I'll take a look into it now...
<ThiagoCMC> The point is, somebody told me that MaaS can be used without Juju, so, I'm trying it...
<ThiagoCMC> Also
<ThiagoCMC> Did you know if it is possible to add a already installed Ubuntu Server 12.04 within MaaS (not installing "through" it)?!
<melmoth> hmm, i think you need juju to deploy nodes. (at least i never did deployment in another way with maas)
<melmoth> you can add a node manually if you know its ip, but i do not know how to have it in ready state without letting maas reinstall stuff on it from scratch
<melmoth> when i added node manually, they ended up in enlisted state i think, so i still had to comission them (wich involve booting and installing stuff on the box)
<ThiagoCMC> Mmm... Okay! Thanks for the info...
<ThiagoCMC> I'm a bit concerned about MaaS HD partitions and RAID1... It just do guided-partitioning... Which sucks...
<dimitern> hey, guys, I'm getting this error after restarting a make run session: setlock: fatal: unable to lock /run/lock/maas.dev.database: temporary failure
<dimitern> any idea?
<dimitern> I just deleted all /run/lock/maas* files and it works now..
<jam> dimitern: that has been my experience. I filed a bug on it, which had the comments 'the locks should go away on reboot', but that has been broken in my experience.
<jam> allenap: bug #1066668
<ubot5> Launchpad bug 1066668 in MAAS "No indication if cluster controller is failing to connect" [High,Triaged] https://launchpad.net/bugs/1066668
<allenap> jam: Ta!
<rvba> rbasak: here is what juju status gives me: http://paste.ubuntu.com/1283078/
<rvba> rbasak: I'm interested in what you have instead of dns-name: 192-168-10-10.master
<rbasak> rvba: I'm just bootstrapping a node now
<rbasak> rvba: will be about 20 minutes
<rvba> Ok, thanks.
<rbasak> rvba: http://paste.ubuntu.com/1284658/
<rbasak> rvba: something must be fundamentally different with the way I'm doing it
<rvba> rbasak: you;re not using MAAS' DNS service.
<rbasak> rvba: I'm installing the maas package only (and juju), and have an external dhcp server
<rbasak> that'll be it
<rvba> Thanks for checking rbasak.
<mgz> fun, `rm -rf db; make sampledata` -> http://pastebin.ubuntu.com/1284868/
<mgz> that did not happen earlier today
<mgz> rvba: is this a known issue with a workaround?^
<rvba> mgz: the dev fixture data relies on the fact that the master nodegroup has id 1.  Which should happen when creating the db from scratch.
<rvba> mgz: wait Problem installing fixture 'src/maasserver/fixtures/dev_fixture.yaml.bz2'
<rvba> Why dev_fixture.yaml.bz2 ?
<rvba> mgz: are you modifying the dev fixture in that branch?
<rvba> mgz: I'm getting the same problem on trunk.  I'm investigating.
<mgz> thanks rvba, confirms it's not just my local setup
<rvba> mgz: I found the problem.  MP coming up.
<rvba> mgz: care to review? https://code.launchpad.net/~rvb/maas/fixture-error/+merge/130120
<flacoste> ThiagoCMC: you could customize the partioning by modifying the preseed data
<dimitern> rvba: I decided to pick up https://bugs.launchpad.net/maas/+bug/1027720
<ubot5> Ubuntu bug 1027720 in MAAS ""maas-nodegroup-worker" is declared in two places" [Low,Triaged]
<dimitern> rvba: so I just set it to in progress and I can begin?
<rvba> dimitern: yep
<rvba> dimitern: that's a good one to get started ;)
<dimitern> rvba: ok :) I have no idea yet, but I'll dig in and ask if needed
<mgz> rvba: looking
<rvba> ta
<mgz> one question, the 'null' behaviour is specified where?
<mgz> because the testing factory.make_node behaviour is different without a nodegroup to what I guess the migration stuff does?
<rvba> mgz: yeah, as I say on the MP, the ensure_master method will take care of the nodes without a nodegroup.
<flacoste> evilnick_: another question/task we should document: how do I modify the partioning done by MAAS
<flacoste> see ThiagoMC questions earlier today in the logs: http://irclogs.ubuntu.com/2012/10/17/%23maas.html
<flacoste> evilnick_: the answer lies in changing the preseed data, but you'll need to chase one of the server guys to get some actual example configuration changes
<flacoste> evilnick_: smoser, roaksoax or Daviey would probably be able to answer that
<mgz> when does that happen though?
<evilnick_> flacoste: ok, thats a great help. I shall check the logs and add it to my list.
<mgz> because if I just make sampledata then look in the db, the nodes don't have a linked nodegroup
<rvba> mgz: indeed, that happens in src/maasserver/start_up.py
<dimitern> rvba: src/maasserver/models/nodegroup.py does not contain any reference to maas-nodegroup-worker or I'm missing something..
<rvba> mgz: maybe we can add a call to ensure_master when make sampledata is run.
<rvba> dimitern: you're right, this was apparently cleaned up in r853.
<dimitern> rvba: :) I figured something like that and started looking at the history to confirm it
<mgz> I think this way is fine, generally sampledata is followed make run or something else for like testing, and that does do the population
<rvba> mgz: Yes, that was my thought.
<mgz> rvba: approved.
<dimitern> rvba: so I commented that and changed the status to Fix Committed
<rvba> dimitern: cool,  that was a very easy one ;).
<rvba> dimitern: I changed is to fix released because r853 has been released already.
<rvba> s/changed is/changed it/
<dimitern> ok cool
<dimitern> I'm picking up #984117 then
<rvba> dimitern: that involves JS stuff.  I'd suggest going after 1061870 (which also is High priority)
<rvba> But it's really up to you.
<dimitern> rvba: after you mean after I do #984117 or instead do #1061870
<rvba> dimitern: I'd suggest doing the API stuff first unless you really like JS :).
<dimitern> rvba: I actually really like JS :) not used YUI though, mostly jquery
<rvba> dimitern: all right then.  Up to you.
<dimitern> rvba: I'm not sure though how to push my changes, etc. when I run bzr status it shows most of the files in the working dir are changed (?!) I'm new to bzr in fact, but I used both git and hg
<rvba> dimitern: that's weird, you've branched off lp:maas, modified a bunch of filesâ¦ and now most of the files are 'modified' (bzr st)?
<dimitern> rvba: I branched of lp:maas trunk, didn't really modify anything yet, just run make (install-deps, run)
<rvba> dimitern: can you paste the result of 'bzr st'?
<dimitern> rvba: just a sec
<dimitern> rvba: http://paste.ubuntu.com/1285115/
<rvba> dimitern: !?
<dimitern> rvba: if I run bzr diff it shows a bunch of properties changes -x to +x
<mgz> heh
<mgz> you're using dodgy filesystems again? :)
<rvba> I was about to ask the same question :)
<dimitern> I remember doing chmod -R before :)
<dimitern> no, this time is a normal fs
<dimitern> in fact chmod -R ug=rwX,o=rX or something :)
<dimitern> because I was not sure what's not working initially with the installation
<dimitern> how I can revert all these changes?
<rvba> bzr revert
<dimitern> ok :) now it looks ok 10x
<dimitern> rvba: is there an easier way to change node statuses instead of working directly with the models from a py shell in django?
<rvba> dimitern: that is really the easiest way if you want to tweak the statuses.
<dimitern> rvba: running bin/maas shell right?
<rvba> dimitern: make harness
<rvba> dimitern: same as bin/maas shell but with the right settings loaded.
<dimitern> rvba: ok 10x :)
<dimitern> rvba: where should this information show (node_views.js) - on the front page with the wheel?
<rvba> dimitern: let me have a look at that bug again first.
<rvba> dimitern: well, we still don't support reserving nodes.
<dimitern> rvba: I saw the https://code.launchpad.net/~allenap/maas/hide-retired-reserved-bug-979902 was merged into trunk, so it should be possible now
<dimitern> at least to show it, if not change it from the ui
<rvba> dimitern: yeah, but there is no point in showing it if it's not possible for a node to be 'reserved'.
<rvba> The branch you mentioned hides the 'reserved' nodes.  And then the bug says: "Once MAAS supports reserving nodes this line can be shown. "
<rvba> This has not happened yet.
<dimitern> I see, so then probably I should pick something else?
<dimitern> like #1061870 you mentioned?
<rvba> Yep, I'm afraid you're back to that APi bug I mentioned.  That's a real one :).
<dimitern> ok :) no sweat
<dimitern> rvba: so I'll set 984117 back to nobody, but what status should I set? Confirmed? I does not seem to have Triaged in the list..
<rvba> 5~
<rvba> arg
<rvba> dimitern: Triaged is in the list, just on top of 'In progress'â¦ sure you don't see it?
<dimitern> rvba: once it's already In Progress it does not show it
<mgz> dimitern might not be in all the correct launchpad groups still?
<rvba> dimitern: I've set it back to 'Triaged'
<dimitern> rvba: could be, who should I ask?
<dimitern> 10x
<mgz> not in ~launchpad yet.
<rvba> I guess flacoste is your man here.
<dimitern> ok
<roaksoax> rvba: so I will be uploading a release to PPA that should be 0-day SRU'ed to quantal for you guys to test and make sure it doesn't introduce any regressions
<roaksoax> allenap: ^^
<roaksoax> since I can test it, you guys will have to help with that
<allenap> Okay.
<flacoste> matsubara: can you help there as well^^^
<flacoste> roaksoax: did you use the branch that Julian created yesterday?
<matsubara> flacoste, sure
<roaksoax> flacoste: in the process of doing so
<flacoste> awesome
<matsubara> roaksoax, let me know when the package is published
<roaksoax> flacoste: network is not really a fun thing here :)
<flacoste> roaksoax: like most non-elmo run conference :-)
<roaksoax> indeed :)
<roaksoax> flacoste: what about bug #1067261 ? It hasn't been merged to the quantal branch
<ubot5> Launchpad bug 1067261 in maas (Ubuntu) "Web reference in UI points to wrong place" [Undecided,New] https://launchpad.net/bugs/1067261
<rvba> roaksoax: is the new package in the experimental ppa?  I don't see it there.
<roaksoax> rvba having trouble uploadib
<rvba> k
<flacoste> roaksoax: that's probably an invonlatary omission by bigjools
<roaksoax> flacoste: alright, will take care of it right before uploading to archive since it is really not a big deal
<roaksoax> big change*
<roaksoax> rvba: should take ~10 mins to be fully uploaded into the archive...
<roaksoax> errexperimental ppa
<roaksoax> matsubara it has been uploaded not yet built
<matsubara> roaksoax, ok
<matsubara> thanks
<ThiagoCMC> Guys, how to disable guided partitioning during the node installation?! Tks!
<ThiagoCMC> I want to do it manually...
<ThiagoCMC> because of my "raid1 -> LVM" setup...
<flacoste> ThiagoCMC: you need to edit the pressed template used to tell d-i how you want your partitioning to take place
<flacoste> ThiagoCMC: unfortunately, i can't provide exact guidance on what you need to put in there
<evilnick_> ThiagoCMC: This probably isn't much help right now, but I am going to find out how to do this and write up the appropriate instructions. Hopefully i'll have something  tomorrow.
<rvba> roaksoax: your package just finished building (I /think/).
<rvba> roaksoax: not yet published apparently.
<rvba> roaksoax: it seems to be stuck :/
<rvba> flacoste: any idea what's happening with the ppa package?  Looks like it's stuck.
<flacoste> rvba: no idea
<roaksoax> yeah stuck in publishing
<martinx_> flacoste, okay... thanks!
<martinx_> BTW, where can I find the IPs of my fresh installed nodes?! Since the managed DNS do not resolv the named that appear in MaaS web GUI...
<martinx_> flacoste, I think that if I use "dpkg get/set selections" I can generate my own preseed... So, MaaS just need to provide a way to import my own preseed, right?!
<martinx_> I'm talking about doing RAID1 -> LVM and copying it to MaaS as a template for same kind of hardware...
<flacoste> martinx_: no, that's a package selection, the preseed is a d-i format which includes package selection but has a special syntax to specify partitoinaing
<martinx_> Mmm... Okay
<ThiagoCMC> Would be very nice if MaaS provides a way to customize its preseeds (hardware related) before starting node installation... Is this kind of feature in the roadmap?
<ThiagoCMC> Do you guys recommends juju PPA repository even if I'm using Ubuntu 12.10 ?
<ThiagoCMC> As I'm reading here: https://juju.ubuntu.com/docs/getting-started.html ... ?
<ThiagoCMC> Guys, the hostname that I'm seeing at MaaS Web GUI is intended to be resolved (with managed DNS enabled)? I'm trying to figure out the IPs of my nodes but I can't... The only way I find my IP is looking into DHCP syslog...   =(
<roaksoax> bug #1067929
<ubot5> Launchpad bug 1067929 in maas (Ubuntu) "maas-region-celeryd connects to the wrong queue." [Undecided,New] https://launchpad.net/bugs/1067929
#maas 2012-10-18
<shang> Hi All, I have 2 questions. 1: Why do we need to manually download the meta file for isos? Is there a reason behind not make it automatically download it at installation time?
<shang> 2. we tried to run: sudo maas-import-pxe-files, but failed. I noticed that: https://maas.ubuntu.com/images/query/precise/ephemeral/released-dl.current.txt is not accessible
<ThiagoCMC> firewall rules?! this machine can browse the web?
<shang> yup, I just tried it on my computer: This page does not exist yet. You can create a new empty page, or use one of the page templates.
<ThiagoCMC> weird...   :-/
<ThiagoCMC> I just run this command few hours ago...
<shang> does that work on your browser?
<freeflyi1g> ThiagoCMC: but why does import-pxe-file need access to that page?
<ThiagoCMC> Have no idea... Sorry but, I'm still learning about MaaS...   =(
<ThiagoCMC> I'm newbie here...    =P
<freeflyi1g> ThiagoCMC: no worries, we're in the same boat :)
<ThiagoCMC> ^^
<shang> sorry for the noise, the link works now
<shang> I can download the files expected :)
<bigjools> o/ lifeless
<lifeless> bigjools: o/
<ThiagoCMC> bigjools, hi! I find more places telling to use two network interfaces with MaaS (I think...)... Look: http://www.ubuntu.com/cloud/private-cloud/reference-architecture "Each nova compute..." and - https://wiki.edubuntu.org/SecurityTeam/TestingOpenStack ...
<ThiagoCMC> I don't know if it is related but, it isn't a requirement, right?!
 * bigjools looks
<bigjools> I don't know about those, they are not maas-related
<ThiagoCMC> ok
<bigjools> I suspect it's for firewalling
<bigjools> so maybe a recommended setup but not necessarily required
<ThiagoCMC> Gotcha!
<mgz> hm, we should probably have a tags index page
<allenap> rvba: Do we pass a queryset to Piston? If so, could we patch the iterator() method for that single queryset, so that it does prefetching? It's a hack, but it's quick as a proof of how to fix the problem.
<rvba> allenap: we pass a queryset of nodes but then the serialization inspect them one by one to get the related MAC addresses.
<rvba> That's what is generating the queries.
<rvba> allenap: does that make sense?
<allenap> rvba: I guess I don't know enough/anything about how Django does prefetching. In Storm that would work, for example, because it the prefetch would pull in the related rows.
<rvba> allenap: yeah, but there is no caching like that in Django.
<allenap> and, depending on how they're related, it could get them from cache.
<allenap> Right, okay.
<jtv> rvba: just noticed something weird...  you made ip_range_low & ip_range_high non-unique because they can be null?
<jtv> Does that mean that we're still storing "null" IP addresses as empty strings?  :(
<rvba> jtv: yes
<jtv> Bugger.  Silly nonsense.
<jtv> If we could store them as proper IP addresses, which can be properly null in the database, this wouldn't be an issue.
<jtv> (because (null = null) -> null)
<rvba> Anyone up for a pre-imp about this performance thing?  I think I possible solution.
<rvba> jtv: allenap ^ ?
<rvba> I see* a possible solution
<jtv> rvba: gimme a sec
<jam> rvba: if you would like
<jam> rvba: I mentioned in the bug that I think you can pre-iterate the queryset by just calling list() on it.
<jam> which I think works around the piston.iterator() issue.
<rvba> jam: I'll try that before I go on with my much complex plan.
<rvba> jam: this does not seem to work.  Because the .iterator() is not problematic on the nodes but on the related MAC addresses.
<jam> rvba: so if it as big issue, the worst case is to return HTTPResponse(json.dumps(stuff))
<jam> but that is a pretty big overhaul to get around piston.
<rvba> But it also means we don't have control over the fields returned.
<rvba> I think I have a solution which simply involve creating a very small specialized emitter.
<rvba> involves*
<jam> rvba: well dumps(stuff) you have full control over what goes into 'stuff'. :)
<rvba> Sure, but I mean you have to fabricate the objects by hand.
<rvba> That means computing the resource_uri stuff by hand etc.
<jam> rvba: looking at the piston Emitter level of local function nested makes my eyes bleed just a little bit :)
<jam> nesting.
<jam> rvba: so the tag_names change, feel free to just land that one.
<rvba> jam: Yeah, I had the same reaction :)
<jam> well, land it with a '.only()'
<jam> so it doesn't have to load all tag columns just for the one attribute.
<rvba> jam: I'll do it, but first, I'll get that specific emitter done.
<rvba> jtv: It looks like the prefetch_related stuff is intelligent enough to restrict the queries it issues if the "original" query is restricted.
<rvba> jtv: unrestricted: http://paste.ubuntu.com/1286632/ / restricted: http://paste.ubuntu.com/1286633/
<rvba> jtv: here is the code that generates these queries: http://paste.ubuntu.com/1286635/
<rvba> jam: with or without the '[:3]' bit of course
<rvba> err, jtv
<jam> rvba: right. Because it doesn't do the prefetch until it has fetched the list of nodes, because it needs their ids to do the other fetch.
<jtv> otp
 * jtv shakes fist at long, unwrapped lines
<jtv> Ah OK
<rvba> jam: now I need to find a way to override a method that is deeply buried.
<rvba> http://paste.ubuntu.com/1286648/
<rvba> I'm talking about the horrible nesting you mentioned.
<jam> rvba: that is *really* hard to do, because the actual 'I_want_to_ovrride_this' doesn't exist until after the def runs.
<rvba> Good pointâ¦ what a mess.
<jtv1> Oops.  Filesystem corruption again.  :(
<jtv1> Better reboot & manually fsck.
<rbasak> rvba: have you seen this before? http://paste.ubuntu.com/1286685/
<rbasak> Nothing has changed that I'm aware of so I presume I've inadvertently changed something as it worked before
<rbasak> locale.getdefaultlocale() returns None
<rvba> rbasak: o_O, never seen that before.
<rbasak> Ah
<rbasak> The preseed I'm using might have changed yesterday
<rbasak> (on the MAAS server)
<rbasak> It's now "C"
<mgz> that's some bad code in django...
<rbasak> "LANG=en_GB.UTF-8 maas createsuperuser" works
<mgz> the locale really should be C.utf8 though
<rbasak> https://code.djangoproject.com/ticket/16017
<ubot5> Django bug 16017 in Core (Management commands) "createsuperuser fails if Python can't detect default locale" [Normal,Closed]
<rbasak> Looks like it's not just me
<jam> rvba: do you know the speed of updating the dailybuilds? If I land something now, is it likely to be in the build tomorrow? or would I have to wait one more day?
<mgz> today is a rather unusual day...
<mgz> probably in a good way though, most of the q work is done and r is not yet open
<rvba> jam: I think it will be there tomorrow.  But apparently, you can request the build to happen manually.
<rvba> I just tried it, it seems to work.
<rvba> (I usually use my own ppa)
<rvba> jam: there is a 'request build' button on https://code.launchpad.net/~julian-edwards/+recipe/maas-daily.
<rvba> If you're not allowed to do that (?) I can trigger the build for you.
<jam> k
<mgz> hm, jam it seems your gzip branch is missing the actual middleware?
<jam> mgz: did that not get pushed?
<jam> hmm...
<jam> mgz: src/maas/settings.py should have django.middleware.gzip.GzipMiddleware
<jam> I see it here.
<mgz> right, but jenkins complains that it doesn't exist
<mgz> might be some djangoy weirdness?
<jam> mgz: no, capitalization typo
<jam> the name is GzipFile, but GZipMiddleware
<jam> I tend not to run 'make test' each time, I should do it more often.
<jam> mgz: thanks for noticing. Also, as I am likely to not be around much tomorrow, can you keep an eye on dimitern, I'd hate for him to run amok on his first week :)
<dimitern> :)
<mgz> :)
<jam> mgz: we've also been missing you on mumble today, but on Monday, we'll all see eachother in person anyway.
<mgz> yeah, not the best environment for mumbling this unfortunately
<mgz> I'm not around til tuesday evening, but am looking forward to it.
<mgz> I think it's only managery things the first two days?
<rvba> I think so, I'll arrive on Tuesday evening as well.
<matsubara> mgz, that worked, thanks! :-)
<mgz> :)
<dimitern> I pushed my first change! :) can I bug someone to review it please?
<dimitern> mgz: :) ^
<mgz> having a look
<dimitern> only for the docstring I'm not sure what exactly to describe... not aware of the full process
<allenap> mgz: Ah, sorry, I beat you to it again :-/
<mgz> allenap: no problem :)
<rbasak> I'm concerned about bug 1068086. Can anyone reproduce on amd64?
<ubot5> Launchpad bug 1068086 in MAAS "juju fails to deploy with MAAS by default" [Undecided,New] https://launchpad.net/bugs/1068086
 * rbasak notes the irony in the last four digits of that bug number
<mgz> rbasak: I've seen something along those lines before, but am failing to search up recollection
<mgz> rbasak: need to go now, but will try to dig into it tomorrow if none of the americans get there first
<rbasak> thanks mgz
#maas 2012-10-19
<ThiagoCMC> What should I type at "DNS master zone" in "Edit Cluster Controller"? My domain?
<ThiagoCMC> I'm seeing "master" there...
<ThiagoCMC> Guys... I'm facing this BUG: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1067929 - What do you guys recommends?
<ubot5> Ubuntu bug 1067929 in maas (Ubuntu) "maas-region-celeryd connects to the wrong queue." [Critical,Fix committed]
<ThiagoCMC> Not fixed for me.
<ThiagoCMC> So, the BUG 1067929 want's fixed. How can I use MaaS from Ubuntu 12.10?! Is it working smoothly regardless this problem?!
<ubot5> Launchpad bug 1067929 in maas (Ubuntu) "maas-region-celeryd connects to the wrong queue." [Critical,Fix committed] https://launchpad.net/bugs/1067929
<ThiagoCMC> shut up bot!
<ThiagoCMC> :-P
<ThiagoCMC> bigjools, can you help me?! I'm facing this BUG: https://launchpad.net/bugs/1067929
<ubot5> Ubuntu bug 1067929 in maas (Ubuntu) "maas-region-celeryd connects to the wrong queue." [Critical,Fix committed]
<ThiagoCMC> That message does not disappear...
<ThiagoCMC> Is MaaS working even with that message?
<bigjools> ThiagoCMC: there is a workaround for that bug
<bigjools> but I see you already saw and decided to change the bug status ...
<ThiagoCMC> It works! I edited that file right after Ubuntu finish the installation, by entering under /target/usr/sbin and editing it before first reboot.
<ThiagoCMC> Message is gone...
<ThiagoCMC> YAY!
<bigjools> how are you getting on otherwise?
<ThiagoCMC> I think that, this file must be edited before running imports-pex etc...
<ThiagoCMC> Also, under "Cluster Edit", what should I type in "DNS zone name"? Is filled with "master" there... It is my "domain.com", right?
<bigjools> editing it any time and restarting the service is enough
<ThiagoCMC> Not for me...
<ThiagoCMC> :(
<bigjools> well something else was going wrong then because I know for a fact what is broken :)
<ThiagoCMC> I tried a lot this... wait 5, 10 minutes... nothing... anyway, it is fine now, I think...
<ThiagoCMC> no problem...
<bigjools> the job only runs every 15 minutes
<bigjools> IIRC
<ThiagoCMC> :-P
<ThiagoCMC> About "DNS zone name", is my domain that I should type there... ?
<bigjools> the zone name can be anything you like - if you are managing DNS with MAAS then set it to a zone name you want to use locally, otherwise set it to an another zone which is able to provide hostnames for your nodes
<ThiagoCMC> ok
<ThiagoCMC> tks
<bigjools> because juju will look up the exact name
<ThiagoCMC> Mmmm...
<ThiagoCMC> Nice info!
<ThiagoCMC> BTW, launchpad.net must have some kind of "undo", because I accidentally change the "status" of it...  :-/
<bigjools-afk> ThiagoCMC: ubuntu's policy is to stop that happening so people can't re-open bug.  the problem is that it lets anyone mark it fixed :(
<shang> has anyone seen the duplicate mac address error in the 12.04 maas? https://pastebin.canonical.com/76900/
<shang> I tried to delete the node in cobbler, but the error still there
<ThiagoCMC> bigjools-afk, yep, that is a problem...  Sorry about this mistake...
<bigjools-afk> shang: deleting things in cobbler directly will completely hose things as its DB gets out of sync with MAAS's
<shang> bigjools-afk: oops
<bigjools-afk> shang: also not many people can see paste.canonical.com :)
<ThiagoCMC> bigjools-afk, is it possible/easy to use my own preseed?! I need my nodes with RAID+LVM but, MaaS just do the "Guided Partitioning" without asking...   :-/
<bigjools-afk> ThiagoCMC: yes, there are preseed files you can edit
<ThiagoCMC> awesome!
<bigjools-afk> I think they are in /usr/share/maas somewhere
<ThiagoCMC> okay...
<shang> sorry, re-paste: http://pastebin.ubuntu.com/1288574/
<bigjools-afk> shang: it's ok I can see it of course :)
<shang> bigjools-afk: that's why I am trying to do the same on maas side too
<ThiagoCMC> Is there any plans to pre-setup the HD layouts through MaaS Web Interface?
<shang> bigjools-afk: yeah, just for the sake of everyone else :)
<bigjools> ThiagoCMC: no plans
<bigjools> patches welcome, as they say :)
<ThiagoCMC> =P
<bigjools> shang: ummm your best bet is to delete all the data in both cobbler and maas. you'll need to use the shell interface to do it in maas as the web ui will complain every time now
<bigjools> shang: or update to 12.10 :)
<ThiagoCMC> Well, if it was manual, I would be happy.
<bigjools> cobbler is a dead dodo in 12.10
<shang> bigjools: I actually just done that. maas flush, delete all the nodes from cobbler system list
<shang> bigjools: do you recommend the newer version of the maas?
<bigjools> shang: what does cobbler's log say?
<ThiagoCMC> After much work, I'm afraid that I'll not be able to use MaaS because of this lack of "manual" node disk partition...   =(
<bigjools> shang: hell yes, it's a lot better
<ThiagoCMC> Anyway, I'll look for the presseds before quit
<bigjools> shang: we're in the middle of backporting some of the 12.10 stuff to a 12.04 PPA so it will also be without cobbler
<bigjools> ThiagoCMC: you can certainly edit partitions via the preseed
<ThiagoCMC> ok
<shang> bigjools: nice!
<shang> bigjools: any ETA ? :)
<ThiagoCMC> I know that, but I prefer manual because I already know how to do that...  ehhehe
 * shang cannot wait
<bigjools> ThiagoCMC: look for a preseed_master file
<ThiagoCMC> ok
<bigjools> shang: hope to get it done in a month or less
<ThiagoCMC> BTW, I really appreciate your help!
<bigjools> shang: uds will interfere :)
<bigjools> ThiagoCMC: my pleasure
<shang> bigjools: right.
<shang> bigjools: any other way that we can sync both system? cobbler log didn't say much: http://pastebin.ubuntu.com/1288589/
<shang> bigjools: with the newer maas, is there going to be an easier way to track done which node is which?
<bigjools> shang: when you try to delete and get the maas stack track about the "uknown cobbler error" there should be a corresponding error in the cobbler log
<bigjools> shang: why is it hard now?
<shang> it is node+<random string> afterwards
<shang> bigjools: or maybe I miss something?
<bigjools> shang: ah ok - it will be based on MAC unless we also know the IP in which case it will be like an IP but with - instead of .
<shang> bigjools: I always get the string like: node-595f0016-199d-11e2-98c5-525400c570c8, I can;t really tell where the mac or the ip sits in the string tho
<ThiagoCMC> Can I ignore this: /var/log/apache2/error.log: [alert] (2)No such file or directory: mod_wsgi (pid=1428): Unable to change working directory to '/home/maas'. ?
<bigjools-afk> ThiagoCMC: yes
<ThiagoCMC> OK
<bigjools-afk> shang: 12.10 maas uses macs, and you can also rename the nodes if you want
<shang> bigjools: ok
<ThiagoCMC> Huston, we have a problem!
<ThiagoCMC> http://pastebin.com/E5G5rtN3 <- maas.maasserver #...# Exception: No user data available for this node.
<ThiagoCMC> another problem... At the MaaS interface, I'm seeing: 192-168-50-2.teste.com but, the IP allocated to it is NOT 192.168.50.2, is 192.168.50.1, according to dhcp log.
<ThiagoCMC> and my arp table says 192.168.50.1 too... for that MAC...
<ThiagoCMC> So, MaaS is not ready for production... right?! On every step, I saw errors and bugs...    =(
<ThiagoCMC> I would LOVE to use this but... I'm feeling now, after a week of work, that it is not ready yet...   snif...
<jtv> ThiagoCMC: did you enable dhcp management (and install maas-dhcp)?
<ThiagoCMC> of course...
<ThiagoCMC> and maas-dns
<ThiagoCMC> Something is wrong.
<bigjools> ThiagoCMC: the host name gets set only once
<bigjools> if the IP changes then the host name does not change
<ThiagoCMC> But what changes it?
<bigjools> this release of maas is production ready
<ThiagoCMC> I wans't me.
<bigjools> DHCP
<bigjools> the hostname is irrelevant in a cloud environment, really
<jtv> But we do need dhcp.  We fish the IP address out of the dhcp leases file.
<jtv> It's not simply two machines, one with the ".1" address and the other with ".2"?
<ThiagoCMC> When I ping the FQDN that I'm seeing at MaaS interface, it does not respond.
<ThiagoCMC> It is wrong...
<ThiagoCMC> =(
<bigjools> hang on
<bigjools> what does it resolv to?
<bigjools> reso;lve
<bigjools> bah
<ThiagoCMC> 192-168-50-2.teste.com resolv to: 192.168.50.2.
<ThiagoCMC> But the IP is .1 now...
<ThiagoCMC> after installation...
<ThiagoCMC> I'm trying to install two nodes at the same time... I'll format my MaaS again... And to one node at a time... All from scratch...
<ThiagoCMC> I think that MaaS with DHCP and DNS management enabled, must have some kind of "glue" between DHCP and DNS... Don't you think?!
<jtv> Yes, but it's all one-way from the DHCP code to the DNS code.
<jtv> That's also why you can enable DHCP management without DNS management, but not the other way around.
<ThiagoCMC> Sounds logic...
<ThiagoCMC> Anyway, it is wrong now.
<ThiagoCMC> hehehhe
<ThiagoCMC> :-P
<jtv> So... it's the same MAC address that was first 192.168.50.1 and later 192.168.50.2?
<jtv> The code should deal just fine with 2 IP addresses pointing to the same MAC, actually.
<ThiagoCMC> ok...
<ThiagoCMC> Well...
<ThiagoCMC> I'll start from scratch again...
<ThiagoCMC> One node at a time...
<jtv> Baby steps...  Good idea.
<bigjools> ThiagoCMC: check the /var/lib/maas/dhcp/dhcpd.leases
<bigjools> and verify that the MACs for your nodes match up with what you think the IP is
<ThiagoCMC> 1 sec...
<bigjools> MAAS keeps track of that file to see what the IPs are
<jtv> It may also be worthwhile for us to have a look at the current sections in the leases file that pertain to this host.
<jtv> Unexpected syntax, unexpected time, that sort of thing.
<jtv> ThiagoCMC: don't be surprised if the start/end times on the dhcp lease are weird -- they should be in UTC and I believe your timezone is far removed from UTC.
<jtv> (Faint memory, so don't freak out if I got that wrong :)
<bigjools> they are utc
<jtv> *If* things are set up right, and we know something isn't right.  :)
<jtv> Oh, I see now -- I was referring to my uncertainty w.r.t. Thiago's timezone
<jtv> I'm going to try and fix up a system problem now.  Should be back in a few minutes.
<ThiagoCMC> BTW, my second node 192-168-50-4.teste.com has the IP 192.168.1.3... lol
<ThiagoCMC> let me check my leases...
<ThiagoCMC> My MAC address (of .2 which is in fact .1), appear more than 1 time within it...
<ThiagoCMC> look: http://pastebin.com/8RYjGe5D
<bigjools> ThiagoCMC: it's normal for them to appear multiple times
<ThiagoCMC> ok
<ThiagoCMC> no problem with that...
<ThiagoCMC> bigjools, have you read my pastebin?
<ThiagoCMC> You saw there?! It start with .1... becomes .2 (and registered at DNS as .2)... and in the end of the day, if was fixed in DHCP as .1.
<bigjools-afk> ThiagoCMC: did you get my last message? "that's the only one maas reconciled with a known node"
<bigjools-afk> I'm on a very flaky 3g connection
<ThiagoCMC> =)
<ThiagoCMC> bigjools, have you read my pastebin? You saw there?! It start with .1... becomes .2 (and registered at DNS as .2)... and in the end of the day, if was fixed in DHCP as .1.
<ThiagoCMC> sorry, don't know if you receive it too...
<bigjools-afk> <bigjools> you'll notice the host block
<bigjools-afk> <bigjools> maas sets that up to force the same IP on future leases
<bigjools-afk> <bigjools> so your 52:54:00:cd:6b:a2 is the .1
<bigjools-afk> <bigjools> that's the only one maas reconciled with a known node
<bigjools-afk> I saw the paste
<bigjools-afk> I see the .2 for some of the leases - I wonder wtf DHCPD is doing allocating a new IP when there's an existing lease ...
<bigjools-afk> very strange
<bigjools-afk> I've never seen this behaviour before and I've enlisted stuff a LOT
<ThiagoCMC> You don't know my friend... I am the Master of the Bugs!! ahhaha
<dimitern> allenap: hey I pushed a fix for the failing test, can you have a look pls?
<bigjools-afk> ThiagoCMC: well let us know how you get on with starting from scratch.  Can you join and email maas-devel@lists.launchpad.net
<ThiagoCMC> I pull milk from stone ... lol
<ThiagoCMC> bigjools-afk, yes!
<bigjools-afk> I have to go shortly
<ThiagoCMC> I'll join there...
<ThiagoCMC> I'm staring again now... I'll let you know...
<ThiagoCMC> Tks again bro!
<bigjools-afk> great
<bigjools-afk> no problem - out of interest, where are you based?
<ThiagoCMC> Sao Paulo, Brazil...
<ThiagoCMC> I'm building a public cloud provider...
<bigjools-afk> nice
<bigjools-afk> I know someone else there :)
<ThiagoCMC> Cool!
<bigjools-afk> you're either up very late or very early!
<ThiagoCMC> hehehe
<ThiagoCMC> It is 5 AM here...
<ThiagoCMC> Can't sleep...
<bigjools-afk> !
<bigjools-afk> ThiagoCMC: well I'd love to help you more - let us know how you get on on the list
<ThiagoCMC> Also, sorry about my english... It is hard to me to understand you guys sometimes...
<ThiagoCMC> ok
<bigjools-afk> not a problem, I am very used to Portuglish :)
<ThiagoCMC> ihhihihihi
<ThiagoCMC> ^^
<jtv> dimitern: https://plus.google.com/hangouts/_/fc64b594dda8f0525e45e08072fe34ceaa325cc0?hcb=2&hii=113299378710739086834&hit=1350635423981&htp=0&hmv=0
<rvba> allenap: care to review that branch? https://code.launchpad.net/~rvb/maas/bulk-load/+merge/130535
<allenap> rvba: Yes indeed!
#maas 2012-10-21
<ThiagoCMC> Hi! Can I ignore the message "The region controller does not know whether any boot images have been imported yet. If this message does not disappear in 5 minutes..." ???
<ThiagoCMC> That message does not disappear even after following this instructions: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1067929
<ubot5> Ubuntu bug 1067929 in maas (Ubuntu) "maas-region-celeryd connects to the wrong queue." [Critical,Fix released]
<ThiagoCMC> Even after 1 hour... That message is still here...  :-@
<ThiagoCMC> =(
<ThiagoCMC> I can add and use new nodes, so, I think I can ignore that message... Right?!
<ThiagoCMC> What this measn: "maas.maasserver ## Exception: No user data available for this node." "MAASAPINotFound: No user data available for this node."... This message appear in my maas.log right after first node's boot (allocated to root)....
<ThiagoCMC> creating my second node: "dhcpd: Can't create new lease file: Permission denied" on syslog.
<ThiagoCMC> "zone teste.com/IN: loading from master file /etc/bind/maas/zone.teste.com failed: CNAME and other data"...
<ThiagoCMC> "zone quilombas.com/IN: not loaded due to errors."...
<ThiagoCMC> tsc...tsc...
<ThiagoCMC> ops
<ThiagoCMC> not quilombas.com, teste.com (I forgot to edit before submit to IRC)...  ehheh
<ThiagoCMC> damn
<ThiagoCMC> When I click "Add node", it does not begin registered at my DNS... When I let the machine boot over DHCP/PXE and appear automatically in MaaS, the DNS seems to work ok.
<ThiagoCMC> s/begin/being/
<ThiagoCMC> Guys, MaaS DNS subsystem is broken: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1069535
<ubot5> Ubuntu bug 1069535 in maas (Ubuntu) "MaaS DNS - named, not loaded due to errors." [Undecided,New]
#maas 2013-10-14
<roaksoax> bigjools: thanks!
<bigjools> back at ya
<Azendale> Is there anywhere to set what targets to get ephemeral images for (in my case I only want saucy and amd64 and i386) when I run maas-import-pxe-files?
<bigjools> jtv: have you ever seen the wsgi wrapper failing to start up, complaining about "Connection reset by peer" ?
<bigjools> it's trying to run refresh_nodegroup so I suspect rabbitmq is being crap
<bigjools> rvba: I have spent all freaking day chasing trivial problems so I can get to examine this bug of yours on my home server
<bigjools> *still* have not fixed all the problems
<rvba> bigjools: trivial problems in MAAS or juju-core?
<bigjools> for some reason, the juju binary on my maas test server is producing a totally bogus Auth header, so maas kicks everything out with a 401
<bigjools> this is my last blocker
<bigjools> same binary works on canonistack and elsewhere
<bigjools> so.... I am at a loss
<rvba> humâ¦
<rvba> I'd draw my McGyver kit and log things inside piston's code to get to the bottom of the errorâ¦
<bigjools> it;s not maas
<bigjools> it's juju
<bigjools> the auth header is wrong
<rvba> Right, but in what way exactly?
<bigjools> https://bugs.launchpad.net/juju-core/+bug/1239496
<ubot5> Ubuntu bug 1239496 in juju-core "1.16 juju and above has oauth errors talking to maas" [Undecided,New]
<bigjools> that way
<rvba> Very odd that I didn't see that problem at all in the labâ¦
<bigjools> only happens on my home server
<rvba> The only obvious difference is realm="MAAS+API" (sent by juju).
<bigjools> not so
<bigjools> check the token
<bigjools> and signature
<rvba> Well, they differ but that's normal.
<rvba> Their size are similar.
<rvba> That's the only check that my naked eye can do.
<bigjools> rvba: the difference is not normal
<bigjools> I compared headers from maas-cli and an old juju and they were almost identical
<bigjools> these ones are vastly different
<rvba> The timestamp and the nonce are used to compute the signature.
<rvba> "almost identical"?
<bigjools> signature was different but token was the same
<bigjools> in this example token is not the same
<rvba> Hum, you're right, the token should be the sameâ¦
<rvba> Did you inspect the DB to see what the tokens are in there?
<bigjools> no
<bigjools> rvba: having said that, I just got one to work and the token sent is different
<rvba> If your juju conf hasn't changed, that is weird.  The token should be dictated by your api key.
<bigjools> rvba: I see different tokens on different requests
<rvba> Maybe I'm missing something thenâ¦
<bigjools> rvba: my bad - it was a celery request I had in there ...
<bigjools> see the last comment on the bug
<rvba> bigjools: repliedâ¦
<bigjools> rvba: see #maas
<rvba> bigjools: you're definitely right, the whole point of having different API keys that can be changed is to accommodate the case where the credentials are compromised.
<bigjools> rvba: people are also using them for different juju environments
<bigjools> hence my other bug
<bigjools> but we need to talk about that scenario anyway
<bigjools> rvba: I am fast-installing one of my microservers - so far 15 minutes later nothing going on
<bigjools> I guess it's broken - possibly with this bug
<rvba> bigjools: not normal, I *just* used it in the lab.
<bigjools> the --commission thing I suspect
<bigjools> but can't get in to the node to find out ...
<rvba> --constraints you meanâ¦?
<bigjools> uhhhh yes :)
<rvba> Try sshing into the allocated node.
<rvba> The node should be up and running.
<rvba> bigjools: check /var/log/cloud-init-output.log on that nodeâ¦ the last line should be the invocation of jujud with the empty --constraints.
<bigjools> too late
<rvba> bigjools: one workaround is to explicitly use constraints when bootstrapping.
<bigjools> anyway juju status was dead
<rvba> Another is to build from source.
<rvba> It's normal, jujud is not up.
<rvba> So juju status can't work.
<rvba> But again, the bootstrap node should be up and running.
<bigjools> soooooo, juju bootstrap --upload-tools seems to be broken
<rvba> Only the juju side of things crashed.
<rvba> ??
<rvba> Well, if all the calls to the API are broken on your machine I'm not surprised.
<bigjools> that's working now
<rvba> Ah.
<rvba> Then run juju bootstrap --upload-tools with "-v" to see what's going on.
<bigjools> I did
<bigjools> http://paste.ubuntu.com/6234893/
<rvba> HAha
<rvba> The dreaded EOF error.
<rvba> Destroy and re-bootstrap.
<bigjools> you know of this error
<rvba> I've seen if from time to time unfortunately.
<bigjools> same again
<bigjools> happening every time
<rvba> Weird.
<gmb> allenap: When you've got a sec, can you take another look at https://code.launchpad.net/~gmb/maas/fix-bug-1184589/+merge/190601 for me?
<allenap> gmb: Certainly can.
<gmb> allenap: Although it hasn't pushed yet...
<gmb> Bear with me; sudden LP FUBAR
<gmb> Wups
 * gmb kills the wget that was eating my bandwidth
<rvba> bigjools: fwiw, allenap and I talked about the bug you were referring to in the call, here is a summary of the two options we talked about (one of which is using tags, the other one basically boils down to using the API key itself as you suggest): http://pad.ubuntu.com/DnNONX6kFB
<rvba> bigjools: well, more precisely, we didn't think of using the API key but to use the environment's UUID to flag the nodes.
<rvba> bigjools: so your idea of using the API key is a third option.
<bigjools> ok
<bigjools> let's talk pros and cons
<rvba> bigjools: well, the first thing to do is to list what it means doing firstâ¦ because I'm still a bit fuzzy on thatâ¦
<rvba> allenap: care to review: https://code.launchpad.net/~rvb/maas/change-freq/+merge/190894 ?
<allenap> rvba: Sure.
<allenap> rvba: Done!
<rvba> Thanks.
<allenap> gmb: Sorry, I wrote the updated review for your wrapper branch then forgot to submit when the team call started.
<allenap> bigjools, rvba: Fwiw, I don't think we should use API key, because the ability to share an environment between people but using different API keys will go away. Perhaps a hash based on admin-secret. Something that comes from Juju anyway.
<rvba> bigjools: as I said before, I agree with allenap.  The ability to use a different key without losing one's environment seems pretty important to me.
<rvba> bigjools: what allenap suggest is the option 2 in http://pad.ubuntu.com/DnNONX6kFB.
<rvba> We discarded it at the time in favor of the tag solution because solution 1 seemed simpler and it didn't imply making changes in MAAS for the solve purpose of accommodating juju.
<lifeless> MAAS kind of exists to support juju though, right?
<bigjools> rvba, allenap: +1
<allenap> rvba: I guess the tag is interesting because it does expose to others those nodes that are part of an environment.
<allenap> lifeless: Shhhh ;)
<rvba> lifeless: true, but it is better to let juju use generic stuff (tags) as opposed to doing thing juju-specific in juju.
<rvba> in MAAS*
<allenap> lifeless: You're right though. We're just trying to be constructively lazy.
<bladernr_> is there an armhf port of maas and dependencies?  I'm (just for fun) trying to install MAAS on an arm system to use as a region and cluster controller.  This is just a crazy idea I had that is fraught with peril, but I thought I'd try for the lulz.
<roaksoax> rvba: howdy!!
<roaksoax> rvba: so I see there are more fixes than last week
<roaksoax> rvba: are you looking to SRU those too?
<rbasak> bladernr_: yes, there is. No specific port needed. It Just Works. Additionally, there is support for some armhf server hardware.
<rbasak> (ie. to bootstrap armhf server machines to use as nodes)
<rbasak> To use MAAS as a region or cluster controller on armhf should Just Work, providing of course that Ubuntu is supported on that particular system. Please file bugs appropriately.
<bladernr_> rbasak: so the rub is that my secret project is putting maas, maas-region-controller and maas-cluster-controller on a raspberry piâ¦ it's currently running a debian port (Raspbian) since Ubuntu doesn't support armv6.
<bladernr_> so I'm trying to run maas in a non-ubuntu environment
 * bladernr_ hides
<rbasak> That's not supported of course, as you already understand. You may have some issues with dependencies and perhaps have to port the occasional Ubuntu patch. Perhaps difficult depending on the relevant experience for the person doing it, but tractable IMHO.
<rbasak> Really what you'd be doing is porting MAAS to Debian, rather than doing anything ARM-specific I think.
<bladernr_> hrmmm.. ok, that gives me an ideaâ¦ thanks.
<roaksoax> allenap: around?
<allenap> roaksoax: What's up?
<bigjools> roaksoax: https://bugs.launchpad.net/bugs/1239758
<ubot5> Ubuntu bug 1239758 in MAAS "isc-dhcp-client needs to be re installed after installing maas" [Undecided,Confirmed]
<bigjools> odd....
#maas 2013-10-15
<bigjools> gmb: I noticed your branch kept getting rejected with failures yesterday, did you figure it out?
<gmb> bigjools: Yeah, I hadn't fixed all the errors in the tests after my changes to how we shell out to external processes. It should be fixed now (or at least I don't get any errors on my machine).
<bigjools> cool
<bigjools> at least we get the test output in the MP comment now, previously it meant diving into a jenkins console
<rvba> bigjools: thanks for the review on https://code.launchpad.net/~rvb/maas/delete-lease/+merge/190969.  I'm wondering where exactly you want that comment added as the only code that this branch adds is already commentedâ¦
<bigjools> rvba: at the site where the node is deleted
<rvba> bigjools: in the test you mean?
<bigjools> rvba: in the Node's deletion method
<bigjools> well, maybe MAC actually
<bigjools> since that's what the signal is connected to
<bigjools> my problem with signals is that they are not obvious
<rvba> bigjools: this is inside Django's code :)
<bigjools> so a comment helps jog one's memories
 * bigjools finds the exact file and line
<rvba> :)
<bigjools> rvba: models/node.py at line 646
<rvba> bigjools: it would be really weird to add a comment there I thinkâ¦
<bigjools> all it would say is that a signal will fire that also deletes leases
<bigjools> and ref the signal method
<bigjools> the only way I found this bug in the first place was by looking at that code
<bigjools> that code has not changed!  So this is all a bit of Django magic.
<rvba> All right, as you like it :)
<bigjools> ta :)
<bigjools> rvba: which doc?
<rvba> bigjools: http://pad.ubuntu.com/DnNONX6kFB
<bigjools> rvba: ah the PAD :)
<rvba> Yes, the *PAD* :)
<rvba> allenap: gmb: my 3 branches are up for review: https://code.launchpad.net/~rvb/maas/agent-bug-1239488/+merge/191140 https://code.launchpad.net/~rvb/maas/agent-bug-1239488-acquire/+merge/191141 https://code.launchpad.net/~rvb/maas/agent-bug-1239488-list/+merge/191142
<allenap> rvba: Okay, I'll take a look once I've remembered Go.
<rvba> allenap: all right :)
<rvba> Thanks.
<allenap> rvba: Would you be able to have a look at https://codereview.appspot.com/14696043?
<rvba> allenap: approved.
<allenap> rvba: Thanks!
<roaksoax> bigjools: bug #1240215
<ubot5> bug 1240215 in MAAS "changing the default arches in import_pxe_files prevents maas-import-ephemerals from running" [Undecided,New] https://launchpad.net/bugs/1240215
<roaksoax> jt^^
<smoser> roaksoax, you're aware that "arches" are i386/generic
<smoser> right?
<smoser> not i386
<roaksoax> smoser: yep
<smoser> roaksoax, i think you might be hitting bug : should not insert version if all items were filte
<smoser> bug 1238227
<ubot5> bug 1238227 in simplestreams (Ubuntu) "should not insert version if all items were filtered" [Medium,Confirmed] https://launchpad.net/bugs/1238227
<roaksoax> smoser: thanks for looking at it btw
<smoser> i noticed that annoying behavior last week when i was doing probably the same thing you were doing.
<bigjools> roaksoax: yay .... :/
<roaksoax> bigjools: it is release team decision whrther to accept fizes though
<bigjools> yeah
<lifeless> bigjools: so how fast is the fastpath install?
<bigjools> lifeless: the actuall installer is blinding, about 2 mins
<lifeless> cool
<bigjools> but you have reboots and cloud-init overhead on that
<lifeless> yeah
<lifeless> thats disk-image fast :>... I suspect it is disk image based, no?
<bigjools> yes
<bigjools> makes my life a little nicer doing the development dance
#maas 2013-10-16
 * bigjools waves at roaksoax
<roaksoax> bigjools: here
<bigjools> roaksoax: howdy
<roaksoax> bigjools: howdy!
<bigjools> just seeing where we're at with that filtering bug on imports?
<roaksoax> bigjools: huh?
<bigjools> https://launchpad.net/bugs/1240215
<ubot5> Ubuntu bug 1240215 in MAAS "changing the default arches in import_pxe_files prevents maas-import-ephemerals from running" [Critical,Confirmed]
<bigjools> I wasn't sure if it needed more work or if the simplstreams library fix helped
<roaksoax> bigjools: well I was thinking it might be easy to fix provided that we split('/')
<roaksoax> bigjools: no I don't think it is the simplestreams stuff
<roaksoax> smoser: ^^
<bigjools> ok
<bigjools> we're all deep fixing the multi-environ thing at the moment
<roaksoax> bigjools: ok i'll see what i can do with the ephemerals stuff
<roaksoax> bigjools: this fixes it: http://paste.ubuntu.com/6243363/
<bigjools> roaksoax: nice.  Add a test and I'll approve it.
<smoser> roaksoax, i replied to that MP
<smoser> but you also have to consider '/highbank' might be in there.
<smoser> you might as well do somethign like:
<roaksoax> smoser: there's something further down for highbank
<smoser> i dont thinkt hat covers it
<smoser> ' '.join([f.split("/")[0] for f in legacy_config.get('ARCHES', '').split()])
<smoser> something like that maybe
<roaksoax> smoser: yeah that works! thakns!
<roaksoax> bigjools: https://code.launchpad.net/~andreserl/maas/fix_lp1240215/+merge/191327
<roaksoax> smoser: ^^
<roaksoax> bigjools: all yours. Im gonna patch the mpackage and upload
<bigjools> roaksoax: looks ok I'll approve
<bigjools> roaksoax: there's another fix landed recently to support multi-environs
<bigjools> can you take that too?
<roaksoax> bigjools: what's the rev?
<bigjools> fixes
<roaksoax> bigjools: rev's and bug number please :)
<bigjools> roaksoax: see branches attached to https://bugs.launchpad.net/maas/+bug/1239488
<ubot5> Ubuntu bug 1239488 in MAAS "Juju api client cannot distinguish between environments" [Critical,In progress]
<bigjools> r1708, 1709, 1710 basically
<roaksoax> bigjools: are all of those the actual fixes?
<roaksoax> bigjools: i mean
<roaksoax> final fixes?
<roaksoax> no more fixes for that bug?
<bigjools> roaksoax: I think so - the rest of the fixes are in juju
<roaksoax> ok cool
<bigjools> roaksoax: also see 1707, bug 1228085
<ubot5> bug 1228085 in MAAS "The commissioning script 00-maas-03-install-lldpd outputs to stderr." [Critical,Triaged] https://launchpad.net/bugs/1228085
<bigjools> possibly bug 1190986
<ubot5> bug 1190986 in MAAS "ERROR Nonce already used" [High,Fix committed] https://launchpad.net/bugs/1190986
<bigjools> not *critical* though
<roaksoax> bigjools: yeah.. that's the thing...
<bigjools> bug 1238376
<ubot5> bug 1238376 in maas (Ubuntu) "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Critical,Confirmed] https://launchpad.net/bugs/1238376
<bigjools> you got that one I think
<roaksoax> bigjools: yeah so only the critical bugs will get in
<roaksoax> there's 4 fixes, already
<roaksoax> and the multienvironment
<roaksoax> is 3 patches
<bigjools> yep
<roaksoax> not sure that the release team will be happy about that
<roaksoax> smoser: ^^
<bigjools> you can merge three into one
<bigjools> they're consecutive
<roaksoax> bigjools: yeah
<bigjools> I can't see why they would be unhappy to take critical fixes
<bigjools> I'd be fucking ecstatic
<roaksoax> bigjools: one day before release... whgen the ISO isprobably read already?
<roaksoax> bigjools: which would cause an ISO re-spin
<roaksoax> i don't think they'll be happy
<bigjools> it's not on the iso though is it?
<bigjools> oh server iso
<bigjools> ok
<bigjools> 0-day fix it is :)
<roaksoax> bigjools: yeah, so the ephemeral stuff is important for the ISO
<bigjools> ok
<bigjools> well apparently the cloud-installer would take much longer without this fix
<roaksoax> bigjools: this is what I have so far: http://paste.ubuntu.com/6243516/
<bigjools> roaksoax: and you need the multi-env one
<roaksoax> yep
<bigjools> ~400 lines though :/
<roaksoax> bigjools: ahh
<roaksoax> bigjools: then release team *wont* be happy
<bigjools> dunno why they would get unhappy about the size, it is either critical or it isn't
<roaksoax> bigjools: yeah i think i know how to approach taht one
<roaksoax> bigjools: let me know when you approve the ephemeral fix plz
<bigjools> done
<roaksoax> bigjools: the fixes for juju don't look so bad.. since I only take the actual patch and not the changes in the test
<bigjools> roaksoax: yeah I have one more fix coming for juju
<bigjools> working on it right now
<roaksoax> bigjools: ok
<roaksoax> alright
<roaksoax> i'll be back later then
<bigjools> thanks roaksoax
<bigjools> remind me to buy you a beer next week :)
<roaksoax> lol will do
<roaksoax> bigjools: remind me to make you drink lots of pisco :P
<bigjools> roaksoax: I can't drink at the moment :(
<roaksoax> boomer
<roaksoax> bigjools: how's the fix looking?
<bigjools> roaksoax: slow .... this is Go after all
<roaksoax> heh
<bigjools> nearly there
<bigjools> well,  nearly ish
<roaksoax> bigjools: i might have to get that package uploaded tomorrow then
<roaksoax> cause I'm about to die
<roaksoax> unless we do sru
<bigjools> roaksoax: yeah it won't get done in the next hour :/
<roaksoax> bigjools: so my take is this, I can upload a new maas package with the fixes, and tomorrow, i make another upload which willk most likely be as a 0day sru
<roaksoax> unless they can process it
<bigjools> roaksoax: ok
<bigjools> that's the best we can hope for then
<roaksoax> bigjools: yeah cause eitherway I don't think they will process the upload today
<roaksoax> bigjools: but at least I ensure that the fixes up until now are uplaoded
<roaksoax> and easier to negotiate on getting them released
<bigjools> roaksoax: great
<bigjools> roaksoax: my juju change landed on trunk
<gnuoy> I'm doing a fresh deployment with juju on a MaaS cluster. I can bootstrap the juju env fine using a tag to specify the bootstrap node but when I try and deploy a charm I don;t see any physical servers getting allocated in the maas UI and after a minute os so juju reports "error: cannot run instances: gomaasapi: got error back from
<gnuoy>       server: 409 CONFLICT" as the agent state info for the new machine
<gnuoy> I have 16 servers in the Ready state using juju-core  1.16 and maas 1.2+bzr1373+dfsg-0ubuntu1~12.04.2
<gnuoy> Not sure if this is relevant but all the machine have a manual tag and I'm not specifying a tag constraint  to juju at the moment
<gnuoy> I'll try removing a tag and trying again
<bigjools> gnuoy: I think the tag for bootstrapping gets applied by default in juju
<gnuoy> hi bigjools, looking in the maas.log I see "NodesNotAvailable: No matching node is available". So if all machines have a tag and you try to deploy a charm without specifying tag is refusing to allocate a machine the desired behaviour ? I'm trying to think if that makes sense
<gnuoy> By not specifying a tag constraint with juju are you saying "This can go anywhere I don't care" or are you saying "This can only go to machines which haven't been preallocated with a tag"
<bigjools> gnuoy: it uses the constraints from the bootstrap node IIRC
<bigjools> so if you put a tag on it then it expects all machines to have that tag
<bigjools> did you say all the machines have that tag?
<bigjools> sorry I have to run - try asking in the juju-dev channel
<bigjools> back later
<gnuoy> bigjools, ok, thanks.
<jamespage> bigjools, still around? I need to understand which MAAS bugs are still outstanding for Saucy
<jamespage> and which ones we must land pre-release (i.e. it matters if they are on the ISO)
<jamespage> allenap, maybe you can help with that if bigjools is eod?
<bigjools> jamespage: I am EOD but I can help
<jamespage> bigjools, marvellous!
<bigjools> I did give a list to roaksoax earlier
<jamespage> yeah - I was just looking through those
<bigjools> bug 122805
<ubot5> bug 8422 in tsclient (Ubuntu) "duplicate for #122805 Error message on ending VNC session" [Medium,Won't fix] https://launchpad.net/bugs/8422
<bigjools> ummm
<bigjools> bug 1228085
<ubot5> bug 1228085 in MAAS "The commissioning script 00-maas-03-install-lldpd outputs to stderr." [Critical,Triaged] https://launchpad.net/bugs/1228085
<bigjools> bug 1238376
<ubot5> bug 1238376 in MAAS "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Critical,Fix committed] https://launchpad.net/bugs/1238376
<bigjools> bug 1239488
<ubot5> bug 1239488 in maas (Ubuntu Saucy) "Juju api client cannot distinguish between environments" [High,Triaged] https://launchpad.net/bugs/1239488
<bigjools> bug 1240215
<ubot5> bug 1240215 in MAAS "changing the default arches in import_pxe_files prevents maas-import-ephemerals from running" [Critical,Fix committed] https://launchpad.net/bugs/1240215
<jamespage> bigjools, ok so bug 1238376 got patched in
<ubot5> bug 1238376 in MAAS "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Critical,Fix committed] https://launchpad.net/bugs/1238376
<bigjools> cool
<jamespage> ditto bug 1240215
<ubot5> bug 1240215 in MAAS "changing the default arches in import_pxe_files prevents maas-import-ephemerals from running" [Critical,Fix committed] https://launchpad.net/bugs/1240215
<bigjools> I saw roaksoax made a patch for everything except 1239488
<bigjools> that's a big one
<jamespage> I think bug 1239488 and bug 1228085 are still outstanding
<ubot5> bug 1239488 in maas (Ubuntu Saucy) "Juju api client cannot distinguish between environments" [High,Triaged] https://launchpad.net/bugs/1239488
<ubot5> bug 1228085 in MAAS "The commissioning script 00-maas-03-install-lldpd outputs to stderr." [Critical,Triaged] https://launchpad.net/bugs/1228085
<jamespage> bigjools, i guess this is related to bug 1229275
<ubot5> bug 1229275 in juju-core "[maas] juju destroy-environment also destroys nodes that are not controlled by juju" [Critical,In progress] https://launchpad.net/bugs/1229275
<bigjools> rvba: is https://bugs.launchpad.net/maas/+bug/1228085 fixed?
<rvba> bigjools: yes
<bigjools> jamespage: that one is all done actually I marked the maas task as fixed
<bigjools> rvba: uh why does it say triaged?
<rvba> bigjools: someone forgot to flip the switch.
<bigjools> jamespage: sorry to be clear, bug 1239488 is all done
<ubot5> bug 1239488 in maas (Ubuntu Saucy) "Juju api client cannot distinguish between environments" [High,Triaged] https://launchpad.net/bugs/1239488
<jamespage> bigjools, landed in maas but not yet in distro right?
<bigjools> jamespage: right
<jamespage> bigjools, and there is an associated juju-core fix I need to pull for that as well I think
<bigjools> jamespage: correct- but that ones a little complicated because after Gavin and I landed branches, they wanted to change something again
<bigjools> so talk to roger peppe
<jamespage> I am
<bigjools> tip top
 * bigjools has the Norfolk lingo down pat
<bigjools> jamespage: ok anything else?
<jamespage> bigjools, I thinks thats OK for now - thanks for the help
<jamespage> (I was being asked in #ubuntu-release)
<jpds> How does one find out which tests failed for a machine in MAAS?
<jpds> roaksoax: Would you know? â
<roaksoax> jpds: the webui should show which failed
<jpds> I see a 02-maas-virtuality.
<roaksoax> jpds: 2 out of 5?
<jpds> roaksoax: Yes.
<roaksoax> jpds: so that's probably the one that failed
<roaksoax> jpds: you can get into the image and debug
<roaksoax> jpds: https://lists.launchpad.net/maas-devel/msg00808.html
<jpds> roaksoax: Is there anything in the docs about debugging?
<roaksoax> jpds: yes
<jpds> roaksoax: And got it to work, helps if I give them a proxy so they can get to the internet.
<roaksoax> jpds:  yep :)
<allenap> smoser: Ought there to be a lock around critical parts of maas-import-pxe-files too?
<smoser> allenap, i suppose probably. i have not really thought about that.
<allenap> smoser: I'll look into doing it. Do you think this is a critical bug?
<smoser> allenap, well i'm not sure how matsubara hit it
<smoser> its trivial to lock such things
<smoser> i pointed at the apparently "correct" way to do it in celery
<smoser> but you could just do it below celery with 'run-one' in ubuntu (although that is in universe).
<allenap> smoser: That's a good solution, but unfortunately it means pulling in django on the clusters, which I'd rather avoid.
<matsubara> smoser, allenap: I tried maas-import-pxe-files with run-one but it didn't work: http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-manual/ARCH=amd64,label=lenovo-RD230-01/113/console, I guess you meant running import_ephemerals.py with run-one then?
<matsubara> allenap, and the qa-lab-tests found that bug. sometimes the tests pass but more often than not it hits the checksum error
<allenap> run-one also uses the whole command line when considering locks, iirc, but we want to restrict anything in the datadir.
<allenap> matsubara: It's great they hit it. Did it take a lot of work from there to figuring it out?
<matsubara> yes
<smoser> allenap, oh. i didn't know.
<matsubara> allenap, annoyingly yes, specially because I thought initially the problem was with the proxy, so wasted a lot of time checking that squid was doing what it was supposed to be doing
<smoser> allenap, i dont know *why* they hit it ghouth.
<matsubara> allenap, and the qa-lab-tests had horrible debug output when maas-import-pxe-files fails (which is now fixed by a another branch of mine)
<matsubara> so it was painful but I think it'll turn out to improve the quality of the tests at least
<matsubara> allenap, if you have a fix, I can help test it in the qa lab
<allenap> matsubara: I'm looking at that now.
<matsubara> allenap, thank you
<adam_g> after updating MAAS to 1.4+bzr1693+dfsg-0ubuntu2~ctools,  'juju status' giving me: ERROR TLS handshake failed: x509: certificate signed by unknown authority 1.
<adam_g> any ideas?
<roaksoax> adam_g: nope, but the updates do not mess with juju sections at all.... unless related to the bson issue... but that';s weird either way
<roaksoax> bigjools: ^^
<adam_g> roaksoax, hmm must have been something unrelated. cleared ~/.juju/environments and its working again.. but now it seems IPMI stopped :|
<adam_g> roaksoax, hmm. nvm. looked like celery wasn't connected properly
<roaksoax> adam_g: hehe ok :)
<bigjools> roaksoax: did a newer version get uploaded with the patch for multi-envs?
<roaksoax> bigjools: nope
<bigjools> roaksoax: SRU then?
<roaksoax> bigjools: we determined that's not release critical
<roaksoax> bigjools: yep
<roaksoax> bigjools: i'll upload tomorrow and bug for it to get in tomorro
<bigjools> it's release day here!
<bigjools> sadly, my desktop is riddled with problems :(
<roaksoax> bigjools: he,... mine running smoothly
<bigjools> I currently have an awful unstoppable noise coming from the speaker... brb, rebooting
<bigjools> ah better
<bigjools> roaksoax: fyi http://maas.ubuntu.com/docs/changelog.html
<bigjools> argh
<bigjools> https://bugs.launchpad.net/bugs/1240570
<ubot5> Ubuntu bug 1240570 in MAAS "please add exynos5 support to ephemerals" [Undecided,New]
#maas 2013-10-17
<bigjools> jamespage:
<bigjools> ah
<bigjools> jtv: this one
<jtv> OK
<bigjools> so let's discuss the actual symptoms
<bigjools> rvba has the most experience so far
<bigjools> and we can fill out the bug report as it's one of those "report the solution" rather than "report the bug"
<jtv> I guess a broken download could easily be hidden behind a retry...
<rvba> symptoms: you run the import-pxe-files and you get: http://paste.ubuntu.com/6244329/
<jtv> Might be worth keeping the failed download, and seeing what "strings" and "cmp" can tell us.
<bigjools> rvba: ok and is the file obviously the wrong size, or just corrupt>?
<bigjools> all useful info missing from the bug :(
<jtv> Because the file itself is missing by the time we see it.
<rvba> bigjools: obviously the wrong size, very small whilst the file should be 200M or so
<bigjools> ok, so maybe it crapped out early
<bigjools> has it ever worked?
<rvba> Yes.  I worked at some point.
<rvba> It*
<rvba> Let's see if we can reproduce it in a clean environment like a fresh canonistack instance.
<bigjools> there's a start.  we need to find a scenario to recreate first then
<bigjools> I'll play there too
<rvba> Yes.
<rvba> Then we need to report a proper bug with all the relevant info.
<jtv> I guess we should also check which seems to be the correct checksum â for all we know we could be getting the wrong one from the index.
<bigjools> very very very unlikely :)
<jtv> All of this is, isn't it?
<jtv> Anyway, we should learn more as soon as we can compare files.
<bigjools> I used debtree on the maas package today.  Holy cow.
<jtv> Try to compare to other, similar applictions before judging.  :)
<bigjools> not judging just noting
<bigjools> so maas-import-ephemerals - not a sausage of debug output?
<bigjools> or any indication it's doing anything
<bigjools> IIRC smoser filed a bug about it
<jtv> I'm looking for the retry/partial-downlod logic.  Should find out how to get more information out of it.
<bigjools> rvba: this happened for you in the lab, right?
<rvba> Correct.
<bigjools> rvba: is it still failing there?
<rvba> Define "still" :)
<rvba> Last time I used the lab, it failed.
<rvba> Last time Diogo used the lab, it failed.
<bigjools> can you try removing the ephemerals completely and try again?
<jtv> Proxy weirdness could be involved...
<jtv> That still fails.
<bigjools> I still think we need a lock BTW
<rvba> bigjools: you mean removing the existing ephemerals?
<bigjools> rvba: yes
<bigjools> rm -rf /var/lib/maas/ephemerals/*
<rvba> In the lab, we create a new VM every time we run tests.
<rvba> So this is long gone.
<bigjools> oh!
<bigjools> ok
<bigjools> thought you were playing around on one instance
<rvba> No.  Yesterday, I saw it on one instance, then Diogo saw the same problem on a completely different instance.
<bigjools> dandy
<jtv> Nor is it anything like a single, incidental flipped bit in the proxy â we've seen it on different images.
<rvba> jtv: it's clearly related to using a proxy.
<rvba> I ran the script without problems on canonistack instances yesterday (without using proxies).
<jtv> I believe you â but it's not a single, incidental flipped bit in its cache.
<rvba> Now I'm trying again on a canonistack instance but with a proxy this time.
<jtv> Now, proxies may easily do weird things with partial downloads, right?
<rvba> Maybe.
<rvba> I suspect the weirdness in on simplestreams' side.
<rvba> But I could be wrong.
<jtv> I wonder if it could be something like "proxy may not be sending exactly the same segment that you asked for, but script assumes that it does."
<rvba> Maybe.  Let's first reproduce the problem in a clean environment.  Then we can start reasoning about the problem.
<bigjools> rvba: install the same proxy in canonistack and get maas to use it
<bigjools> in fact we install a deb proxy by default anyway
<rvba> That's precisely what I've done :).
<bigjools> ha!
<bigjools> also is it my imagination or is the python script really slow?
<jtv> It's doing humongous downloads.
<rvba> It is painfully slow.  But it has huge files to download.
<bigjools> I mean, it's taken 8 minutes to do two distroarchseries so far and the old script took about 1 minute TOTAL
<jtv> Quarter-gig files.
<bigjools> (on canonistack)
<bigjools> I smell a rat
<bigjools> I'm going to look at the simplestreams code
<rvba> Hum, I got a TCP_MISS_TIMEDOUT in squid and the script failed with unexpected checksum 'sha256'.
<rvba> But I don't remember seeing TCP_MISS_TIMEDOUT in the labâ¦
<bigjools> hmmmmm
<rvba> I probably need to configure the proxy better.
<bigjools> this is all related
<rvba> So that it can cope with very long downloads.
<jtv> bigjools: look for contentsource.py
<jtv> Unfortunately it has a lot of different ways of doing things.
<jtv> Fundamentally, it creates a file-like object with the URL, and does a seek() for the offset of the current batch.
<jtv> Oh, that's for "file://" URLs.
<jtv> There's a lot of different data paths here.
<rvba> bigjools: http://paste.ubuntu.com/6249803/ this is squid's logs
<rvba> The first request is made by simplestreams.
<rvba> The second one by wget.
<bigjools> batch?
<rvba> And simplestreams explodes with the checksum error right after the download occurs.
<bigjools> my conclusion is that it's buggy as hell
<bigjools> rvba: the log doesn't say if it's using Range:
<jtv> If we're going to look at the source, we'll need to know which method it chooses for reading URLs.
<jtv> File size for that first download is rather small, isn't it?
<jtv> I'm surprised that gets a 200 response.
<bigjools> the fact that this is slow as hell means it's doing something stupid
<jtv> And the wget request was faster?
<rvba> bigjools: you're right about the download time as wellâ¦ it takes 11s to download an image with wget (without any proxy).
<bigjools> massively
<jtv> Actually, I've been assuming that the really huge number is the response size... what's that number right before the IP address?
<rvba> So simplestreams is really doing something stupid.
<jtv> I'm also a bit concerned about there being a deflate-aware version of the URL reading stuff.
<jtv> The combination with batches is one of those things that look like they could easily go wrong
<jtv> Ouch.  The code that selects the http download mechanism is nontrivial, but quietly swallows any error and makes assumptions about the reason.
<jtv> Not a recipe for failure at all, that.
<bigjools> IOError: [Errno 28] No space left on device
<bigjools> crap
<rvba> bigjools: yeah, you need a large instance, and use /mnt instead of /var/lib/maas
<bigjools> what are you putting on /mnt?
<rvba> I create a symlink so that the images are stored on /mnt
<bigjools> is it already mounted then?
<rvba> bigjools: cd /mnt/ ; mv /var/lib/maas .; cd /var/lib/; ln -s /mnt/maas/
<rvba> bigjools: yes
<bigjools> ok
<bigjools> oh I already have a /mnt with lots of space
<rvba> bigjools: yes you do :)
 * bigjools moves 10Gb over
<jtv> We really should sabotage the code that cleans up the broken file.
<jtv> I think that's simplestreams/objectstores/__init__.py:
<jtv>         if not cksum.check():
<jtv>             os.unlink(partfile)
<jtv>             if orig_part_size:
<jtv>                 LOG.warn("resumed download of '%s' had bad checksum.", path)
<bigjools> https://bugs.launchpad.net/simplestreams/+bug/1240838
<ubot5> Ubuntu bug 1240838 in simplestreams "simplestreams is several orders of magnitude slower at downloading than wget" [Undecided,New]
<rvba> Yay!
<bigjools> ahem
<jtv> We've got batching, checksumming, and compression all handled at more or less the same level in the code.  It'd be nice to have a clear indication of what's happening.
<jtv> I can think of horror scenarios for any combination.
<bigjools> jtv: prefer functions to do one simple thing
<jtv> Doesn't that lead to a puzzle of lots of small functions?  :)
<bigjools> the puzzle remains the same - the reading it becomes easier
<bigjools> of it*
<jtv> In that case I shall give less attention to complaints about my small functions.  :)
<bigjools> jtv: I like them :)
<bigjools> it's a trade-off of course
<jtv> Down with spaghetti code, yay for macaroni code.
<bigjools> more like lasagna
 * jtv wonders whether it's a good idea to define "pizza code" in any detail
<rvba> bigjools:  I really wonder what that script is doingâ¦ it takes minutes before I see a single hit in the proxy's cache.
<bigjools> !
<bigjools> please add that as a data point to the bug
<jtv> The proxy only reports at the end of the request, not the start.
<rvba> bigjools: of course the log entry is written *after* the download but since we know that the download itself it pretty quick, it does not explain anythingâ¦
<jtv> Or so it seems to me â Apache does that, plus, it reports a return code doesn't it?
<rvba> bigjools: I will, of course.
<bigjools> rvba: or is it quick ...
<bigjools> we need debugging in the main script
<rvba> Yeah, that's the only option leftâ¦
<rvba> :/
<jtv> I think we need debugging code in the simplestreams code... For starters, which "opener" we get.
<jtv> Well no, for _starters_, we need that file so we can look at it.
<gmb> allenap: When you wrote the comment on line 248 of http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/view/head:/src/maascli/api.py#L248, what were you actually meaning?
 * bigjools guesses unicode
<bigjools> and print leaves a \n IIRC
<bigjools> gmb: he's not around today BTW
<gmb> bigjools: Right, hence the "files downloaded with the API get an extra newline"
<gmb> OIC.
<bigjools> yeah - easy to fix I reckon :)
<gmb> bigjools: Oh, definitely :). Question: do we _want_ that extra newline for prettification of textual output to stdout?
<gmb> (If so, I'll put it on stderr, which is what we do for headers when printing those out)
<bigjools> there's a fun question
<bigjools> we probably do
<bigjools> and that's a neat solution
<gmb> Cool beans.
<allenap> gmb: I think it was two things, "trailing newline" and "might encode", but you seem to be there already.
<gmb> allenap: Right - I'm on dangling modifier patrol today, so I didn't know if it was two statements or one.
<rvba> smoser: ping
<smoser> rvba, here.
<rvba> Hi smoser, I'd like to talk about bug 1240652.
<ubot5> bug 1240652 in MAAS "maas-import-ephemerals crashes with "unexpected checksum 'sha256'" when using a proxy" [Critical,Triaged] https://launchpad.net/bugs/1240652
<rvba> smoser: my testing shows that simplestreams is simply stuck in contentsource.py:RequestsUrlReader.read
<smoser> rvba, can i see a system where you reproduced ?
<rvba> After a while, the proxy times out, closes the request;  then simplestreams tries to checksum a partial file and that fails.
<rvba> smoser: yes, one sec
<smoser> i can follow your doc its fine in comment f4 excetp for squid config.  that'd be the only part dependent on maas.
<rvba> smoser: right, I didn't do any modification to the config, so it's whatever maas ships.
<rvba> smoser: and we also saw the same problem in the lab, using a different proxy (i.e. not the one shipped with maas).
<rvba> smoser: ssh ubuntu@10.55.61.23
<rvba> smoser: join the byobu session there, I'll show you around
<smoser> rvba, k. there are 2 things.
<smoser> which url reader are we using.
<rvba> smoser: the default one, python-requests
<rvba> smoser: I put lots of debugging output :)
<smoser> yeah. so i got rid of that, and now it seems functional.
<smoser> this sucks.
<smoser> rvba, so, it uses python-requests if it is found with sufficient version.
<smoser> the reason for using it is that  it supports compressed encoding
<smoser> (which clearly isn't useful on already compressed data, but for metadata it is)
<rvba> smoser: I seeâ¦ well, using a different url reader it seems to work indeed.
<rvba> Which, like you said, sucks.
<smoser> rvba, i found http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=712915
<ubot5> Debian bug 712915 in python-requests "python-requests: adapters.py uses undefined name ProxyManager" [Important,Fixed]
<smoser> earlier in the cycle.
<smoser> so its fairly clear that this is unfortunately not really well tested code
<smoser> i'll debug, rvba
<smoser> thanks for your debugging.
<smoser> whats weird is that we could not reproduce in the lab after the first time.
<rvba> Thanks for your help smoser.
<rvba> I think that's because the partial download stuff hides the problem somehow.
<smoser> rvba,
<smoser> why cann't i reproduce with
<smoser> http://paste.ubuntu.com/6251217/
<smoser> any ideas? the "this fails" doesn't really fail.
<rvba> smoser: let me try and see if I can spot a difference with when we run maas-import-ephemeralsâ¦
<smoser> found it
<smoser> python2 -> python3 fixes
<smoser> joy.
<rvba> Great joy indeed :)
<smoser> rvba, fwiw, https://bugs.launchpad.net/simplestreams/+bug/1240838
<ubot5> Ubuntu bug 1240838 in simplestreams "simplestreams is slower at downloading than wget" [Undecided,Triaged]
<smoser> "it takes several minutes before the proxy is actually used" is probably wrong
<smoser> i've been confused by this before.
<smoser> squid prints to log only after downloading
<rvba> smoser: yeah, that's the same isse.
<rvba> issue*
<rvba> Yep.
<smoser> it is very confusing when you're tailing logs :)
<smoser> and downloading large files.
<rvba> But since the download was supposed to take 10 seconds, it didn't make sense for the script to be blocked for several minutes.
<rvba> Now we know this is all because of python-requests acting up.
<jpds> smoser: Hey.
<smoser> matsubara, rvba fudge.
<smoser> https://bugs.launchpad.net/maas/+bug/1240652
<ubot5> Ubuntu bug 1240652 in MAAS "maas-import-ephemerals crashes with "unexpected checksum 'sha256'" when using a proxy" [Critical,Triaged]
<smoser> "
<smoser> FWIW, I ran into this bug yesterday while testing the ISO and there was no proxy involved.
<smoser> "
<smoser> i sort of dont believe that. as i cannot recreate it, and *with* a proxy, its trivial to recreate.
<smoser> oh. wait. that is the locking bug.
<smoser> that is basically unerstood.
<smoser> never mind.
<smoser> jpds, whats up?
<jpds> smoser: You heard any reports about pserv failing to come back up on an upgrade from yesterday's package?
<rvba> smoser: really?  There is no cron-like system running the script at the moment, so I don't believe in the locking thing :)
<rvba> smoser: I don't see why the bug we saw couldn't be triggered from time to time without the usage of a proxy.
<jpds> smoser: I just upgraded, and maas-pserv went away: http://paste.ubuntu.com/6251438/
<jpds> smoser: But doing: service maas-pserv start --- made it come back.
<rvba> We just know it can be reproduced with a proxy.  We're not sure it only happens with a proxy.
<smoser> roaksoax, ^ see jpds above, thoughts?
<jpds> I can see in line 86 that pserv was suppose to come back, but then it disappeared...
<smoser> jpds, /var/log/upstart/maas-pserv.log have nything?
<roaksoax> uhmmm
<roaksoax> yeah the upstartlog should show more
<jpds> http://paste.ubuntu.com/6251469/
<smoser> release week is fun.
<roaksoax> yeah
<roaksoax> uhmm thays a new message ive never seen before
<jpds> Of course, there's no timestamp which helps.
<smoser> timestamps are for weenies
<smoser> hm..
<smoser> it would seem that if maas is expecting any value out of squid3 that it needs to configure it.
<smoser> #Default:
<smoser> # maximum_object_size 4 MB
<smoser> anything over that size (from /etc/squid3/squid.conf)  is not getting cached.
<smoser> so small debs will get cached. but images not nor large debs.
<matsubara> smoser, the proxy in the qa lab have maximum_object_size 1000 MB which should be enough to cache the images (not sure if your comments above are related to bug 1240652)
<ubot5> bug 1240652 in MAAS "maas-import-ephemerals crashes with "unexpected checksum 'sha256'" when using a proxy" [Critical,Triaged] https://launchpad.net/bugs/1240652
<smoser> well, they are related.
<smoser> but the point still stands.
<smoser> squid3 default config is basically useless to maas.
<smoser> so if we're installing it, we need to be configuring it
<smoser> (also, default is no on disk cache, memory only)
<matsubara> smoser, in any case, when I tested the ISO and got the checksum error, I wasn't using a proxy but my connection to the datacenter is not so great. Could it be that I managed to trigger the bug locally without a proxy because m-i-e was waiting for data, didn't get it for some time, then it retried the download, triggering the bug?
<matsubara> smoser, I think squid3 is part of the dependencies for maas because of the s-d-p. I don't think we require the default install of MAAS to have any proxy configured.
<smoser> ah. i forgot that we have squid-deb-proxy
<smoser> yeah, the dfeault config of that should make more sense.
<smoser> but still wont help for images
<smoser> i think i have a fix for bug
<smoser> http://paste.ubuntu.com/6251802/
<smoser> matsubara, where did you do this test ?
<smoser> where you saw it from the iso
<smoser> i think there is a proxy in your path
<smoser> in your network path
<matsubara> smoser, locally on my laptop. I have a proxy running on my laptop too but I didn't use it during the install (i.e. the installer asks if I want to use a http proxy for which I answered no)
<smoser> you could be proxied along your path
<smoser> ?
<smoser> ie, another transparent proxy.
<roaksoax> Ruetobas: ping
<roaksoax> err
<roaksoax> sorry
<roaksoax> rvba: ping
<matsubara> smoser, I might. I'm pretty sure there's no transparent proxy inside my network. If my ISP has a tranparent proxy running, then it's beyond my power to change any config (and would make this bug even more critical). Is there any way to check if I'm going through a transparent proxy? I guess not right, that's the point of it being transparent
<smoser> matsubara, only to find bugs like this :)
<roaksoax> allenap: around?
<smoser> you're right. clealy its not your fault, and you have no control. i was just trying to explain what i think is wrong but didn't make sense for your path.
<roaksoax> adam_g: where's the wikipage that you were maintining showing how to install openstack with maas& juju? https://help.ubuntu.com/community/UbuntuCloudInfrastructure ?
<adam_g> roaksoax, here? https://wiki.ubuntu.com/ServerTeam/OpenStackHA
<roaksoax> adam_g: that's for the HA stuff.. i mean the one you were maintaining on help.ubuntu.com/community?
<adam_g> roaksoax, there was one at the URL you posted, dunno where it got moved to
<adam_g> roaksoax, jorge also started https://help.ubuntu.com/community/UbuntuCloudInfrastructure/JujuBundle.
<roaksoax> adam_g: thanks!
<Spideyman> does anyone here have much experience with MAAS restful API? What authentication header do I need to send with the key?
<Spideyman> Authorization: Basic"
<Spideyman> ?
<Spideyman> X-Auth?
<smoser> Spideyman, i'd look at maas-cli
<smoser> (as in look there for the details)
<Spideyman> smoser got it, thanks
<bigjools> smoser: please can you help debug https://bugs.launchpad.net/bugs/1240838, it makes ephemeral download practically unusable
<ubot5> Ubuntu bug 1240838 in simplestreams "simplestreams is slower at downloading than wget" [Undecided,Triaged]
#maas 2013-10-18
<darkmanta> hi everybody, I'm new here, I'd like to pose some questions about maas. how to do it?
<bigjools> ask away
<darkmanta> thank you. I'm trying MAAS on Virtualbox virtual machines. One of the problems I have met is that ALL ephemeral .tgz look crc-corrupted. I've tried to download the images through maas service and wget but the result is always the same
<bigjools> darkmanta: is this the latest maas in saucy by any chance?
<bigjools> https://bugs.launchpad.net/maas/+bug/1240652
<ubot5> Ubuntu bug 1240652 in simplestreams (Ubuntu T-series) "maas-import-ephemerals crashes with "unexpected checksum 'sha256'" when using a proxy" [High,In progress]
<darkmanta> I've tried 12.04.3, 12.10, 13.04 versions (desktop and server versions) for the virtual maas server. the result is always the same. can't complete the 'import boot images' procedure from web interface and from maas-import-pxe
<bigjools> log please
<darkmanta> i'm not on the right PC so I can't send the logs. but if anybody tries to download and check the ephemeral image...
<bigjools> darkmanta: we see this only on saucy - it's the bug above
<darkmanta> ok thank you
<darkmanta> at the moment I'm unable to try MAAS at all.
<darkmanta> trying to start a client machine not by PXE but through a ubuntu boot cd, following the MAAS procedure menu, fails again. the procedure ends up with a machine reset.
<darkmanta> bigjools thank you for helping me. do you know a configuration that works well?
<darkmanta> at the moment it seems I've tried all possible combinations of ubuntu versions, server/desktop included...
<darkmanta> anybody knows a MAAS configuration that simply works?
<rvba> darkmanta: the only configuration with a known blocking problem is 13.10.  And a fix is coming up soon.  If you have problems with another versions, it's not normal, but we need the error log to help you as it's probably related to your particular setup.
<gmb> rvba, allenap: Can you take another quick look at https://code.launchpad.net/~gmb/maas/fix-stray-newlines-bug-1187851/+merge/191611 please?
<gmb> Now complete with isatty() goodness.
<darkmanta> rvba: thank you. I've set up a 12.04.03 virtual machine following the procedure at http://maas.ubuntu.com/docs/install.html
<darkmanta> the problems i've found are: maas-install-pxe-images always fails with 404 file not found
<darkmanta> 2) client PC never receive PXE booting image
<darkmanta> 3) client PC booting by CD procedure always fail with a system reboot after selecting the IP of the MAAS server
<rvba> All of the problems seem related to the import script not running correctly.
<rvba> This smells like bug 1212434 but this bug has been fixed.
<ubot5> bug 1212434 in MAAS "maas-import-pxe-files breaks on 404" [Critical,Fix released] https://launchpad.net/bugs/1212434
<rvba> So I'd like to know which version you're using exactly (apt-cache policy maas) and the error message you're getting when you run the import script.
<allenap> gmb: You can avoid the buf.seek(0); buf.read() with buf.getvalue().
<gmb> allenap: You remind me of Benji sometimes. And I mean that as a compliment.
<allenap> gmb: Otherwise, looks grand :)
<gmb> Thanks
<allenap> gmb: I don't know benji well enough to know :)
<darkmanta> yes, iv'e patched the import script by removing the non-AMD63/non-i386 images downloads so the script continues with ephemeral images. but after the download these images always fail due CRC-error
<gmb> allenap: I learnt more about Python from getting him to review my code than I have anywhere elseâ¦ Looks like you're a more than worthy successor :)
<rvba> darkmanta: CRC error?  That, as far as we know, is only happening on Saucy.
<rvba> darkmanta: if you're using Saucy as the host of the MAAS server and seeing a checksum error, then there is a workaround (which is nothing more than that fix that is coming up shortly).
<darkmanta> try to download the ephemeral images by wget... you can't expand it because of crc-error
<rvba> I just downloaded all the images on a test system and all went fine.
<rvba> darkmanta: can you confirm you're seeing the error described on bug 1240652 ?
<ubot5> bug 1240652 in simplestreams (Ubuntu T-series) "maas-import-ephemerals crashes with "unexpected checksum 'sha256'" when using a proxy" [High,In progress] https://launchpad.net/bugs/1240652
<darkmanta> the problem i have found seems different. anyway i see there is a patch committed 12 hours ago. I will try again to see if it's reproducible
<rvba> darkmanta: I just added a comment on that bug.  It shows how to apply the fix manually if you don't want to wait for the package to be published :).
<darkmanta> rvba: thank you so much
<rvba> Welcome.
<baffle> Hi, does MAAS acutally support defining multiple DHCP ranges on one controller that are not directly connected? I.e. so that I can PXE boot servers on different L2 networks, that have an L3 router with dhcp/tftp/etc proxying?
<rbasak> I think that could be made to work. I assume that you'd make everything route right on the MAAS server(s), and provided that you proxy everything, what support would MAAS need for that? Everything would be transparent to it, right?
<baffle> rbasak: Yes, everything would be transparent. I have this running using TheForeman right now, and have various manual setups earlier.
<baffle> rbasak: The only "obstacle" is that MAAS requires an interface. And also seem to only allow one subnet pr. interface. :)
<rbasak> What makes you say that MAAS only allows one subnet per interface?
<baffle> rbasak: Guessing. :)
<baffle> rbasak: It looks like tht from the GUI at least.
<baffle> rbasak: But I guess I could just be wrong.
<rbasak> I would expect the MAAS server to be on an L3 network that can route to all the others, and for you to sort that routing out, together with the network configuration on the MAAS server itself.
<rbasak> So the MAAS server would have a single IP, and everything would route to that.
<rbasak> I don't see why the MAAS GUI would need to be any different to normal.
<rbasak> Alternatively you could have separate region controllers but I'm not familiar with that side of MAAS.
<baffle> rbasak: Currently in the GUI I need to specify an interface and the IP MAAS should have on this interface.
<rbasak> I don't understand the details then, sorry. But I don't see it as a big hurdle unless I'm missing something.
<baffle> rbasak: I think you understand it; Everything network-wise works as it should (dhcp-requests arrive at the IP for the MAAS server) but configuration-wise there doesn't seem to be any way to define subnets without also configuring the interface on the server.
<baffle> ISC DHCP handles stuff perfectly, as long as it has proper configuration. :)
<rvba> baffle: you can always config the dhcp server managed by maas manually.
<baffle> rvba: Doesn't MAAS use omapi etc to control dhcp/dns?
<rvba> baffle: it does, but this doesn't mean you cannot change the configuration manually.
<rvba> baffle: well, you cannot configure it any way we want.  For instance MAAS relies on the dhcp information it has to figure out to which cluster a connecting node belongs.
<rvba> s/way we want/way you want/
<baffle> baffle: Yes, I would assume I would have to add the DHCP scopes into MAAS someway. I just can't really see where. :-/
<rvba> Right, I think you've hit something that MAAS does not currently support.
<baffle> I kindof assumed this would be a quite normal usecase, to be honest. :-) Is everyone using a dedicated cluster controller for every subnet/rack of servers?
<rvba> Yeah, that's the current recommended way to do this.  But we have plans to extend this.
<baffle> rvba: Hmm, should be fairly straight forward...
<rvba> Yes indeed.
<baffle> Instead of having an "Interfaces" section under "CLuster Controller" it should just say "Networks". And the "interface" part of a network definition (I.e. wich physical network interface + IP address to configure (should be network address if no interface "selected")) should be optional. And it would be nice to have an option to override the "DNS zone name" pr. Network, instead of having one master zone.
<baffle> Would also be nice to actually query/browse the current leases on DHCP as well. :)
<rvba> baffle: trueâ¦ could you please put that in a bug?
<baffle> Actually, an "advanced" field where you can specify extra options for the dhcp-scope would be nice too.. :)
<baffle> Sure.
<rvba> Ta.
<baffle> https://bugs.launchpad.net/maas/+bug/1241473
<ubot5> Ubuntu bug 1241473 in MAAS "Support non-local DHCP scopes" [Undecided,New]
<baffle> Does this seem OK?
<baffle> Hmm. I'm a bit at a loss now, I guess I'll just jump on Cobbler instead for now.. I use TheForeman currently, but it is also a bit lacking and has tight Puppet integration. Razor seems to be in a state of flux, so that probably can't be used as well. Ideas?
<baffle> Hmm, why does maas use pxelinux instead of ipxe? Is there compatability issues?
<baffle> For previous setups, I've always loved how ipxe supports things like http:// and variables... I.e. having it chainloading deployment configuration from something like http://deployserver.com/deploy?mac=$mac is always nice. :)
<rbasak> ARM compatibility with ipxe would be harder, but I don't think that's the reason since pxelinux was used before then.
<baffle> rbasak: That's true. I see there is no ARM port for iPXE yet.
<baffle> So I guess the current model is to dynamically create pxelinux.0/<macaddress> files?
<rbasak> Right
<Nik_> marcoceppi: remember when I was saying that LSHW was out of sync with maas so tags wouldn't apply... I found out why - We added disks to the machines and the machines needed to be recommissioned so that lshw output would be picked up for purposes of tagging.
<Nik_> This could technically be a bug or an enhancement that can be submitted. Not sure how to check...
<Nik_> http://askubuntu.com/questions/353814/maas-out-of-sync-with-lshw ^
<fizzi_> Hi all
<fizzi_> a question ...
<fizzi_> is possible to install maas + nodes on vmware esx 4/5 ?
<Nik_> Not an expert, but I don't see why not
<Nik_> I think there's only a technicality regarding pxe booting a VM
<Nik_> so I see why there may be issues
<fizzi_> the pxe works fine
<fizzi_> but when i start machine from maas
<Nik_> Maas assumes IPMI also
<fizzi_> the machine dont start
<Nik_> for bare metal
<Nik_> Yeah, it uses IPMI to boot up metal
<Nik_> Hmm I guess I dunno
<marcoceppi> Nik_: fizzi_ there is "vMAAS" (Virtual MAAS) that might help
<marcoceppi> Nik_: thanks for the information, probably an enhancement, but open a bug against maas anyways
#maas 2013-10-19
<synergy__> Is this the place to get help with MAAS?
<synergy__> It's kicking my a**.
<synergy__> Commissioning...
<synergy__>  The action "Start selected nodes" could not be performed on 8 nodes because their state does not allow that action.
<synergy___> My Network card has Wake on Lan and PXE support.
<synergy___> You can boot this node using Avahi-enabled boot media or an adequately configured DHCP server. See https://maas.ubuntu.com/docs/nodes.html for instructions.
<synergy___> When creating a node in the web interface, am I supposed to have a mac address from somewhere else, or enter one I create?
<synergy___> Is there something I can type in the server's command window that would provide an answer?
<synergy___> Or a clue?
#maas 2013-10-20
<darkmanta> hi everybody
<darkmanta> may I ask a question about the process of importing boot images?
<darkmanta> maas-import-pxe-files gives me this message:
<darkmanta> http://ports.ubuntu.com/ubuntu-ports//dists/saucy/main/installer-armhf/current/images//highbank/netboot//vmlinuz: 2013-10-20 09:30:45 ERROR 404: Not Found.
<darkmanta> is there anybody? :-)
<darkmanta> does anybody know why quantal-daily-maas-i386.tar.gz returns gzip: stdin: invalid compressed data--crc error
<svetli> hello
<svetli> i have a huge problem with maas and i don't know what to do
<svetli> can you help
#maas 2014-10-13
<bigjools> any takers https://code.launchpad.net/~julian-edwards/maas/bulk-op-crash-bug-1376023
<rvba> bigjools: I'll take it
<gmb> rvba, allenap, jtv: For reviewage: https://code.launchpad.net/~gmb/maas/start-and-allocate-bug-1365591/+merge/238130
 * gmb -> lunch
<allenap> gmb: Iâll have it.
<gmb> allenap: Ta
<rvba> gmb: I added a comment on https://code.launchpad.net/~gmb/maas/start-and-allocate-bug-1365591/+merge/238130; I think there is something missing in that branch.
<rvba> Branch up for review: https://code.launchpad.net/~rvb/maas/explore-intf-bug-1375360/+merge/238139
<gmb> rvba: Thanks for the comment on my branch. For clarity: do we definitely want to move to a world where acquisition on start is transparent, even for API calls? I canât see it doing any harm, I just havenât seen anything to suggest that (but I canât think of a reason not to do it)
<rvba> gmb: well, if we do it in the UI, it would be weird not to do it in the API.
<gmb> rvba: Agreed. I guess Iâm assuming that there was a reason for doing things differently in the API, but I guess thatâs not really the caseâ¦?
<rvba> gmb: now that I think about it, you're not really changing an existing behavior, you're in fact adding a shortcut.
<gmb> Right.
<rvba> gmb: i.e. a shortcut that we can probably decide *not* to add to the API.
<gmb> rvba: Right. Iâll leave it as UI for now; we can always fix the API later. Can I have your r= then please?
<rvba> gmb: okay
<rvba> allenap: care to have a look at https://code.launchpad.net/~rvb/maas/explore-intf-bug-1375360/+merge/238139 again?
<allenap> rvba: Bootiful.
#maas 2014-10-14
<bigjools> any takers https://code.launchpad.net/~julian-edwards/maas/localboot-amt/+merge/238232
<gmb> rvba, bigjools, jtv, allenap: Any idea where I can get a copy of MAAS 1.6.1? Trying to reproduce https://bugs.launchpad.net/maas/+bug/1379924, and upgrading to 1.6.1 is part of the process.
<ubot5> Ubuntu bug 1379924 in MAAS "maas 1.7 flooded with OOPSs" [Critical,In progress]
<rvba> gmb: the easiest is probably to build the package your self from lp:maas/1.6
<rvba> yourself*
<gmb> rvba: Iâm just waking up, clearly, because you said that and I groaned out loud.
<gmb> But youâre right
<rvba> bigjools: reviewing (and testing) https://code.launchpad.net/~julian-edwards/maas/localboot-amt/+merge/238232.  Looks nice!
<roaksoax> gmb: try sudo apt-get install maas=1.6.1 or something along those lines if you are using the testing PPA
<gmb> roaksoax: Aah, thank you!
<rvba> \o roaksoax
<roaksoax> rvba: o/
<roaksoax> rvba: ok, so I have the packaging done, the only thing is not done yet is dhcpdv6d
<roaksoax> to dhcpd6
<roaksoax> rvba: so that's what we want to do?
<rvba> roaksoax: you mean change the name to dhcpd6?
<roaksoax> rvba: yup
<rvba> Seems good to me, that seems to be what's used by the daemon itself.
<rvba> dhcpd6.conf dhcpd6.leases
<jtv> Yeah, with the naming I tried to fit in with what was already out there â even if it didn't always seem quite rightl
<jtv> .
<roaksoax> ok I'll get that changed
<jtv> Were we adding a dhcp probe on all network interfaces to the commissioning stage?
<jtv> If we report all knowable network connections at that point, that could be another nail in the coffin of the leases parser.
<rvba> jtv: right now the dhcp probe relies on the leases parser to gather that information (i.e. the information from the discovered interface is not sent to MAAS: the leases parser will connect the dots)
<rvba> interfaces*
<jtv> rvba: right â but as soon as we turn it into a commissioning output...
<jtv> We already have all the major pieces in place for that.
<rvba> jtv: yeah, we could do that indeed.
<jtv> Not a lot of time for it now, I understand.  But one likes to dream of the demise of that parser...
<jtv> Or rather, those two parsers <shudder>
<rvba> heh
<rvba> This is definitely something to keep in mind for next cycle, I'm with you on this.
<jtv> Because it's the one place where we really rely on the parser(s).
<jtv> And I think with last week's IPv6 changes, we're very close indeed to fully static IPv4 configuration.
 * jtv takes break from typng
<bigjools> gmb: release PPA
<gmb> bigjools: Built it myself in the end, but thanks :)
<bigjools> jtv: coming to the daily call?
<jtv> Oh!  Yes.
<roaksoax> 3/win 4
<rvba> jtv: why do we have the 'disable ipv4' on the cluster and not on the cluster interface?
<jtv> rvba: think it through.  :)
<jtv> For starters, it only applies in cases where you have multiple cluster interfaces.
<jtv> So we'd have to have it on the IPv4 one but not the IPv6 one, or something â but you can change IPv4/IPv6 on the fly.
<rvba> jtv: I see
<jtv> There's more:
<rvba> jtv: about the other branch: Looks generally good but I wonder why you decided to override the field with a forms.HiddenInput() widget instead of removing the field altogether?
<jtv> To keep it working in the API.
<rvba> jtv: okayâ¦ that might be worth a comment in the code then.
<jtv> OK
<jtv> (BTW it's also explained in the MP)
<rvba> Yeah, I missed it somehow.
<gmb> rvba, allenap: https://code.launchpad.net/~gmb/maas/acquire-and-start-node-bug-1381007/+merge/238279 needs a review please, gentlemen.
 * gmb -> lunch
<allenap> rvba: May I land https://code.launchpad.net/~allenap/maas/cluster-name-override--bug-1380805/+merge/238307?
<rvba> allenap: this is a follow-up for Julian's branch right?
<rvba> allenap: looking at Julian's branch nowâ¦
<allenap> rvba: It's a replacement for Julian's branch.
<rvba> allenap: right, that's what I figured out reading his MP.
<allenap> rvba: I only meant to ask if I could land it now, seeing as you and kiko are the Guardians Of Trunk right now.
<rvba> allenap: you have my blessing my child
<allenap> rvba: Thank you!
<allenap> roaksoax: A review for you: https://code.launchpad.net/~allenap/maas/respawn-cluster-until-it-works/+merge/238331. I've tested it.
#maas 2014-10-15
<bradm> anyone seen a maas node thats been added to juju hang around in agent-state: pending ?
<rvba> bigjools: I Can Has Review? https://code.launchpad.net/~rvb/maas/explore-intf-bug-1375360-3/+merge/238394
<rvba> allenap: any reason why the text starting with "A shared secret" is indented differently from the rest of the code?
<allenap> rvba: There's a tab there. The <<-'EOF' syntax with bash only trims tabs, not spaces.
<rvba> allenap: ah, okay.
<allenap> s/bash/bash and sh/
#maas 2014-10-16
<bradm> anyone seen maas stop doing dynamic updates for juju hostnames when you deploy?  if I start a boostrap, I can ssh into the host via IP, but the bootstrap fails because the cname for the host doesn't seem to be in place
<bradm> bigjools: ^^ ? :)
<bigjools> bradm: CNAMEs have gone away in 1.6 onwards
<bigjools> but never heard of that happening unless the celeryd is down
<bigjools> region celery that is
<bradm> bigjools: this is with maas 1.5, it was working and then has suddenly stopped
<bigjools> is region-celery running?
<bigjools> and connected
<bradm> bigjools: it appears to be working, how do I tell if its connected?
<bradm> er, s/working/running/
<bigjools> is it processing jobs?
<bigjools> also is the cluster celery running and uploading dhcp leases?
<bradm> bigjools: sure looks like it, I'm seeing things about "Received task" in the logs
<bigjools> and the cluster celery?
<bradm> bigjools: looks like it
<bigjools> and the zone file doesn't have the CNAME?
<bradm> bigjools: nope
<bradm> bigjools: juju bootstraps are failing because I can't ssh to the hostname, but if I use the IP its fine
<bigjools> can you restart the region celery
<bradm> sure, doing so
<bradm> ok, kicking off a bootstrap now
<bradm> bigjools: ok, I see the dhcp offer and ack, but I can't resolve the hostname
<bigjools> can you restart the cluster celery
<bradm> how about I stop them both, then start
<bigjools> and then can you examine the database to see if the dhcp lease is in
<bradm> aha, how do we do that?
<bigjools> I gotta run, dinner time.  "sudo maas-region-admin dbshell --installed"
<bigjools> select * from maasserver-dhcplease;
<bigjools> err _ not -
<bradm> bigjools: perfect, thanks.  will see how it goes
<bigjools> if the lease is not there there's something wrong with the cluster celery lease parsing and/or upload to the region
<bigjools> ttfn
<allenap> mgz: http://paste.ubuntu.com/8573190/
<mgz> ta
<th3rt> Silly question here,  but MaaS seems to selectively pick up and display IP addresses assigned to hosts.  Host discovery and boostrapping seems to get the IP fine,  but the MaaS web interface doesn't display it.
<th3rt> Any idea why?
<th3rt> Never mind.  Looks like I missed the part where the docs say that if I use MaaS dhcp then maas won't be able to statically assign an address.  I suspect it is this static address assignment that populates the web interface.
#maas 2014-10-17
<Rastin> Hi
<Rastin> I have a question about MAAS
<Rastin> does eanyone know the nodes of must must be connected directly to internet? or the use MAAS Server to connect to the internet?
<sfeole> Rastin: i just happened to look over here, while trolling through my irc channels,  the nodes to not need to be directly connected to the internet,  They MUST be accessible via the maas cluster where that may be.   You can pass a HTTP / HTTPS proxy to your maas nodes so that they can access the internet, download tools, etc
<bigjools> Rastin: they need a little access but it can be via the proxy installed on the region controller
<jhobbs> you can run totally offline on MAAS 1.7, but it's not released yet. you do need to setup offine ubuntu archive and image streams mirrors
<bigjools> they just download a few packages when installing
<bigjools> what jhobbs said :)
<Rastin> but i get can not connect to the proxy server error
<Rastin> what i missed in network connection of cluster and region controller
<Rastin> ?
<Rastin> do I have to do some steps in network configuration of Cluster/Region COntroller?
<Rastin> The node can not connect to the internet, but MAAS server can
<bigjools> unless you're using 1.7, it's best to set up ipforwarding on the maas server
<Rastin> i havet to setup ip forwarding?
<jtv> allenap: some background on the pserv.yaml-rewrite bug...
<jtv> Configuring an IPv6 address there won't work yet, and that's a documented limitation.
<jtv> BUT when people do do that and realise their mistake, the bug stops them from reverting.
<allenap> jtv: Okay, worth fixing for that then.
<jtv> I think so.  I wish it would also magically IPv6-enable provisioning etc, but that's a bridge too far.
<mwenning> k2, can you send me the google hangout link?
<allenap> rvba: https://code.launchpad.net/~allenap/maas/respawn-cluster-until-it-works--1.7/+merge/238709 is ready to go into Utopic's package branch. It's already been reviewed into trunk, so I'm just looking for a rubber-stamp into 1.7.
<rvba> allenap: cool
<mburleigh> hello, MaaS newbie question. I have a SuperMicro server that shows a 0 CPU count, 0 MB ram. Where do I start to find out why its not reporting correctly. And does these actually mean anything in regards to deployment?
<jhobbs> mburleigh: have you commissioned the node yet?
<mburleigh> jhobbs: no, will that re-collect that information?
<jhobbs> mburleigh: it will collect the information yes
<mburleigh> jhobbs: commissioned and still cpu and mem are 0
<mburleigh> and in ready state
<jhobbs> mburleigh: that's interesting - what version of MAAS?
<mburleigh> i guess commissioning is not completing, because it is still an option for this node. I have two other, different servers that get the information correctly.
<jhobbs> mburleigh: you can commission a node multiple times
<mburleigh> hmm
<jhobbs> mburleigh: is there lshw output in a "Discovery Data" section at the bottom of the node's page?
<mburleigh> discovered as a switch, not server ;-)
<gmb> allenap: Asked for your thoughts over on https://bugs.launchpad.net/maas/+bug/1382575
<ubot5> Ubuntu bug 1382575 in MAAS "Same node handed out in response to two parallel 'acquire' requests" [Critical,In progress]
<jhobbs> gmb: what good are django's transactions if they don't commit prior to returning info to the user?
<jhobbs> transactions per request that is
<jhobbs> i wonder if claim_sticky_ip is broken the same way
<gmb> jhobbs: There are 11 more words in your question than you really needed :)
<gmb> jhobbs: Thatâs entirely possible.
<jhobbs> haha
<jhobbs> adding a transaction.commit() in node.acquire(), after the save, doesn't fix this
<gmb> jhobbs: Arse. Then we probably need to lock the table.
<gmb> Or at least the row
<gmb> Havenât done that myself in MAAS beforeâ¦ allenap: any advice?
<mburleigh> jhobbs: ok, so the switch info is always there, but there is no additional data (like other servers have), what next?
<jhobbs> mburleigh: what version of MAAS is this?
<mburleigh> 1.5.4
<mburleigh> should I upgrade to 1.7?
<jhobbs> no
<jhobbs> mburleigh: there should be a link on your node's page: "Commissioning Output"
<jhobbs> mburleigh: sorry, it will be something like "8 output files" under the header "Commissioning Output"
<jhobbs> mburleigh: can you go there and see if there is lshw output?
<mburleigh> doesn't exist, that node only has 6 output files
<jhobbs> mburleigh: i don't know then - it sounds like a bug to me - please file one. have you tried deploying to the node?
<mburleigh> modaliases contains: dmi:bvnAmericanMegatrendsInc.:bvr2.0:bd01/11/2013:svnSupermicro:p nX9DRi-LN4+/X9DR3-LN4+:pvr0123456789:rvnSupermicro:rnX9DRi-LN4+/X9DR3-LN4+:rvrREV1.20A:cvnSupermicro:ct3:cvr0123456789:
<mburleigh> jhobbs: yes, how relavant is the cpu and mem count in maas, besides constraints?
<jhobbs> mburleigh: that's all it's used for
<mburleigh> ah, good to know. of course I can manually edit these and still constraints, yeS?
<mburleigh> still use constraints
<jhobbs> mburleigh: yes
<mburleigh> thx
<allenap> gmb: You could use a new DatabaseXactLock; see maasserver.locks for a hint.
<gmb> jhobbs: ^^
<gmb> allenap: ta
<allenap> gmb, jhobbs: That'll help address this issue, but there are going to be others.
<gmb> Oh yes there are
<allenap> We need a more general solution.
<gmb> Anyway, Iâm juggling three conversations here, so let me back out of this one, much as Iâd love to be involved.
<allenap> Okay.
<jhobbs> allenap: yeah, DatabaseXactLock worked
#maas 2015-10-12
<mup> Bug #1505034 opened: hwe kernel naming convention in the UI needs to change <MAAS:New> <https://launchpad.net/bugs/1505034>
<bogdanteleaga> I am trying to add and deploy a custom image in maas using the cli (sudo maas root boot-resources create title=<name> name=<name> architecture=amd64/generic content@=<name>.img) and while I see it in images I cannot use it when I click deploy
<bogdanteleaga> does anybody have an idea?
<mgz> bogdanteleaga: we may be short of maas people today, US holiday and swap days from sprint last week
<mgz> bogdanteleaga: but I understood you to mean an image added via the cli can't be used through the web gui?
<bogdanteleaga> mgz: that explains a lot
<bogdanteleaga> mgz: I may have figured it out give me a moment
<mgz> I think when we did it for the ci maas, just did everything through the web page.
<bogdanteleaga> mgz: but can you add custom images from the webui?
<mgz> we followed the instructions in your guys guide I think, maybe with a few changes
<bogdanteleaga> mgz: http://imgur.com/a/vR1LN
<bogdanteleaga> mgz: picture is a thousand words :)
<mgz> ha. can we boot off nano?
<bogdanteleaga> :)
<bogdanteleaga> soonâ¢
<bogdanteleaga> looks like the cluster is out of sync, can I force it to sink somehow? it's a 200mb image it shouldn't take long at all
<mgz> bogdanteleaga: I don't see a command for that, can try just turning it off and turning it on again :)
<bogdanteleaga> mgz: already tried that :(
<bogdanteleaga> mgz: I seem to be getting maas.start-up: [WARNING] Database error during start-up (PostgreSQL error 40P01); pausing for 3 seconds. a lot
<bogdanteleaga> which looks like it's a database deadlock
<mgz> that does seem pretty terrible
<mgz> got any full disk partitions?
<bogdanteleaga> nope
<bogdanteleaga> mgz: https://bugs.launchpad.net/maas/+bug/1474967 this looks similar
<mup> Bug #1459896 changed: [UI]  Scaling node listing page down, causes Add Hardware text to basically disappear from button <MAAS:Won't Fix by ricgard> <https://launchpad.net/bugs/1459896>
<mup> Bug #1505378 opened: websocket error on 1.9 alpha 4 <MAAS:Incomplete> <https://launchpad.net/bugs/1505378>
<mup> Bug #1505386 opened: Error messages regarding missing packages should be able to be pasted into the apt-get command directly <MAAS:Triaged> <https://launchpad.net/bugs/1505386>
<mup> Bug #1505386 changed: Error messages regarding missing packages should be able to be pasted into the apt-get command directly <MAAS:Triaged> <https://launchpad.net/bugs/1505386>
<mup> Bug #1505386 opened: Error messages regarding missing packages should be able to be pasted into the apt-get command directly <MAAS:Triaged> <https://launchpad.net/bugs/1505386>
<mup> Bug #1505386 changed: Error messages regarding missing packages should be able to be pasted into the apt-get command directly <MAAS:Triaged> <https://launchpad.net/bugs/1505386>
<mup> Bug #1505386 opened: Error messages regarding missing packages should be able to be pasted into the apt-get command directly <MAAS:Triaged> <https://launchpad.net/bugs/1505386>
#maas 2015-10-13
<mup> Bug #1505613 opened: IPMI power template sets configuration of IPMI chassis just before running each power on/off command and can make the IPMI interface unresponsive, causing commissioning to fail <MAAS:New> <https://launchpad.net/bugs/1505613>
<mup> Bug #1505613 changed: IPMI power template sets configuration of IPMI chassis just before running each power on/off command and can make the IPMI interface unresponsive, causing commissioning to fail <MAAS:New> <https://launchpad.net/bugs/1505613>
<mup> Bug #1505613 opened: IPMI power template sets configuration of IPMI chassis just before running each power on/off command and can make the IPMI interface unresponsive, causing commissioning to fail <MAAS:New> <https://launchpad.net/bugs/1505613>
<mup> Bug #1505613 changed: IPMI power template sets configuration of IPMI chassis just before running each power on/off command and can make the IPMI interface unresponsive, causing commissioning to fail <MAAS:New> <https://launchpad.net/bugs/1505613>
<mup> Bug #1505613 opened: IPMI power template sets configuration of IPMI chassis just before running each power on/off command and can make the IPMI interface unresponsive, causing commissioning to fail <MAAS:New> <https://launchpad.net/bugs/1505613>
<jwitk0> Hey All,  I am having trouble getting MaaS to use a proxy.  I have configured my proxy in the web ui settings but I am still getting the error  "Failed to import images from boot source http://maas.ubuntu.com/images/ephemeral-v2/releases/: HTTPConnectionPool(host='maas.ubuntu.com', port=80)"
<jwitk0> The command line maas-import-pxe-files asks for input of a sources-file but I am unsure of where this is or how to create one ?
#maas 2015-10-14
<mpontillo> jwitk0: see https://askubuntu.com/questions/310153/how-do-i-get-maas-and-juju-to-recognize-proxy-settings-in-a-deployment - basically you need to set the HTTP proxy during the image import.
<mpontillo> jwitk0: my other suggestion was going to be to create a local mirror of the images and point MAAS to that instead, which you could try if that doesn't work
<dimitern> hey, any maas guys around?
<dimitern> I have an old 1.5.4 maas cluster on 14.04 which I need to upgrade to 1.9 - should I do it in a special way (e.g. first to 1.6 then 1.9 etc.) ? I seem to recall people having issues with upgrading from 1.5
<mpontillo> dimitern: this is the only thing I know about - https://bugs.launchpad.net/maas/+bug/1495064 - fixed in the latest 1.9 alpha, and also 1.8.3 (unreleased)
<mup> Bug #1506053 opened: Data Loggers PDU unit does not shutdown outlet after commissioning or deploying a connected node <arm64> <data> <loggers> <xgene> <MAAS:New> <https://launchpad.net/bugs/1506053>
<ChuckD_> HI
<ChuckD_> I am curious if anyone has experienced SSL cert issues with MAAS during enlistment?
<ChuckD_> I get 'SSL certificate problem: self signed certificate in certificate chain'
<ChuckD_> Anyone awake?
<mup> Bug #1424549 opened: enlisting of nodes: seed_random fails due to self signed certificate <MAAS:In Progress> <https://launchpad.net/bugs/1424549>
<mup> Bug #1506087 opened: The leases parser needs some cleanup <techdebt> <MAAS:New> <https://launchpad.net/bugs/1506087>
<ChuckD_> Anyone available?
<mup> Bug #1506143 opened: ipaddresses?op=release only releases one address <api> <MAAS:Triaged> <https://launchpad.net/bugs/1506143>
#maas 2015-10-15
<mup> Bug #1506247 opened: NIC configured as "DHCP" in the UI gets configured as manual <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1506247>
<mup> Bug #1506247 changed: NIC configured as "DHCP" in the UI gets configured as manual <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1506247>
<mup> Bug #1506247 opened: NIC configured as "DHCP" in the UI gets configured as manual <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1506247>
<Guest86099> hi
<Guest86099> wanted some help in configuring MAAS with CentOS image node deployment
<Guest86099> has anyone in the group tried it?
<mup> Bug #1506441 opened: maas web UI: cannot remove default gateway on cluster interface (once set) <MAAS:New> <https://launchpad.net/bugs/1506441>
<mup> Bug #1506453 opened: statically assigned ipv6 address is deconfigured on procps restart <ipv6> <MAAS:New> <https://launchpad.net/bugs/1506453>
<fandi> hi all
<fandi> i have question when try to pxe boot .. os always power off
<fandi> when it finish ..
<mup> Bug #1506453 changed: statically assigned ipv6 address is deconfigured on procps restart <ipv6> <MAAS:Invalid> <https://launchpad.net/bugs/1506453>
<mup> Bug # opened: 1506516, 1506517, 1506518, 1506519, 1506520, 1506523
<mup> Bug # changed: 1506516, 1506517, 1506518, 1506519, 1506520, 1506523
<mup> Bug # opened: 1506516, 1506517, 1506518, 1506519, 1506520, 1506523
<mup> Bug # changed: 1506516, 1506517, 1506518, 1506519, 1506520, 1506523
<mup> Bug # opened: 1506516, 1506517, 1506518, 1506519, 1506520, 1506523
<mup> Bug # opened: 1506524, 1506525, 1506528, 1506529
<sa-shukla> Hi All
<sa-shukla> I wanted some help for https://bugs.launchpad.net/maas/+bug/1502839
<sa-shukla> have someone tried with CentOS deployment through MAAS?
<mgz> sa-shukla: jog has just been doing it recently, I'll ask him if he has any ideas on your problem
<mup> Bug # changed: 1506524, 1506525, 1506528, 1506529
<mup> Bug # opened: 1506524, 1506525, 1506528, 1506529
<sa-shukla> thanks mgz. Have been trying out different things since last few days, but things seem quite complicated with the CentOS
<sa-shukla> Document does not mention about any complications though
<mgz> sa-shukla: we're using maas 1.9, and had a different set of issues
<mgz> sa-shukla: so, probably not much help to you I'm afraid, apart from to say yeah, this stuff is not straight forward unfortunately
<sa-shukla> mgz - are you able to get it working on MAAS 1.9? What steps are involved?
<roaksoax> sa-shukla: did you download the centos image from MAAS 1.9 directly?
<sa-shukla> nope ... I am running MAAS 1.8 and downloaded it from daily repo provided by MAAS
<sa-shukla> earlier I was not aware and had compiled my own centos image using image-builder tool
<sa-shukla> somehow looks like the error is unchanged
<sa-shukla> looks like while try to fetch the image, URL contructed is using network instead of region / cluster controller IP
<sa-shukla> In the bug I have tried to attach screenshots as well
<mgz> sa-shukla: we were, but it involved fiddling with nics and how our setup stuff worked
<sa-shukla> mgz: ok, is it possible for you to send me the steps involved. I will try to see if I can figure out what is going on with my environment
<sa-shukla> it is possible that issues might be similar
<sa-shukla> on 1.8.2 and 1.9
<sa-shukla> my email ID is sa_shukla@yahoo.com
<mgz> sa-shukla: have you seen http://wiki.cloudbase.it/juju-centos
<sa-shukla> nope ... will take a look
<sa-shukla> but would be good if you can share the info
<sa-shukla> atleast that will have something for me to work on
<roaksoax> sa-shukla: install MAAS 1.9, change the 'Sync ULR' from http://maas.ubuntu.com/images/ephemeral-v2/releases/ to http://maas.ubuntu.com/images/ephemeral-v2/daily/
<roaksoax> sa-shukla: then go to the Images page
<roaksoax> sa-shukla: you'll see CentOS
<sa-shukla> yes ... I have downloaded using this approach
<sa-shukla> for 1.8.2
<roaksoax> sa-shukla: ok, so 1.8.2 hould work just fine
<roaksoax> sa-shukla: for 1.9: https://bugs.launchpad.net/maas-images/+bug/1499558
<roaksoax> sa-shukla: you need to change the storage layout
<roaksoax> sa-shukla: https://bugs.launchpad.net/maas-images/+bug/1499558/comments/3
<mgz> roaksoax: bug 1502839 is his experience so far
<mgz> ...this channel needs moar mup
<roaksoax> mgz: yeah, but I'm deploying centos right now on 1.9 and works like a charm :
<sa-shukla> you mean just importing the image through daily works out of the box?
<sa-shukla> I thought there were issues with 1.9 as well in deploying CentOS nodes
<sa-shukla> also 1.9 seems to be in alpha state ... is it stable enough for deploying?
<sa-shukla> roaksoax: my error message is different from storage layout error given in bug # 1499558
<sa-shukla> I can give it a try though if you feel this might be one of the possible issues looking at bug # 1502839
<roaksoax> sa-shukla: i just deployed 40 machines with centos using the flat layout
<roaksoax> sa-shukla: downloading the image directly from MAAS
<sa-shukla> ok, thanks. Will give it a try
<sa-shukla> roaksoax: what version of MAAS you are running? Is 1.9 stable enough to deploy?
<roaksoax> sa-shukla: yup
<roaksoax> sa-shukla: running 1.9
<roaksoax> sa-shukla: 1.9 is still alpha due to missing UI features
<roaksoax> sa-shukla: but provided those land later this week/early next week, MAAS will be moving to beta
<sa-shukla> roaksoax: will try with the workaround, else will try the upgrade
<sa-shukla> also occassionally, I have run into issues where I deleted a node in the UI, but it still trying to boot under MAAS provisioning
<sa-shukla> any way cleaning up the nodes?
#maas 2015-10-16
<mup> Bug #1506856 opened: Show devices with a parent on the node details page <ui> <ux> <MAAS:Triaged by carlaberkers> <https://launchpad.net/bugs/1506856>
<mup> Bug #1428831 changed: USB drive tagged "rotary" <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1428831>
<mup> Bug #1506909 opened: Can't change node name due to unrelated error <MAAS:New> <https://launchpad.net/bugs/1506909>
<mup> Bug #1497392 opened: maas text-mode "node-group-interface" bug fails with MAAS 1.8.2+bzr4041-0ubuntu1~trusty1 <MAAS:Confirmed> <https://launchpad.net/bugs/1497392>
<mgz> roaksoax: wotcha, trying to get juju working with maas 1.9 and centos - the auto image import however is using 'centos70' rather than 'centos7' as the release name,
<mgz> and then manually importing a custom image with the right name doesn't seem to get used when selecting release
<mgz> have you tried doing this at all?
<mup> Bug #1497392 changed: maas text-mode "node-group-interface" bug fails with MAAS 1.8.2+bzr4041-0ubuntu1~trusty1 <MAAS:Invalid> <https://launchpad.net/bugs/1497392>
<mup> Bug #1497392 opened: maas text-mode "node-group-interface" bug fails with MAAS 1.8.2+bzr4041-0ubuntu1~trusty1 <MAAS:Invalid> <https://launchpad.net/bugs/1497392>
<mup> Bug #1497392 changed: maas text-mode "node-group-interface" bug fails with MAAS 1.8.2+bzr4041-0ubuntu1~trusty1 <MAAS:Invalid> <https://launchpad.net/bugs/1497392>
<pmatulis> i have a kvm guest with two network adapters. do i need to do anything special for it to be enlisted?
<roaksoax> win 13
<roaksoax> mgz: how are you importing a custom image? as custom or as centos?
<roaksoax> mgz: if it is as "centos" then, it is not really a custom image, but rather, the above image will take presedence and the imported manually ignored
<roaksoax> mgz: please, do file a bug that highlights the change, so we can provide an alias that centos7 -> centos70
<mup> Bug #1506991 opened: maas should tell cloud-init not to try metadata service on every boot <falkor:New> <MAAS:New> <https://launchpad.net/bugs/1506991>
<roaksoax> pmatulis: no not really? MAAS will discover both NIC's and add them in MAAS
<mup> Bug #1506991 changed: maas should tell cloud-init not to try metadata service on every boot <falkor:New> <MAAS:New> <https://launchpad.net/bugs/1506991>
<mup> Bug #1506991 opened: maas should tell cloud-init not to try metadata service on every boot <falkor:New> <MAAS:New> <https://launchpad.net/bugs/1506991>
<pmatulis> roaksoax: ok
#maas 2015-10-17
<mup> Bug #1408875 changed: Uefi pxe boot can't written to the drive by fast installer using MAAS 1.5.4 <MAAS:Expired> <https://launchpad.net/bugs/1408875>
#maas 2015-10-18
<mup> Bug #1507359 opened: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:New> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 changed: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:New> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 opened: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:New> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 changed: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:Opinion> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 opened: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:Opinion> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 changed: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:Opinion> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 opened: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:Opinion> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 changed: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:Opinion> <https://launchpad.net/bugs/1507359>
<mup> Bug #1507359 opened: MAAS 1.9 image names and Juju streams disagree <add-machine> <centos> <maas-provider> <streams> <juju-core:Triaged> <juju-core 1.24:Triaged> <MAAS:Triaged> <https://launchpad.net/bugs/1507359>
#maas 2016-10-17
<pmatulis> smgoller, what version of maas? 2.0 or 2.1
<smgoller> 2.0
<smgoller> 2.1 is out?
<smgoller> hm. seems worth it to upgrade
<smgoller> do i need to go to 16.10 in order to use 2.1?
<smgoller> pmatulis, guess i'll test upgrading to 16.10 :)
<pmatulis> smgoller, no, you can use ppa:maas/next on Xenial
<smgoller> ok
<jamespage> congrats on the release maas team!
<mup> Bug #1423626 changed: Inconsistent device naming depending on install method - biosdevname/no biosdevname <maas-provider> <networking> <oil> <juju-core:Won't Fix> <MAAS:Fix Released> <maas (Ubuntu):Fix Released> <https://launchpad.net/bugs/1423626>
<mup> Bug #1634196 opened: [2.0,2.1] If I change what vlan an interface is on, then I cannot return it <MAAS:New> <https://launchpad.net/bugs/1634196>
<mup> Bug #1634196 changed: [2.0,2.1] If I change what vlan an interface is on, then I cannot return it <MAAS:New> <https://launchpad.net/bugs/1634196>
<mup> Bug #1634196 opened: [2.0,2.1] If I change what vlan an interface is on, then I cannot return it <MAAS:New> <https://launchpad.net/bugs/1634196>
<mup> Bug #1633598 changed: [2.1.1] Unable to set local boot-source via the API <MAAS:Invalid by ltrager> <https://launchpad.net/bugs/1633598>
<x58> Is there an easy way to pull the IPMI information from MaaS for all machines with the hostname of the machine?
<mup> Bug #1634294 opened: MAAS wants to serve authoritative reverse DNS for every subnet it finds <MAAS:New> <https://launchpad.net/bugs/1634294>
<mup> Bug #1634294 changed: MAAS wants to serve authoritative reverse DNS for every subnet it finds <MAAS:New> <https://launchpad.net/bugs/1634294>
<mup> Bug #1634294 opened: MAAS wants to serve authoritative reverse DNS for every subnet it finds <MAAS:New> <https://launchpad.net/bugs/1634294>
#maas 2016-10-18
<mcs31> I'm not able to deploy a node in MAAS 2.0. Can anyone help me?
<mcs31> Error displayed => Node installation failure - 'curtin' failed: curtin command install
<nathan_> Hello, i'm having trouble in MAAS web interface flickering (Safari 10.0 & Chrome 53.0.2785.143). I've installed MAAS on Ubuntu Server 16.04 and tried to configure VLAN. If I select the VID text input and try to insert any value as '21', ui starts to flicker, text inputs fly around the screen etc. Additionally under the VID input box there shows an error message: 'primary_rack: Select a valid choice. 1 is not one of the available choices
<nathan_> I'm accessing the web ui from native macOS 10.11.6, Ubuntu Server is installed in VM. VM's NIC is bridged with my native wlan interface and it uses dhcp to get IP. Everything seems to be working fine, but web ui seems to be broken. I'm using the newest 16.04 version, completely updated and upgraded to 4.4.0-43-generic kernel.
<mup> Bug #1634595 opened: [2.1] Reverse DNS authority per subnet should be configurable in the UI <ux> <MAAS:Triaged> <https://launchpad.net/bugs/1634595>
#maas 2016-10-19
<mup> Bug #1633209 changed: Daily Images appear to have broken for xgene-uboot <MAAS:Invalid> <https://launchpad.net/bugs/1633209>
<mah_> Hi all I had maas deployed on orange box
<mah_> but I did upgrade
<mah_> then all is messed up
<mah_> when I try to install something related to maas
<mah_> it ask for dependencies
<mah_> so how to solve the issue
<brendand> mah_, can you give details on the versions and what exactly fails
<brendand> e.g. what OS version, which maas version, which command you are running and what the output is
<mah_> I thought i have maas 1.7
<mah_> and when I did upgrade
<mah_> and I figure out then a problem when connecting to maas url
<mah_> connection refused
<mah_> so I removed maas
<mah_> and trying to install it again
<mah_> but the dependencies show 1.9 version
<mah_> so I guess that the upgrade is reason for that
<brendand> mah_, yes. 1.9 is the supported version in trusty
<mah_> maas : Depends: maas-region-controller (= 1.9.4+bzr4592-0ubuntu1~14.04.1) but it is not going to be installed         Depends: python-django-maas (= 1.9.4+bzr4592-0ubuntu1~14.04.1) but it is not going to be installed
<mah_> so why that message appears
<mah_> when I tried to install maas
<brendand> you might need to e.g. do an apt-get update
<brendand> and upgrade the system
<mah_> I did that
<mah_> but also the same problem
<brendand> what if you do apt-get -f install?
<mah_> E: Unable to correct problems, you have held broken packages.
<brendand> that should try and fix any issues
<mah_> apt-get -f install works ok
<mah_> this is the error I got when install maas
<mah_> The following packages have unmet dependencies.  maas : Depends: maas-region-controller (= 1.9.4+bzr4592-0ubuntu1~14.04.1) but it is not going to be installed         Depends: python-django-maas (= 1.9.4+bzr4592-0ubuntu1~14.04.1) but it is not going to be installed E: Unable to correct problems, you have held broken packages.
<brendand> you might have to incrementally try to install each package and see what the root cause is
<brendand> so try to install python-django-maas, see what the problem is with that
<BoogPowell> Good afternoon. I just installed MAAS Region Controller from DVD into a VM. The install seemed to go fine, except trying to access "Nodes" and "Networks" from the web UI return "failed to detect a valid IP address from -1."
<BoogPowell> Any pointers as to what'e causing this?
<BoogPowell> Anyone?
<BoogPowell> Wow - thanks for the help! You may go back to sleep now.
<mup> Bug #1635061 opened: [2.1] Event log "load 30 more days" button is for loading 1 additional day <oil> <MAAS:New> <https://launchpad.net/bugs/1635061>
#maas 2016-10-20
<maas_noob> Hi there.. looking for a little help.. I've just setup a "vanilla" maas server following the instructions here: http://maas.io/get-started... my nodes are connected to IPMI so I reboot into pxe and the node itself seems to get a DHCP response from the maas server and proceeds to start the boot process. but then it gets stuck with: http://169.254.169.254/2009-04-04/meta-data/instance-id' failed. error.
<maas_noob> the error message says "Failed to establish new connection".
<maas_noob> after cloud-init fails out I end up at an ubuntu login screen but I don't see any nodes getting added into maas.
<mup> Bug #1635069 opened: [web UI] subnet settings are dangerously easy to change <docteam> <MAAS:New> <https://launchpad.net/bugs/1635069>
<mup> Bug #1635097 opened: Can't assign SSH-Keys to user via maas-cli <MAAS:New> <https://launchpad.net/bugs/1635097>
<mup> Bug #1635097 changed: Can't assign SSH-Keys to user via maas-cli <MAAS:New> <https://launchpad.net/bugs/1635097>
<mup> Bug #1635097 opened: Can't assign SSH-Keys to user via maas-cli <MAAS:New> <https://launchpad.net/bugs/1635097>
<mup> Bug #1635107 opened: maas.power Error changing power state (on) while commissioning the node <MAAS:New> <https://launchpad.net/bugs/1635107>
<mup> Bug #1635107 changed: maas.power Error changing power state (on) while commissioning the node <MAAS:New> <https://launchpad.net/bugs/1635107>
<mup> Bug #1635107 opened: maas.power Error changing power state (on) while commissioning the node <MAAS:New> <https://launchpad.net/bugs/1635107>
<sk_> hi , I have setup MAAS(version 1.9) using VM's (ubuntu 14.04). I want to deploy a juju charm on the node having storage disk attached. I have created one called sdb.
<sk_> But when i deploy the charm, I see that the status in MAAS UI us Failed Deployment for the node
<sk_> I see the below errror in the MAAS install logs
<sk_> Installing for i386-pc platform. grub-install : error: unable to identify a filesystem in hostdisk//dev/sdb; safety check cant be performed failed to install grub! Command: ['install-grub', 'tmp/tmpqwTtyG/target', '/dev/sdb]'
<sk_> can anyone please help me in debugging this issue
<Mohit> Hello
<sk_> Hi team, I deployed a charm with MAAS as cloud provider, the charm got deployed successfully, but in the installation logs on MAAS console, i see this error message : Error: /dev/sdb: unrecognised disk label
<sk_> can anyone please help me on this
<mup> Bug #1635298 opened: Ability to assign deployed MAAS nodes to a "group" or "project" would be helpful <MAAS:New> <https://launchpad.net/bugs/1635298>
<mup> Bug #1633468 changed: [Subnets page, DD] Add a warning message when Device discovery is disabled <ui> <MAAS:In Progress by newell-jensen> <https://launchpad.net/bugs/1633468>
<swjen> Looking for some help. I've setup a new MAAS server and enabled my hosts to network boot (PXE). upon booting, they get a DHCP response from maas and start the "Booting under MAAS..." process.. but I never see the new machine pop up in the MAAS machine list.
<brendand> swjen, which version?
<swjen> Hi brendand. I am running MAAS 2.0.0
<brendand> swjen, what do you see in /var/log/maas/rsyslog/ on the maas server?
<swjen> brendend: I don't see anything actually. the directory is empty.
<brendand> swjen, not even maas-enlisting-node?
<swjen> nope.. completely empty./
<swjen> brendand: if a little backstory helps, All I have done is follow the "quick start" steps here: http://maas.io/get-started. I did notice that the maas server has "maas_region_controller" installed, but it does not have "maas_cluser_controller" installed. from some googling I wasn't sure if this was necessary or not.
<spaok> does anyone know if you can set the upstream DNS and NTP for MAAS 2.0 via the maas commandline
<spaok> ah, found it
<swjen> brendand: are you still here?
<brendand> swjen, if you can get console access to the machine you should watch what that's doing. even better if you can copy it somewhere and show us
<brendand> it's a bit late for me so i might stop responding
<brendand> swjen, try back tomorrow if i don't
<swjen> brendand: thanks! I do have IPMI access and have been watching the systems that way.. difficult to copy out stuff during the process but I have taken some screen shots.. basically.. immediately after the PXE boot I see "Booting under MAAS direction..." then a bunch of stuff scrolls across the screen.. then it throws some could-init errors about establishing a new connection.. then it stops at an ubuntu  login screen..
#maas 2016-10-21
<mup> Bug #1635097 changed: Can't assign SSH-Keys to user via maas-cli <MAAS:Invalid> <https://launchpad.net/bugs/1635097>
<mup> Bug #1635097 opened: Can't assign SSH-Keys to user via maas-cli <MAAS:Invalid> <https://launchpad.net/bugs/1635097>
<mup> Bug #1635097 changed: Can't assign SSH-Keys to user via maas-cli <MAAS:Invalid> <https://launchpad.net/bugs/1635097>
<mup> Bug #1635493 opened: A wishlist to be able to destroy root filesystem after release <MAAS:New> <https://launchpad.net/bugs/1635493>
<gaurangt> hi.. I've deployed MAAS in virtual environment and testing a charm.. but it is failing to use the storage disk which is already attached to the VM (though commissioning detected the disks properly).
<gaurangt> I saw an error Error: /dev/sda: unrecognised disk label Error: /dev/sdb: unrecognised disk label
<gaurangt> Any help would be really appreciated.
<mup> Bug #1635560 opened: MAAS2: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 277: ordinal not in range(128) <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1635560>
<gaurangt> anyone tried KVM + MAAS + Juju ?
<mup> Bug #1635560 changed: MAAS2: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 277: ordinal not in range(128) <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1635560>
<mup> Bug #1635560 opened: MAAS2: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 277: ordinal not in range(128) <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1635560>
<mup> Bug #1635653 opened: Maas xenial dailies no longer able to deploy <MAAS:New> <maas-images:New> <https://launchpad.net/bugs/1635653>
<Braven> hello all
<Braven> I need help with setting up MAAS DNS setting for dhcp.
<nturner> Are 2.1 updates going to be pushed to ppa:maas/stable?
<brendand> nturner, it's in lp:maas/next
<brendand> nturner, we could push it to stable as well i suppose
<nturner> brendand: if one wanted to track 2.1 (vs. rolling on to 2.2alpha or whatever comes next), what's the best way?
<nturner> also, is the fix for https://bugs.launchpad.net/maas/+bug/1632395 in maas/next?
<brendand> nturner, maas/next is best. that will always be the latest development release
<brendand> nturner, maas/stable is what's stable for that os version
<nturner> brendand: so there's no way to track 2.1-stable on xenial?
<brendand> nturner, the way to get 2.1 on xenial atm is to use next
<nturner> tracking the dev version is great, but it could be nice to have one deployment that's tracking a stable release
<brendand> nturner, and the bug is fixed in there
<nturner> 2.1 seems pretty solid right now, but during the alpha phase, there was some breakage
<nturner> presumably that will happen again during the next dev cycle
<brendand> nturner, indeed
<nturner> so if i want to maintain a deployment that tracks 2.1, I should upgrade to yakkety and track maas/stable it sounds
<nturner> like.
<brendand> nturner, well stable is defined as 'what's stable for that os'
<brendand> eventually 2.1 will be backported to xenial, but for now it's only considered stable on yakkety
<brendand> and it's not in that ppa yet
<brendand> i say os, i mean release
<brendand> so yeah, for now if you want a deployment using 2.1 that is stable your only choice is to just use yakkety straight up
<brendand> no ppa
<nturner> brendand: ok, that's cool
<suchvenu> Hi
<suchvenu> I am trying to deploy Openstack charm from the store : https://jujucharms.com/openstack-base/
<suchvenu> I have a KVM and MAAS is configured in one guest from that box. Juju is installed and could deploy a sample charm as well.
<suchvenu> However when i deploy Openstack charm from store I get "Failed deployment" as the status
<suchvenu> The MAAS log ahows as :
<suchvenu> Oct 20 15:38:32 maascontroller maas.node: [INFO] vm3: Status transition from DEPLOYING to FAILED_DEPLOYMENT Oct 20 15:38:32 maascontroller maas.node: [ERROR] vm3: Marking node failed: Node operation 'Deploying' timed out after 0:40:00. Oct 20 15:38:37 maascontroller maas.node: [INFO] vm4: Status transition from DEPLOYING to FAILED_DEPLOYMENT
<suchvenu> Any idea on this error ? Will increasing the timeout fix the issue ?
<swjen> hi! does any know why machines wont show up in MAAS after PXE boot? I am running MAAS 2.0 on bare metal and trying to "enlist" another bare metal machine. both are on the same vlan.
<deej> Hey all, I'm running into an issue with MaaS 1.9 where nodes are getting an IP set up properly in DNS, but then getting a totally different IP via DHCP
<deej> And I'm not sure where that DHCP IP is coming from or how to get rid of it, it only seems to happen occasionally
<deej> Has anyone seen that before and/or can point me in the direction of how to get that extraneous IP out of the node definition?
<pmatulis> what exactly does the 'abort' node action do again?
<deej> pmatulis: Aborts a ongoing action, i.e. commissioning/deploying
<pmatulis> deej, does it apply to any action?
<deej> I assume so
<deej> I actually don't know for sure
<pmatulis> deej, ok
<mup> Bug #1635735 opened: MaaS 1.9 not deleting discovered addresses from commissioning <MAAS:New> <https://launchpad.net/bugs/1635735>
#maas 2016-10-22
<jwitko> hey guys if I wanted to make every discovered node automatically comission what would be the best/preferred method to do this ?
<jwitko> or is there currently no one who does this?
<gmackey> Hey folks, just a quick question. I'm looking for some better documentation on creating a custom image for MAAS. I am trying to deploy RancherOS to my enrolled nodes, and my first attempt of just dropping the .tgz into the custom images field from the maas cli didn't work.
<gmackey> I've noticed some folks have build custom centos/rhel/windows images with the maas image creator. Is this a generic image building tool, or does it have some nice features that don't correlate with unsupported OSes?
<mup> Bug #1611761 changed: [2.0 RC3 ] Unhandled Error - maasserver.models.regioncontrollerprocess.DoesNotExist <oil> <oil-2.0> <MAAS:Expired> <https://launchpad.net/bugs/1611761>
#maas 2016-10-23
<mup> Bug #1281758 changed: re-installing maas-region-controller fails <docteam> <MAAS:Expired> <maas (Ubuntu):Expired> <https://launchpad.net/bugs/1281758>
<mup> Bug # changed: 1281758, 1558752, 1566115, 1587163, 1594325, 1596428, 1597345, 1598358, 1604339
<mup> Bug #1281758 opened: re-installing maas-region-controller fails <docteam> <MAAS:Expired> <maas (Ubuntu):Expired> <https://launchpad.net/bugs/1281758>
<mup> Bug # opened: 1281758, 1558752, 1566115, 1587163, 1594325, 1596428, 1597345, 1598358, 1604339
<mup> Bug #1281758 changed: re-installing maas-region-controller fails <docteam> <MAAS:Expired> <maas (Ubuntu):Expired> <https://launchpad.net/bugs/1281758>
<mup> Bug # changed: 1281758, 1558752, 1566115, 1587163, 1594325, 1596428, 1597345, 1598358, 1604339
<junaidali> Hi, i'm facing an issue where enlisting of new nodes and commissioning of enlisted nodes fails with the error: http://imgur.com/a/kDitJ
<junaidali> any idea how can i solve this?
<PCdude> mmcc: I have a question about conjure-up, you were also involved with that project right?
<PCdude> I know this is the MAAS channel, but I only saw you idle in here
<jwitko> junaidali, you have connection errors
<junaidali> jwitko, yup i was facing the issue
<junaidali> couldn't figure out what was the cause, so i purge maas and reinstalled which worked for me :)
<junaidali> purged*
<junaidali> jwitko, i was able to access maas ip
<junaidali> while facing this issue
<junaidali> was there any better solution for that?
<jwitko> I don't know what went wrong so I don't know
#maas 2017-10-16
<damaya> Question: Do I need to manually add nodes, or should enlistment automatically add them so that they show up under the nodes tab as machines?
<damaya> I see the systems in "Device Discovery" (they have no OS, so I'm just seeing the IPMI IPs). From there I can click them and add as a device or interface. They are not showing up under nodes. The only node in nodes was manually added by me.
<damaya> Which leads me to my next question: If I do have to add each node manually, then do I also need to know the MAC address for its interface as well?
<mwe1> damaya: you have initial reboot the machines with PxE-Boot via MaaS.
<mwe1> If the machine starts via PxE-boot it enables a maas user for ipmi and is available for commissioning
<parlos> damaya, there is more than one way to do it (AFAIK). You can add the nodes manually, but you need the correct info (quite alot). Best is, as mwe1, proposes. Let the nodes PXE boot.
<parlos> If the PXE boot does not work, your will have problems. Then if you added the nodes manually, you done the work in vain (unless its scripted).
<plumo> Hi everyone,
<plumo> is anyone able to use the MAAS API? I'm unable to find any documentation regarding how am I suppose to authenticate against my server.domain.tld:5240/MAAS/api/2.0/ endpoint.
<plumo> anyone?
<mup> Bug #1723944 opened: [2.2.2] Node auto-assigned address is not always shown while in rescue mode <MAAS:New> <https://launchpad.net/bugs/1723944>
<roaksoax> plumo: https://docs.ubuntu.com/maas/1.9/en/api-authentication
<plumo> roaksoax: Don't you got a not programing example? Using cURL for instance ^^, I'll adapt the code one to a human readable one.
<mup> Bug #1723955 opened: [doc] API authentication is missing from 2.x docs <MAAS:Triaged by morrisong> <https://launchpad.net/bugs/1723955>
<mup> Bug #1723978 opened: [2.3b2, UI] I cannot filter the list by Pods <MAAS:Triaged> <https://launchpad.net/bugs/1723978>
<mup> Bug #1724001 opened: [2.3b1, UI] The list of Events has too much space <ui> <MAAS:New> <https://launchpad.net/bugs/1724001>
<mup> Bug #1724002 opened: [2.3b1, UI] Swap the date and the event name in the Events list <ui> <MAAS:Triaged> <https://launchpad.net/bugs/1724002>
<mup> Bug #1724005 opened: [2.3b1, UI] The requested sequence of items on top navigation has changed <MAAS:New> <https://launchpad.net/bugs/1724005>
<mup> Bug #1724007 opened: [FUJ, UX] Improve the experience around DHCP configuration when landing on FUJ or the application for the first time <MAAS:New for m-vrachnis> <https://launchpad.net/bugs/1724007>
<saudk_> ~`.
<mup> Bug #1724096 opened: Unable to delete a user via CLI <MAAS:New> <https://launchpad.net/bugs/1724096>
#maas 2017-10-17
<bdx> hey whats up guys
<bdx> SOS https://imgur.com/a/2pm0g
<bdx> here's the bug https://bugs.launchpad.net/maas/+bug/1724111
<mup> Bug #1724111 opened: instances fail to enlist <MAAS:New> <https://launchpad.net/bugs/1724111>
<bdx> roaksoax:^
<bdx> possibly thats a cloud-init bug?
<bdx> hmmm
<bdx> sorry for blowing you up if it is
<bdx> everyone
<roaksoax> bdx: seems like the data source was not found
<roaksoax> bdx: which could mean that the IP being given to use as a datasource, is not correct
<roaksoax> bdx: check that /etc/maas/rackd.conf has a correct IP address the node can contact
<bdx> I see
<roaksoax> bdx: other than localhost
<bdx> roaksoax: yeah ... so this is what I thought ... the pxe boot network isn't the same as the api/web endpoint network
<bdx> possibly I can get access to the router/firewall and allow access between the pxe and mgmt nets
<roaksoax> bdx: yeah as long as the machines can reach it, it should be good to go
<bdx> yeah I knew the network that they were pxe booting on didn't have access to the network where the web_url lives, I guess I assumed the node would phone home on the same network it had pxe booted on, but yeah it makes sense, the api isn't listening there
<roaksoax> bdx: yeah, the way it works is base don the IP the rack knows of the region
<bdx> got it, that makes sense
<bdx> roaksoax: thanks
<mup> Bug #1724111 changed: instances fail to enlist <MAAS:Invalid> <https://launchpad.net/bugs/1724111>
<bdx> roaksoax: is there anyway I can configure the snap to have a separate pxe net?
<mup> Bug #1711760 opened: [2.3] resolv.conf is not set (during commissioning or testing) <MAAS:Fix Committed by andreserl> <cloud-init (Ubuntu):Confirmed> <resolvconf (Ubuntu):In Progress> <https://launchpad.net/bugs/1711760>
<mup> Bug #1724155 opened: [2.3b1, UI] In the hardware test list, when a test doesn't have metrics remove the chevron <ui> <MAAS:New> <https://launchpad.net/bugs/1724155>
<mup> Bug #1724158 opened: [2.3, UI] In hardware test list move the chevron to the end of the row to follow the pattern <ui> <MAAS:New> <https://launchpad.net/bugs/1724158>
<mup> Bug #1724181 opened: maas-cli missing dependencies: netifaces, tempita <cpe-onsite> <maas (Ubuntu):New> <https://launchpad.net/bugs/1724181>
<mup> Bug #1724181 changed: maas-cli missing dependencies: netifaces, tempita <cpe-onsite> <maas (Ubuntu):New> <https://launchpad.net/bugs/1724181>
<mup> Bug #1724181 opened: maas-cli missing dependencies: netifaces, tempita <cpe-onsite> <maas (Ubuntu):New> <https://launchpad.net/bugs/1724181>
<mup> Bug #1724235 opened: [2.3, HWTv2] Aborted test should not show as failure <MAAS:New> <https://launchpad.net/bugs/1724235>
<mup> Bug #1724236 opened: not possible to restart a service while a browser session is open <MAAS:New> <https://launchpad.net/bugs/1724236>
<mup> Bug #1724181 changed: maas-cli missing dependencies: netifaces, tempita <cpe-onsite> <MAAS:New> <MAAS 2.2:New> <https://launchpad.net/bugs/1724181>
<mup> Bug #1724236 changed: not possible to restart a service while a browser session is open <MAAS:Invalid> <https://launchpad.net/bugs/1724236>
<mup> Bug #1724181 opened: maas-cli missing dependencies: netifaces, tempita <cpe-onsite> <MAAS:New> <MAAS 2.2:New> <https://launchpad.net/bugs/1724181>
<mup> Bug #1724252 opened: maas is unable to discover a bonded interface on a region controller if interfaces comprising it are 'disconnected' in the database and do not belong to a fabric <cpe-onsite> <MAAS:Triaged> <https://launchpad.net/bugs/1724252>
<bdx> how can I assign a node to a zone?
<bdx> is there a config in the ui somewhere, or do I need to use the api?
<bdx> or cli
<roaksoax> bdx: you can edit hte node and change the zone
<bdx> roaksoax: really, excuse my blindness
<bdx> omg
<bdx> its the first param at the top
<bdx> arrrggggg
<bdx> roaksoax: thx
<roaksoax> :)
<TJ-> Is it possible to have Zone -> Sub-Zone hierarchy ?
<roaksoax> TJ-: no
<TJ-> Grrr. Can you suggest an alternative way I could design a deployment where I've got multiple power controllers and want 1 zone per controller, but want all the machines on those controllers to be part of a service-specific Zone too? Region seems not to be what I need since it runs a DB (which is elsewhere)
<roaksoax> TJ-: you can use tags
<TJ-> roaksoax: Aha. And they can be treated as groups for deploying, for example, HA, services ?
<mup> Bug #1718016 changed: MAAS failed to respond to POST'd deploy request but still deployed node <cdo-qa> <cdo-qa-blocker> <foundations-engine> <internal> <MAAS:Invalid> <MAAS 2.2:Invalid> <https://launchpad.net/bugs/1718016>
<roaksoax> TJ-: yeah, I would group per zone for HA services. e.g. 3 Zones (AZ's)
<TJ-> roaksoax: it quickly gets to be a brain-twister, figuring out the logical relationships :)
<TJ-> one other question if I may. Starting from scratch, a single machine that has the MAAS services aboard, is it possible later to bring that machine into the care of MAAS, or does that machine have to stay outside the managed machines? Maybe it's a 'device' rather than a 'machine' ?
<roaksoax> TJ-: hehe, well, yeah the ZONES will be soon renamed to AZs to better clarify that they are availability zones. In that case, you will have machines in different zones for HA, and you can tag machines to sub-group them. e.g. 'storage' tags, and you will have machines with such tag across all zones for HA
<roaksoax> TJ-: that's what we normally do
<roaksoax> TJ-: on the latter question, what do you mean exactly?
<TJ-> roaksoax: well, in a brand new deployment we need to boot-strap the MAAS installation. So we start with a virgin 16.04 server install, add MAAS (regiond and rackd) and configure. Then we start discovering devices/machines and so forth. Those are then managed/known by MAAS, but can MAAS be aware of its own regiond/rackd hosts
<TJ-> roaksoax: in the power example, for example we have a CDU per rack, so if that's an AZ, a 'machine' on that CDU will have rackd. Does MAAS treat that machine differently to the machines 'under' it in the rack?
<roaksoax> TJ-: MAAS will automatically make the machine a rack controller
<roaksoax> TJ-: in 2.3, you could even deploy a rack controller yourself
<roaksoax> TJ-: or if you deploy ubuntu on it yourself, and then you manually install a rack controller and point it to the same maas that deployed it, MAAS will automatically make that machine a rack controller
<TJ-> That's the answer I was after, thanks
<roaksoax> TJ-: what version of MAAS are you using though ?
<TJ-> Not deployed yet so that's open to choice :)
<TJ-> roaksoax: the question came about because I got to pondering the chicken-and-egg scenario of a completely new deploy of equipment
<roaksoax> TJ-: well MAAS 2.3 already supports deploying rack controllers on its own
<roaksoax> TJ-: however, it will use the snap instead of debian packages
<roaksoax> TJ-: but you can use 2.2, which is the latest stable and if you install the rack yourself, maas will handle the transitition
<TJ-> Thanks. Now I'm clearer now. After reading the Concepts and Terms and other Docs these questions weren't answered
<mup> Bug #1724329 opened: curtin: Installation failed <MAAS:New> <https://launchpad.net/bugs/1724329>
<mup> Bug #1724401 opened: cannot exit rescue mode <MAAS:New> <https://launchpad.net/bugs/1724401>
<mup> Bug #1724402 opened: no output for failing test <MAAS:New> <https://launchpad.net/bugs/1724402>
#maas 2017-10-18
<bdx> I'm hitting some issues -> https://imgur.com/a/u8QCD
<bdx> I don't even know where to start debugging
<bdx> I've deleted/re-enlisted the servers, 0'd the block devices, tried what seems to be everything I can think of
<bdx> I can't seem to get past what you see ^
<bdx> I probably just need to take a step back, and a breather, but I'm having a really hard time getting any introspection into these issues
<bdx> I've filed bugs on what I seem to be hitting
<bdx> I figured I would come around asking here before I waste another day
<bdx> I guess the bugs are a good start
<bdx> if anyone knows how to get some info on the commissioning and/or rescue mode errors from ^
<bdx> please share
<bdx> thx
<mup> Bug #1724553 opened: maas shows negative resources in pods <MAAS:New> <https://launchpad.net/bugs/1724553>
<mup> Bug #1724556 opened: MAAS not configuring proper IPMI passwords <MAAS:New> <https://launchpad.net/bugs/1724556>
<mup> Bug #1724556 changed: MAAS not configuring proper IPMI passwords <MAAS:New> <https://launchpad.net/bugs/1724556>
<mup> Bug #1724556 opened: MAAS not configuring proper IPMI passwords <4010> <MAAS:New> <https://launchpad.net/bugs/1724556>
<[Kid]> what is the easiest way to have maas keep a server powered on during the first discovery?
<[Kid]> i have booted several servers on and they pxe booted, but then they powered off. they are in MAAS, but i don't know which auto-generated name is which server.
<[Kid]> there is no mapping
<catbus> [Kid]: the only way I know is https://docs.ubuntu.com/maas/2.1/en/troubleshoot-faq#debugging-ephemeral-image, but would knowing MAC addresses of the nodes help you map them to the physical server? If so, you can click on 'mac address' on the nodes overview page to switch the display from node name to node mac address.
<bdx> hello all, I'm getting "00-maas-06-get-fruid-api-data" failed on all nodes
<bdx> its causing commissioning to fail
<bdx> what's odd, is there is a comment in the code around that that indicates not to fail commissioning if the test fails
<bdx> any idea whats going on here?
<mup> Bug #1724618 opened: [2.3b1, UI] I should be able to edit Tags in place <nys-feedback> <ui> <MAAS:New for m-vrachnis> <https://launchpad.net/bugs/1724618>
<mup> Bug #1724624 opened: failing commissioning script <MAAS:New> <https://launchpad.net/bugs/1724624>
<mup> Bug #1724627 opened: 00-maas-06-get-fruid-api-data fails relentlessly, causes commissioning to fail <MAAS:New> <https://launchpad.net/bugs/1724627>
<bdx> is there a way to revert maas database migration?
<bdx> bdx: nm^
<[Kid]> catbus, yes MAC would be awesome
<[Kid]> thanks!
<[Kid]> i completely missed that
<[Kid]> is there a way to bulk edit, or is it mainly scripting?
<catbus> [Kid]: edit what? the node name?
<catbus> [Kid]: you might be looking for https://docs.ubuntu.com/maas/2.1/en/manage-cli
<[Kid]> yes, the node name, the interface IP address, the IPMI settings?
<mup> Bug #1724677 opened: [critical] TFTP back-end failed right after node repeatedly requests same file via tftp <cdo-qa> <cdo-qa-blocker> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1724677>
<catbus> [Kid]: The interface IP addresses should be managed by MAAS DHCP and IPMI settings are automatically detected and credentials are created in enlistment. What's the need to change them?
<catbus> [Kid]: certainly the node names can be scripted to change in bulk via cli.
<[Kid]> catbus, i have Dell iDRAC and i noticed it did pull the right IP, but there is no user of maas on the Dell side
<[Kid]> so it couldn't manage it
<[Kid]> and what if i want static IPs?
<[Kid]> these are servers.
<[Kid]> should be a pretty common item i would think. sorry if that sounds rude, i am not trying to be
<catbus> [Kid]: in the enlistment phase, maas should be able to create the 'maas' user account on the BMC. When it fails, it's almost because the node has problem accessing the internet to pull down packages to do so.
<[Kid]> yeah, i was getting a connection time out when it was trying to connect to BMC it said
<[Kid]> ohhh, i think i know what it was
<[Kid]> i changed the Dell to allow IPMI over LAN
<[Kid]> but i also changed the username and password in the IPMI settings in MAAS
<[Kid]> and then it was able to control
<catbus> [Kid]: I can't tell from your description if MAAS was able to create 'maas' username on BMC when it first pxe boots the node.
<catbus> maas supports ipmi 1.5 and 2.0 and I believe it works with dell idrac out of box, meaning you don't have to change the bmc username/password, once machine pxe boot in the network and maas picks it up, maas can power on/off the node as it needs for commissioning and OS deploying.
<catbus> [Kid]: my understanding is maas is deigned to provide cloud style provisioning for physical servers, hundreds of it ie minimal manual intervention as possible, the common use case is to use maas to manage the IP addresses for the cluster.
<catbus> designed
<damaya> Question regarding Global Kernel Parameters in the MAAS GUI: Do these params have any impact on PXE booting or are they just set in place for the installed OS?
<damaya> MAAS GUI = MAAS Web Frontend.
<kevingc> What is the recommended network topology for a MAAS region controller and slaves?
<kevingc> I currently have them all connected to the same router, with the WAN port on the router connected to a core router. Is this correct?
#maas 2017-10-19
<damaya> Is there any way to modify the pxe boot menu (typically this would be pxelinux.cfg/default)?
<mup> Bug #1724904 opened: Changing PXE lease in DHCP snippets global sections does not work <MAAS:New> <https://launchpad.net/bugs/1724904>
<xygnal> what is the order/process when maas net
<xygnal> oops, net boots a system for build.
<xygnal> we are seeing what looks like successful pxe where it claims its under maas control
<xygnal> yet it hangs before finishing boot
<xygnal> roaksoax: ping
<xygnal> we get as far as Loading ubuntu/amd64/ga-16.04/xenial/daily/boot-initrd...ok
<xygnal> then it hangs
<mup> Bug #1724973 opened: [2.3]package-repositories should be searchable by name <MAAS:New> <https://launchpad.net/bugs/1724973>
<mup> Bug #1724989 opened: Deployment fails if server's EFI variable storage is full <MAAS:New> <https://launchpad.net/bugs/1724989>
#maas 2017-10-20
<mup> Bug #1724556 changed: MAAS not configuring proper IPMI passwords <4010> <MAAS:Invalid> <https://launchpad.net/bugs/1724556>
<TheGreatDoc> Hi all!
<TheGreatDoc> Im having some troubles using maas to provision a Dell C6100
<TheGreatDoc> Can anyone give me a hint on that?
<TheGreatDoc> Is there any way to "clean" or "wipe" the discovery database so I can start again without reinstalling ubuntu/maas?
<TheGreatDoc> I have the devices discovered but with wrong address, dunno why
<TheGreatDoc> Also, dont know if you have something to do, but the search in the mass.io/docs doesnt work
<TheGreatDoc> How can I debug this: 1 node cannot be powered on. To proceed, update your selection.
<TheGreatDoc> ipmipower -D LAN_2_0 -h 10.101.0.191 -u maas -p CxquVbbxQIfl --stat <-- This works from console, also --on and --off
<TheGreatDoc> but from maas gui is impossible. I've tried dedicated and shared ipmi interface, both without success and with the same error
<TheGreatDoc> Also, I dont get any cores/ram/disk/storage info in the maas gui
<TheGreatDoc> And there is nothing in /var/log/maas/maas.log about this
<[Kid]> if i wanted to change IP or hostname, do i have to do a full re-deploy or will MAAS detect it and change it automatically?
<Miouge> My MAAS server has 2 interfaces: ens3 (internet access) and ens8 (where my baremetal servers are), I have the DHCP working on ens8, but TFTP ansers only on ens3 and localhost. What am I missing here?
<Miouge> Mmm I tracked it down to ens8 using 169.254.0.0/16 which is considered âUnconfiguredâ by MAAS :(
<[Kid]> so i changed hostnames and it wouldn't let me re-deploy. It looks like it is making me go through the entire commission and deploy process. is that the easiest way to do it?
<edmz> is it possible to try MAAS with Virtualbox?
<edmz> I've only seen a couple of very old tutorials about that (ca 2012). And some of the stuff that they have is not in the latest MAAS. (Like Clusters)
#maas 2017-10-21
<TheGreatDoc> Hi all. How can I log in to a ephemeral image with maas 2.2? The backdoor cheat for 1.9 doesnt work for 2.2
<TheGreatDoc> Finally found it. Little hard to find in the 2.2 documentation. Maybe a link in the troubleshooting directly to https://docs.ubuntu.com/maas/2.2/en/nodes-hw-scripts with a bold in the needed username would be helpful for other users
<TheGreatDoc> Finally got the error, but is a little bit confusing. The smartctl-validate script error output is about Dialog
<TheGreatDoc> debconf: unable to initialize frontend: Dialog
<mup> Bug #1725770 opened: focus scroll to nodes table <MAAS:New> <https://launchpad.net/bugs/1725770>
<mup> Bug #1725770 changed: focus scroll to nodes table <MAAS:New> <https://launchpad.net/bugs/1725770>
#maas 2017-10-22
<mup> Bug #1725908 opened: deleting user with static ip mappings throws 500 <MAAS:New> <https://launchpad.net/bugs/1725908>
