#maas 2012-07-16
<roaksoax> rvba: howdy!
<rvba> Hi roaksoax.
<roaksoax> rvba: so bug #1021382
<ubot5> Launchpad bug 1021382 in maas (Ubuntu) "The COMMISSIONING_SCRIPT setting uses a relative path." [Critical,Confirmed] https://launchpad.net/bugs/1021382
<roaksoax> rvba: I hav enot yet done an SRU because doing so means that when on upgrade
<roaksoax> it will request to upgrade the config file, if the user selects 'No' things will be broken
<roaksoax> rvba: does maas define defaults for the path in question?
<rvba> roaksoax: what was fixed is the default.
<roaksoax> rvba: right, is default in the config, which means tht econfig has changed, whichs means possible broken updates
<rvba> roaksoax: the package question you're referring to concerns the user-defined config right?
<roaksoax> rvba: yes
<rvba> Well, if the user has defined a specific path then that will override the new value but things won't be broken.
<roaksoax> rvba: right, but my concern is that we shouldn't be SRU/ing config updates
<rvba> roaksoax: well, all I know is that default config settings are really part of the application... but you know more about SRU than I do.
<rvba> What other options do we have?
<roaksoax> rvba: I was thinking maybe that the code can test the path, if it is relative, make it absolute
<roaksoax> rvba: but, let me lot into it further, actually prepare the package, do some more testing and see what are the effects of it
<roaksoax> allenap: howdy! what's the status of python-tx-tftp?
<allenap> otp
<roaksoax> k
<allenap> roaksoax: Hi! Right, I still have to complete the upstream work. I'm going to do that this evening. So... not landed yet. Sorry.
<roaksoax> allenap: no worries :). Do you have a patch I can apply in the mean time?
<roaksoax> i missplaced the one you gave me last time
<allenap> roaksoax: Coming up...#
<allenap> roaksoax: http://paste.ubuntu.com/1095124/ should be good, if my git fu is up to it.
<roaksoax> allenap: thanks
<roaksoax> allenap: btw.. I'm also guessing that the celeryconfig import from the tftp server is not yet in trunk?
<allenap> roaksoax: No, not yet... too many things to do :) It's in progress though, but it's taking quite a lot more work that was at first obvious. I /could/ have hacked it, but it would have come back to haunt us (indeed, using celeryconfig was a hack in the first place).
<roaksoax> yeah
<roaksoax> allenap: i don't know whether that's the issue, but i've seen a weird behaviour on which maas-pserv fails to start
<allenap> Very possibly related.
<roaksoax> allenap: but when just adding a raise in the import, it works. However, there's no output
<roaksoax> so I can't really determine what the issue is
<allenap> That's really odd. Removing the weirdness with celeryconfig will remove one more moving part. I hand-wavingly hope that'll help.
<roaksoax> yep there's definitely something weiird going on
<guimaluf> I have a node stuck at "Booting..." for a long time. It is doing something, as I can check at /var/log/maas/rsylog/node/message, but it is very strange, all nodes I've installed never take so long!
<guimaluf> anyone have any idea why one of my nodes hangs on "Booting..."?!
<burnbrighter> NickServ identify sweetpea1
#maas 2012-07-17
<jtv> smoser, could you help me with this?  I need maas to retrieve some settings that are debconf'ed by the maas-dhcp postinst script.
<jtv> roaksoax maybe?
<jtv> rvba: if we have dev_fixture.yamls and initial_data.yamls, a dev setup will load both and a production setup will only load the latter, right?
<jtv> Oh no, please tell me this is not true.
<jtv> User.email must be unique, even if it's blank.
<jtv> What #$@% use is that?
<rvba> jtv: How would dev_fixture.yamls be loaded by the dev service?
<jtv> I thought a demo run has all the data from dev_fixture.yaml..?
<rvba> jtv: no, make sampledata loads dev_fixture.yaml
<jtv> Well that's what you do for a demo run, so a dev setup will load both, right?
<rvba> Yes.
<jtv> Well that exactly answers my question.  :)
<rvba> initial_data.yamls is loaded automatically but not dev_fixture.
<rvba> Ok then :).
<jtv> Unfortunately I'm running up against this weirdness where we can't have two built-in users with email address '', even though we can have one.
<rvba> I thought we "fixed" that... let me have a look at the code and refresh my memory.
<jtv> Yes, South, I'm making a field nullable.  And, since you seem to insist, specifying a default value of None.  Now, honestly, what do you think I want the default to be?  Do you still need to stop because you don't know?
<jtv> Ah.  Thank you internet.  This is South's way of saying: what if I ever want to _reverse_ this migration?
<jtv> rvba: I don't think we have the email-may-be-empty-but-must-still-be-unique problem solved.  :(
<rvba> jtv: really?  IIRC we wanted to use that only to be able to create a system user with an empty email address.
<jtv> Yes
<jtv> That's what I'm trying to do.
<jtv> But we already have one with an empty email address.
<jtv> I have a feeling that the solution we found at the time was to use a blank string, and leave the rest of the problem for the future.
<jtv> Providing a null violates a not-null constraint, providing an empty string violates a unique constraint.
<jtv> Ahhh looks like django 1.4 makes the email address optional.
<jtv> Or maybe that's only in the check that creates it, rather than in the database.
<rvba> But unfortunately we have to be compatible with Dj 1.3.
<jtv> I'm not even sure 1.4 solves it anyway.  :(
<jtv> What I found was about create_user:
<jtv> âThe username parameter is now checked for emptiness and raises aÂ ValueErrorÂ in case of a negative result.â
<jtv> And by a negative result in this case they mean a positive result.
<jtv> Off-by-True error.
<jtv> What's the system username we create?  Just maas?  I could just use maas@localhost.
<jtv> Althoughâ¦ what do we do for the _next_ system user?  I'll just make it a full description of the user, and a nonexistent address. TFB.
<roaksoax> jtv: sure, how do you want to retrieve them?
<jtv> roaksoax: never mind, I think I'll just leave that as a packaging job, something postinst can do.
<roaksoax> jtv: cool
<roaksoax> rbasak: howdy! this is what we need to do in setup.py: http://bazaar.launchpad.net/~andreserl/maas/maas_updates_bzr745/revision/56
<roaksoax> errr
<roaksoax> rvba: ^^
<roaksoax> rbasak: sorry worng person :)
<rbasak> np
<roaksoax> rvba: debian/rules mainly
<rvba> roaksoax: looking.
<roaksoax> rvba: override_dh_auto_build
<roaksoax> rvba: i just figured it was faster and less painfull for me to build it in package build time rather than copy it manually. But we need to handle it in the setup.py
<rvba> roaksoax: looks all right time.
<rvba> roaksoax: looks all right to me.
<rvba> :)
<roaksoax> rvba: :)
<roaksoax> jtv: so now that maas-import-isos references are being erased, does this mean that we are now using MAAS own PXE server?
<jtv> Not quite there yet.
<jtv> For that we need to assume control of DHCP.
<jtv> Or the administrator needs to configure their DHCP for MAAS.
<jtv> Currently we still use Cobbler.
<roaksoax> jtv: right, ok then. So do you have an ETA of when this would happen (getting rid of cobbler)?
<jtv> We're working on itâ¦ the devil is very much in the details.
<jtv> To a large extent the plan unfolds as we execute it.
<jtv> It's the price we pay for getting a quick start using Cobbler.
<roaksoax> hehe
<roaksoax> jtv: alright, I was just wondering so that I can do the appropriate packaging updates then, and not now :)
<jtv> I'm afraid we're going to go through some stages of brokenness before we come out the other end working but without Cobbler.
<roaksoax> i guess that's fine.
<roaksoax> I just want to be aware when cobbler is not longer being used so I don't release broken stuff
<roaksoax> allenap: how would you describe the fixes for python-tx-tftp?
<allenap> roaksoax: It adds tsize support; it already has timeout support, so this should make it fully comply with RFC 2349.
<allenap> pxelinux requires tsize support.
<roaksoax> allenap: ok cool, just updating patch description :)
<allenap> roaksoax: Fwiw, here's the pull request: https://github.com/shylent/python-tx-tftp/pull/5
<roaksoax> thanks
<cropalato> Hi, is there some documentation about configure maas with a external dhcp?
<guimaluf> cropalato, iis the same thing. the difference is allow the dhcp to redirect the pxe server:
<guimaluf> allow bootp;
<guimaluf> allow booting;
<guimaluf> class "pxeclients" {
<guimaluf>         match if substring(option vendor-class-identifier,0,9)="PXEClient";
<guimaluf>         # Servidor de PXE
<guimaluf>         next-server 150.164.3.236;
<guimaluf>         server-name "brontes";
<guimaluf>         filename "pxelinux.0";
<guimaluf> }
<cropalato> guimaluf, thanks
<burnbrighter> It appears in the latest release of maas, the default route needs to be set manually? Is that a bug or by design?
<burnbrighter> latest apt-get release rather
<roaksoax> smoser: ping
<roaksoax> smoser: https://bugs.launchpad.net/ubuntu/+source/cobbler/+bug/858867/comments/5
<ubot5> Ubuntu bug 858867 in cobbler (Ubuntu Quantal) "XMLRPC allows unauthed users access to various methods (which it shouldn't) " [Medium,Triaged]
<roaksoax> burnbrighter: what's your issue?
<roaksoax> burnbrighter: what do you mean by default route?
<burnbrighter> roaksoax: on enlist, it appears the default route isn't being configured
<roaksoax> burnbrighter: you mean it is no enlisting?
<burnbrighter> on pxe boot, the default route is not being sent
<burnbrighter> maybe its cobbler not sending the info correctly
<burnbrighter> or setting up the configuration?
<burnbrighter> this worked in my previous maas release
<roaksoax> burnbrighter: so if machines are not PXE booting, it is a matter of configuring the DHCP correctly.
<roaksoax> burnbrighter: if they are PXE booting but not enlisting, maybe the IP address the MAAS server has is not the one in the configuration
<burnbrighter> it PXE boots, but then it's not picking up the default route
<burnbrighter> so as it does its initial configuration and is doing the install
<burnbrighter> its not getting a default route via dhcp
<burnbrighter> it gets an IP address, but no default route
<roaksoax> burnbrighter: ah so that's a problem of the DHCP server or even of the installer itself
<burnbrighter> correct - but that's configured by maas
<burnbrighter> right?
<burnbrighter> or maas -> cobbler
<roaksoax> burnbrighter: maas attempts to configure it, and tells it to cobbler
<burnbrighter> ok, so something broke there
<roaksoax> burnbrighter: so what's the pastebin from /etc/cobbler/dnsmasq.template
<roaksoax> q1
<burnbrighter> sure, hang on
<burnbrighter> sorry, what is ubuntu's pastebin?
<burnbrighter> url
<roaksoax> pastebin.ubuntu.com
<roaksoax> burnbrighter: you can install pastebinit
<roaksoax> and run: pastebinit /etc/cobbler/dnsmasq.template
<burnbrighter> http://paste.ubuntu.com/1097228/
<roaksoax> burnbrighter: is this your default gateway? dhcp-option=3,172.16.100.11
<burnbrighter> yes
<roaksoax> burnbrighter: run sudo cobbler sync and pastebin the output please
<roaksoax> burnbrighter: so maas has a caching server, would you be hitting it maybe?
<burnbrighter> sorry - need to step away for a bit , will be back
<roaksoax> sure
<guimaluf> i give up of this maas stuff!!! past one week I couldnt make it work!
<guimaluf> i've sync the clocks. changed ephemeral images. raw configs. nothing works!
<roaksoax> guimaluf: i would say otherwise, I have had various deployments of MAAS working
<roaksoax> guimaluf: what are your issues anyway?
<guimaluf> (0x7f33b5f30700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:47231] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
<roaksoax> guimaluf: that seems to be juju issue, not maas
<guimaluf> this time. I've pass so many things... everytime clock problem.
<guimaluf> but maas cant deploy the ssh-key properly
<roaksoax> guimaluf: edit /var/lib/cobbler/kickstarts/maas.preseed and search for the passwrod line and remove the ! and set your password there
<roaksoax> guimaluf: then you would be able to ssh in an debug the problem of why the keys are not being installed
<guimaluf> it install the os but I needed to include a default password by myself and even after deploy the sshkey by hand i got this juju problem
<guimaluf> I check everything on web, and the problem is the same. time sync oauth clock problem
<guimaluf> your maas deployments are working properly? deploy sshkeys and everything else?
<roaksoax> if that's the issue then would be matter of making maas' server clock the same as the clients. We are currently working on trying to get that fixed
<roaksoax> guimaluf: yes my maas deployments work fine. I've tested the ssh key inyection in various escenarios as well
<guimaluf> roaksoax, I fix it in many ways.
<guimaluf> maybe the problem occurs because the maas server isnt the dhcp server
<roaksoax> smoser: ^^ oauth issues causing ots of pain, is there a recommended fix?
<guimaluf> now even with the keys set properly this juju thing dont work :/
<guimaluf> i'm really tired :( whole week just for this.
<guimaluf> i've installed so many times, so many times!
<smoser> work around is described in comment 5 at http://paste.ubuntu.com/1097303/
<smoser> err.
<smoser> comment 5 at https://bugs.launchpad.net/maas/+bug/978127
<ubot5> Ubuntu bug 978127 in MAAS "incorrect time on node causes failed oauth" [High,Triaged]
<guimaluf> sorry my whinning
<roaksoax> guimaluf: https://bugs.launchpad.net/maas/+bug/978127/comments/5
<roaksoax> that's the work around for the issue you are having
<roaksoax> smoser: btw... did you read the cobbler stuff?
<smoser> roaksoax, i did.
<roaksoax> smoser: any thoughts?
<guimaluf> roaksoax, I allready did that. :/
<roaksoax> guimaluf: is the juju client accessible to the nodes via DNS ?
<guimaluf> yes, I put the node address in /etc/hosts
<smoser> roaksoax, just commented in bug.
<roaksoax> guimaluf: well I think we'd need logs in order to determine what is wrong
<guimaluf> roaksoax, which ones?
<roaksoax> smoser: ok ;)
<roaksoax> guimaluf: in the bootstrap node, we'd need cloud-init logs, juju logs
<smoser> guimaluf, roaksoax is right. if you can get to the bootstrap node (via ssh), then we need to see what went wrong.
<smoser> i suspect there are errors in /var/log/cloud-init-output.log if you pastebin that
<guimaluf> smoser, I can ssh the node just because I've put the keys by hand. :/ anyway. i'm pasting, just a sec
<smoser> (just in case you were unaware, guimaluf there is a tool called 'pastebinit' that pastes to paste.ubuntu.com from file or stdin)
<guimaluf> smoser, yes, ofcourse :)
<guimaluf> cloud-init: http://pastebin.ubuntu.com/1097316/
<guimaluf> juju -v status: http://pastebin.ubuntu.com/1097319/
<smoser> guimaluf, /var/log/cloud-init-output.log ?
<smoser> your maas server seems to have given empty user-data to cloud-init.
<guimaluf> smoser, there's no file with this name
<smoser> (as evidence by 'Unhandled non-mlutipart userdata '')
<smoser> right.
<smoser> lok.
<smoser> (duh, it wouldn't be there... you didn't get any cloud-config user data to tell it to write it :)
<smoser> so now, your issue is that that node is not geting to the maas server.
<guimaluf> but this user is set in the maas.preseed?
<smoser> user?
<smoser> ok. i have to run
<guimaluf> which user are you talking about?
<smoser> you said "this user"
<smoser> i didn't know what that meant.
<smoser> ah. user-data.
<guimaluf> sorry, poor english :/
<smoser> so, to debug from here.
<smoser> the issue is that cloud-init does not get data from the maas server
<roaksoax> guimaluf: can you pastebin /etc/maas/maas_local_settings.py?
<smoser> so you need to debug that
<smoser> roaksoax, i have to run
<smoser> but there should be files in /etc/cloud.cfg.d
<roaksoax> smoser: alright thanks for the input
<smoser> that have Maas config info
<smoser> and /usr/share/pyshared/cloudinit/DataSourceMAAS.py is a maas user-data client of sorts
<guimaluf> smoser, thanks alot man! :) I will bother roaksoax a little more!
<smoser> python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/91*.cfg crawl <htat-url>
<smoser> find which file has "maas" in it in /etc/cloud/cloud.cfg.d (grep would work)
<smoser> open it.
<smoser> it has a url in it
<smoser> and do like the above
<guimaluf> roaksoax, http://pastebin.ubuntu.com/1097339/
<smoser> that will show you an error, or at least trigger one hopefully
<smoser> later.
<roaksoax> guimaluf: I think I found the issue
<roaksoax> guimaluf: is this DEFAULT_MAAS_URL = "http://150.164.3.236/" the IP address for the local network?
<roaksoax> MAAS's IP address for the local network?
<guimaluf> yes
<roaksoax> guimaluf: can you pastebin apache's log? (/var/log/apache2/error.log and /var/log/apache2/access.log)
<burnbrighter> roaksoax: sorry, now back - here is the pastebin you requested: http://paste.ubuntu.com/1097349/
<roaksoax> burnbrighter: are the client machines precise or quantal?
<guimaluf> roaksoax, http://pastebin.ubuntu.com/1097352/
<guimaluf> invalid consumer :/
<roaksoax> guimaluf:
<roaksoax> 150.164.3.240 - - [17/Jul/2012:17:37:22 -0300] "GET /MAAS/metadata//2012-03-01/meta-data/public-keys HTTP/1.1" 404 230 "-" "Python-urllib/2.7"
<roaksoax> 150.164.3.240 - - [17/Jul/2012:17:37:22 -0300] "GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404 242 "-" "Python-urllib/2.7"
<roaksoax> that seems to be an issue
<roaksoax> guimaluf: So, nodes have enlisted, they have commissioned, and they were ready to deploy ebfore using juju?
<guimaluf> roaksoax, but my ssh is there in the root preferences
<roaksoax> guimaluf: what it seems is that no cloud-init meta-data or user-data have been generated, as if the node hasn't commissioned
<guimaluf> but every step occurs properly.
<guimaluf> the node is enlisted, comissioned and ready
<guimaluf> now is Allocated to root
<burnbrighter> roaksoax: precise
<roaksoax> burnbrighter: that's strange then, it might be an issue with dnsmasq
<burnbrighter> it appears something broke between the latest release and the last because this worked with out problems before
<roaksoax> burnbrighter: when was the last time you upgraded quantal's maas version?
<burnbrighter> today - I had been not working on things for about two weeks
<roaksoax> burnbrighter: can you sudo apt-get update && sudo apt-get install python-txtftp and see if an update gets installed
<roaksoax> guimaluf: uhmm so the problme seems to be the fact that the client is not getting the meta-data / user-data from the MAAS server. Config file seems just fine
<guimaluf> roaksoax, is not a problem of time syncronization. what else can be?
<roaksoax> guimaluf: would be helpful if you juju bootstrap and pastebin the error/access log from apache while the node is PXE booting and installing, and what happens after install
<roaksoax> guimaluf: and check whether the node is complaining about not being able to reach the meta-data server (client node)
<guimaluf> roaksoax, i think the Invalid Consumer in the error.log happen when I was installing the nodfe
<roaksoax> guimaluf: ok, so do you have physical access to the client maas node?
<guimaluf> yes
<roaksoax> guimaluf: do you have a screen connected to it?
<guimaluf> yep
<guimaluf> should I flush maas db again and reinstall the node?
<guimaluf> I did that 1208397198237 times xD
<roaksoax> guimaluf: ok, so, juju bootstrap, after installation finishes it will reboot. Once it boots into logging prompt, it should show that 1. either is failing to get meta-data or 2. it is being able to obtain meta-data. If yo could see what shows there and let me know, that would be very helpful
<roaksoax> guimaluf: or 3. when you fluished the db, you might have flashed node information that has the cloud-init meta-data
<guimaluf> roaksoax, I didnt flushed the db this time
<guimaluf> I start from scratch again
<guimaluf> was my last try before "give up"
<roaksoax> guimaluf: so the screen output should tell if something wrong is going on
<roaksoax> it will 1. either obtain the user-data and do stuff with it (cloud-init). 2. or cloud-init will complain about not being able to reach the maas-server
<guimaluf> roaksoax, so I should reboot my node right now and check what happens before the login prompt?
<roaksoax> guimaluf: reinstall the node please
<guimaluf> ok
<guimaluf> there's a secure way for removing the node?
<guimaluf> I always do cobbler system delete and maas shell Node.objects.all().delete()
<roaksoax> guimaluf: yeah I guess that would work. however, before you do that
<roaksoax> guimaluf: could you pastebin: sudo cobbler system getks --name node-XYZ
<roaksoax> where node-XYZ is the node-XYZ from sudo cobbler system list
<roaksoax> that is your bootstrap node
<guimaluf> sure :)
<guimaluf> roaksoax, http://pastebin.ubuntu.com/1097392/
<roaksoax> cloud-init   cloud-init/maas-metadata-url  string http://150.164.3.236/MAAS/metadata/ -->that seems right, its telling that it should contact the MAAS server for meta-data
<roaksoax> guimaluf: please proceed with the reinstall
<guimaluf> ok
<guimaluf> removing nodes...!
<guimaluf> wow, I can delete the node from the UI it isnt allocated to root!
<roaksoax> yeah
<guimaluf> :)
<guimaluf> good sign!
<guimaluf> roaksoax, I should keep eye on "meta-data" information right?
<roaksoax> guimaluf: yes, it should appear right on the screen after the installation proices, after the reboot (before/after showing the logging prompt)
<guimaluf> after comissioning right?
<roaksoax> guimaluf: 1. enlist. 2. accept & commission. (turn on the node and allow it to commission) 3. juju bootstrap. 4. wait installation to finish and reboot. 5. watch :)
<guimaluf> juju bootstrap?
<guimaluf> juju bootstrap runs on the node?
<roaksoax> guimaluf: yes
<roaksoax> guimaluf: it runs the bootstrap node which is juju
<roaksoax> guimaluf: it runs the bootstrap node which is juju's zookeeper
<roaksoax> guimaluf: you always need that node to deploy with juju
<guimaluf> but I dont run manually right?
<roaksoax> if juju bootstrap doesn't start a node automatically, then you need to start it manually
<guimaluf> juju bootstrap is for the local machine?
<roaksoax> guimaluf: make sure your .juju/environments.yalm is also correctly configured
<guimaluf> roaksoax, in my maas server right?
<roaksoax> guimaluf: if you are running juju ocmmands from your maas server, then yes
<roaksoax> juju bootstrap will take 1 node from the maas pool of servers
<guimaluf> there is a error
<guimaluf> util.py
<roaksoax> huh?
<roaksoax> pastebin
<guimaluf> util.py[WARNING] 'addres.../metadata/DATE/meta-data/instance-id' failed http error [403]
<guimaluf> I cant pastebin this one
<roaksoax> that;s commissioning
<guimaluf> yes
<roaksoax> guimaluf:  that's oauth issue then i thjink
<roaksoax> please apply the patch as above
<guimaluf> which patch?
<guimaluf> still commissioning. if I reboot the node it will commit
<roaksoax> err work around for the ephemeral iomage
<roaksoax> https://bugs.launchpad.net/maas/+bug/978127/comments/5
<ubot5> Ubuntu bug 978127 in MAAS "incorrect time on node causes failed oauth" [High,Triaged]
<guimaluf> its done already
<roaksoax> maybe the image iupdateed by the cron job that's why it failed
<guimaluf> I rebooted the node, it will be commissioned now.
<guimaluf> roaksoax, after the commissioning many py errors showed up on the screen, but I couldnt get it
<roaksoax> guimaluf: can you show the log from apache? error and access please
<guimaluf> error.log is the same
<guimaluf> roaksoax, http://pastebin.ubuntu.com/1097427/
<roaksoax> guimaluf: so does the MAAS Web UI say the node is ready?
<guimaluf> roaksoax, strangely now it is allocated to root
<roaksoax> there's an error there: 150.164.3.240 - - [17/Jul/2012:18:55:55 -0300] "GET /MAAS/metadata//2012-03-01/meta-data/instance-id HTTP/1.1" 403 239 "-" "Python-urllib/
<guimaluf> yes, the same error.
<guimaluf> reboot and goes normal
<roaksoax> so it seems ot be failing commissioning
<guimaluf> now it is installing
<roaksoax> ok so lets wait and see :)
<guimaluf> this python-urllib is happening everytime, then I reboot and everything runs as expected
<roaksoax> but after it installed ubuntu
<roaksoax> before/after logging pronmt onreboot after installatio n
<roaksoax> it should show cloud-init trying to do some work
<guimaluf> no instance data found in start-init
<roaksoax> yeah that's what's failing
<roaksoax> i need to run for a bit now
<guimaluf> ok
<guimaluf> any glue or tip?
<guimaluf> any glue or hint?
<guimaluf> i'll search for that
<guimaluf> thank you very very much for everything
<guimaluf> worked!
<guimaluf> :D
<guimaluf> AEUhaUEhaUEHA
<guimaluf> LOL
<guimaluf> it works!
<guimaluf> but maybe it is because I put the authorized_keys inside the /home/ubuntu/.ssh/ of the node
<guimaluf> and i think the juju bootstrap before the installation process helped alot!
<guimaluf> roaksoax, thanks man! really thanks! :D
<burnbrighter> roaksoax: http://paste.ubuntu.com/1097463/
#maas 2012-07-18
<roaksoax> allenap: ping
<allenap> roaksoax: otp, done soon.
<guimaluf> is it possible to add maas server as a maas node? I would like to set up the maas server as the cloud-controller, glance and keystone.
<roaksoax> guimaluf: yeah you can enlist localhost, but for juju is not gonna be usefull because the maas server is the onein chanrge of turning on the machines you need
<allenap> roaksoax: Going to be a bit longer.
<roaksoax> allenap: no worries, let me know when you done :)
<guimaluf> roaksoax, so my maas server should be uniq for this purpose? cause I've a powerfull server and run only the maas server will be a waste of resources.
<roaksoax> guimaluf: yeah, unless you deploy VMs with it
<guimaluf> then should I put the maas server in a virtual machine ?
<guimaluf> deploy vms with juju?
<roaksoax> guimaluf: you could, or you could have VMs contorlled by maas
<roaksoax> guimaluf: for juju they would only be nodes
<guimaluf> hmm
<guimaluf> I can't figure out how I'll use my 16 servers in juju + openstack IaaS
<guimaluf> roaksoax, another issue: when I run juju status openstack-dashborad there is public-address:null
<guimaluf> I dont know how will I make my cloud avaible.
<guimaluf> when will my node get the hypervisor installed?
<guimaluf> I'm bit confuse :(
<guimaluf> and bothering you alot, I thin
<guimaluf> think
<roaksoax> TBH I have not done many openstack deployments with juju/MAAS
<roaksoax> so I don't know much about those bits
<roaksoax> guimaluf: someone who would be able to help you is adam_g
<roaksoax> but he is at OSCON atm
<guimaluf> :(
<smoser> well, if juju doesn't have the address, it would seem a general juju<->maas issue.
<allenap> roaksoax: What's up?
<allenap> The stuff still hasn't gone upstream.
<allenap> :)
<roaksoax> allenap: ok, so, there's something wrong with maas-pserv
<roaksoax> allenap: there's some weird thing happening. maas-pserv fails to start
<roaksoax> but then magically, it starts
<allenap> roaksoax: Do you have some logs for it?
<roaksoax> allenap: there's nothing :)
<roaksoax> allenap: can you spawn a canonicastack instace and install maas from quantal
<roaksoax> and see it for yourself to see if you have any ideas of what it might be?
<allenap> roaksoax: I wonder if this is still the celeryconfig problem?
<roaksoax> allenap: might be, I imported the path
<allenap> roaksoax: I've never used canonistack, so it might take me a while to figure it out.
<allenap> roaksoax: If it's that, I'm working on it.
<roaksoax> allenap: so upstart says twistd doesn't know who 'maas-pserv' is
<roaksoax> allenap: so it is failing somewhere when the plugin does the import
<allenap> roaksoax: Yeah, that sounds right. It can't import celeryconfig probably, but the import exception is being suppressed.
<roaksoax> allenap: I added the path in provisioningserver/plugin.py so I know it is importing thepath correctly
<allenap> roaksoax: Let me fix the celeryconfig thing. If the problem is still there then I'll spend more time on it. Will that block you?
<roaksoax> allenap: yeah we have a broken maas in quantal :)
<allenap> roaksoax: It's fairly broken in Precise too ;)
<roaksoax> allenap: heh but precise at least works
<guimaluf> roaksoax, when I expose a service I get the following error: http://pastebin.ubuntu.com/1098491/
<guimaluf> I'm  completely discouraged
<roaksoax> guimaluf: AFAIK, you cannot expose services with maas as that's like opening a port in ec2
<roaksoax> that's only a feature with ec2 i thi nk
<guimaluf> so how will I make my service avaible?
<roaksoax> guimaluf: the service is available
<roaksoax> guimaluf: if you are deploying instances with juju inside openstack, then you can use the expose command I believe
<guimaluf> roaksoax, I have onde node on my juju environment, I deployed, related and exposed many services, but where will I access my service? I cant get the public-address value, and I got this error on node juju log
<roaksoax> guimaluf: "Service exposing works by opening appropriate ports in the firewall of the cloud provider."
<roaksoax> if you are deployimng in baremetal that ncommand is not necessary
<guimaluf> but when I access my node address nothing happens
<guimaluf> all my services have agent-state:pending
<guimaluf> :/
<roaksoax> that means that they are trying to get installed or something wrong is going on there
<guimaluf> roaksoax, probabily something wrong in the node right?
<roaksoax> guimaluf: yes in the node
<guimaluf> there is nothing in /var/log/juju :/ only the error above
<guimaluf> I'll install openstack without juju :/ time is near :(
<roaksoax> guimaluf: is that on the client machines?
<guimaluf> roaksoax, yep
<guimaluf> /var/log/juju/provision.log
<roaksoax> guimaluf: there's no machine-agent.log nor provision.log?
<guimaluf> provision log http://pastebin.ubuntu.com/1098491/
<roaksoax> rvba: bug #1026228
<ubot5> Launchpad bug 1026228 in MAAS "No way to DELETE or get a node back to 'Ready' from WebUI" [Undecided,New] https://launchpad.net/bugs/1026228
<roaksoax> guimaluf: there was an error with juju apparently, do you have any cloud-init logs?
<guimaluf> roaksoax, http://pastebin.ubuntu.com/1098589/
<roaksoax> guimaluf: were you seeing this error? 2012-07-18 13:14:46,016:1882(0x7fe067770700):ZOO_ERROR@handle_socket_error_msg@1579: Socket [127.0.0.1:44068] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
<guimaluf> roaksoax, nop. only ZOO_INFO
<roaksoax> sok
<roaksoax> smoser: there seems to be an issue when importing the data source after the bootsrrap
<smoser> roaksoax, ?
<roaksoax> smoser: juju does not get installed after juju bootstrap with maas
<roaksoax> i'm redploying
<smoser> because cloud-init isn't getting its datasource
<roaksoax> smoser: it is getting the ssh keys from MAAS
<roaksoax> not the juju stuff though
<smoser> roaksoax, can you give me more info and/or access?
<roaksoax> smoser: is in my local environment, let me finish re-bootstrapping
<roaksoax> smoser: apache: http://pastebin.ubuntu.com/1098663/
<roaksoax> smoser: ah never mind I think I found the issue
<roaksoax> it seems to be proxy
<roaksoax> smoser: for some reason sources.list ends up with ubuntu-mirror
<roaksoax> smoser is that done by cloud-init?
<smoser> roaksoax, yes.
<smoser> you have broken dns
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/974509
<ubot5> Ubuntu bug 974509 in cloud-init (Ubuntu) "cloud-init selects wrong mirror with dns server redirection" [Low,Confirmed]
<smoser> cloud-init noticed that you had a host on your network (named 'ubuntu-mirror') and assumed that you wanted to use that mirror.
<roaksoax> smoser: I don't this is MAAS on a VM
<smoser> your issue is that you also have a host named "roaksoax-uses-a-broken-dns-provider-that-does-dns-hijacking"
<roaksoax> dns/dhcp is libvirt
<smoser> try it.
<smoser> thats the problem. cloud-init does a dns lookup and its getting a valid response for that 'ubuntu-mirror'
<smoser> (rather than a not found response)
<roaksoax> smoser: right, but it is libvirt the one controlling DNS
<smoser> well libvirt is letting dns pass through
<smoser> dnsmasq is cacheing only for libvirt.
<smoser> change your hosts dns to use 8.8.8.8 and restart libvirt.
<smoser> (note, it is annoying, and something that will get fixed in one way or another).
<roaksoax> smoser: did you happen to figure out how to select a particular dns server on a wifi connection at all times?
<Daviey> good luck with that
<smoser> man dhclient-script
<smoser> it would seem its possible there.
<Daviey> possible == hack
<smoser> but that will likely cause issues on public wifi
<smoser> its not a hack. its explicitly listed as how to do it.
<smoser> oh. or also
<smoser>  /etc/dhcp/dhclient.conf
<smoser> hm..
<smoser> but no, with resolvconf that wont work
<roaksoax> and againa another reason why I hate the change in resolvconf :)
<roaksoax> smoser: but anyuways, any way to disable cloud-init trying to use a mirror? What If I don't really want it to automatically use one
<smoser> you can feed it one with user-data.
<roaksoax> smoser: right, but shouldn't default behavior be not to try to do that?
<roaksoax> smoser: cause, in my cause for exmaple, is breaking my maas implementation and might be breaking others too
<roaksoax> smoser: cuase either way, that's the reason why we pass http://pastebin.ubuntu.com/1098696/ to the preseed
<roaksoax> s/to the/in the
<smoser> roaksoax, shouldn't the default behavior be to not have broken dns?
<roaksoax> smoser: then I guess we should go all the way to resolveconf :)
<smoser> but having this feature off by default would essentially defeat its purpose.
<smoser> the idea was to not have to modify the image and get magical selection of your local mirror
<smoser> i'm confused though
<smoser> oh.
<smoser> yeah, that does suck
<smoser> you preseed the install, and that gets it right
<smoser> and then cloud-init finds a better mirror
<smoser> you can fix it through late_command here.
<roaksoax> smoser: right, then I guess we should have an option within MAAS to disable that
<smoser> well, this [miss]feature is probably going to change.
<roaksoax> smoser: cuase, for example, in my particular case, how was I supposed to know I had a broken DNS on a clean machine and when I haven't played with DNS?
<roaksoax> i wouldn't have notice at all if it wasn't for what cloud init found
<roaksoax> so that means, like me, there might be others with the same issue
<roaksoax> having mirrors on a MAAS network, but not wanting to use them
<smoser> so.. i'm told that the way chrome handles this is that it tries dnslookups of never-will-exist domains.
<smoser> and then says "oh, you have broken dns"
<smoser> roaksoax, so..
<smoser> cobbler  cobbler/password  password xcobbler
<smoser> is there any reason that i'd pre-seed that in quantal?
<roaksoax> smoser: in pserv.yaml?
<roaksoax> smoser: yeah it gets replaced on postinst
<smoser> hm..
<smoser> in zimmer-build
<roaksoax> ah lol
<smoser> i used to do that
<smoser> to set a password for cobbler
<roaksoax> no not really needed
<roaksoax> maas should take care of settings its own user now
<smoser> so should i remove that even if on precise?
<roaksoax> I believe so
<smoser> what about updating /var/lib/cobbler/snippets/maas_proxy
<smoser> any reason to do that on quantal
<smoser> ?
<roaksoax> yeah it should probably stay
<smoser> ok... what about /etc/cobbler/cobbler.conf ?
<smoser> is that still around?
<roaksoax> smoser: you mean settings? yes still needed until we drop it
<smoser> i'm looking at zimmer-build/ud-build.txt
<smoser> and there are hacks/modifications for cobbler that are done
<smoser> and i was just trying to see if each is still necessary
<roaksoax> smoser: I don't really think we need any since all the setup is done by maas itself
<roaksoax> though we need to ensure that maas set's the correc tproxy, and next_server
<smoser> it updates /var/lib/cobbler/snippets/maas_proxy ?
<smoser> err... wait.
<smoser> is there a way that i can tell maas to use my *other* proxy?
<smoser> that is what the proxy hack was doing
<roaksoax> smoser: nope there's not. so that's why we should keep that.
<roaksoax> and making sure next_server and server and MAAS_DEFAULT_ULR are set correctly
<smoser> and then the cobbler init hack
<smoser> are you looking at this?
<smoser> i make cobbler update its network address on start
<roaksoax> smoser: so the cobbler network address update should be kept, + make sure it is updated on /etc/maas/maas_local_settings.py
<roaksoax> cause they should be the same
<smoser> so i should just change that hack to read it from /etc/maas/maas_local_settings.py ?
<smoser> from DEFAULT_MAAS_URL ?
<roaksoax> smoser: yes,
<roaksoax> that could also be a way to fix this
<smoser> PYTHONPATH=/etc/maas python -c 'import maas_local_settings, urlparse; print urlparse.urlsplit(maas_local_settings.DEFAULT_MAAS_URL)[1]'
<roaksoax> smoser: looks good to me
<smoser> roaksoax, http://paste.ubuntu.com/1098737/
<smoser> i'm gonna test that
<roaksoax> cool, looks good to me
<smoser> roaksoax, wait.
<smoser> does maas try to figure that value out on its own?
<smoser> or how can i tell it to re-get it.
<smoser> the problem is that i build the zimmer image, and then save it. and when it boots it has a new ip address.
<roaksoax> smoser: ah riught, it tries to figure out on postinst, so you need to obtain it the way you were doing it, and update maas_local_settings, and update cobbler
<smoser> bugger.
<smoser> roaksoax, which of the /etc/init/maas-* files do i need to change so they dont start until MAAS_URL iset correctly?
<roaksoax> smoser: actually that part is run by apache2
<roaksoax> through the wsgi.py file
<smoser> fun.
<smoser> roaksoax, so what do you think.
<smoser> i get the right address
<smoser> and then what?
<smoser> do i update /etc/maas/local_settings.py ?
<smoser> and dpkg-reconfigure maas ?
<smoser> will that do it?
<roaksoax> smoser: dpkg-reconfigure maas updates maas_local_settings.py and cobbler settings
<smoser> updates it from what?
<roaksoax> smoser: is a debconf questions that asks for the IP address
<smoser> ok. so i'm going to just go with this:
<smoser>  sudo dpkg-reconfigure -pmedium maas
<smoser> roaksoax,
<smoser> in apt-get install of maas (during zimmer build for quantal)
<smoser> /tmp/maas.config.12013: 32: /tmp/maas.config.12013: dbc_go: not found
<smoser> maas failed to preconfigure, with exit status 127
<smoser> Fetched 62.5 MB in 7s (8324 kB/s)
<smoser> Selecting previously unselected package libavahi-common-data:amd64.
<smoser> (Reading database ... 46830 files and directories currently installed.)
<roaksoax> smoser: that seems like dbconfig-common is missing or failing or something
<smoser> can you depend on that from config?
<roaksoax> smoser: yeah
#maas 2012-07-19
<bigjools> lifeless: did you see rvb's email about dns?
<lifeless> probably not, I've been fighting the 'flu
<lifeless> running at something like 50% efficiency
<smoser> roaksoax, are you around?
<bigjools> hey smoser
<smoser> hey.
<bigjools> smoser: did you see rvb's email about dns?
<bigjools> classless networks
<smoser> where?
<bigjools> on maas-devel
<lifeless> I see it now
<smoser> bigjools, this is all really only in regard to provisioning networks, right?
<bigjools> smoser: yes
<bigjools> but we're only running one dns server on the maas server side
<smoser> in which case i don't think the /8, /16, or /24 is an issue.
<bigjools> the problem is the reverse zone authority overlap
<bigjools> for masks not on /8 boundaries
<smoser> i'm missing something.
<bigjools> read http://www.indelible.org/ink/classless/
<smoser> i dont think it would help
<bigjools> if we don't do something then the reverse zone authority will overlap.
<bigjools> if that's not a problem then we can continue with no change
<smoser> for a real maas environment, i think that maas will control the "provisioning network", and the dns on it.
<bigjools> yes
<bigjools> but if there is an adjacent network with its own dns then there may be problems
<lifeless> I've replied
<lifeless> there is a very simple solution, and I'm sure there is a good reason we aren't doing it :)
<lifeless> so my reply was basically a question
<bigjools> we don't do that because the hostnames can be set separately in maas
<lifeless> why do we support that ?
<bigjools> I'd prefer to use nsupdate here
<bigjools> but this was all done while I was away
<lifeless> Somewhere along the way we got this hostname setting complexity.
<bigjools> because it was in the mockups? :)
<lifeless> But we're aiming at hyperscale deployments.
<bigjools> right
<lifeless> Manually setting hostnames there is, shall we say, pointlless.
<bigjools> well it's not manual as such but maas does generate a hostname
<lifeless> do we expect / permit nodes to change IP address on reboots ?
<bigjools> I went through all this on Monday with the guys... let me try and recall the conversation because twins make me forget stuff
<lifeless> Sure;)
<lifeless> So I'll put one more note out and then wait for your ack :>.
<lifeless> juju will break if the ip address of the bootstrap node changes.
<lifeless> So if we do permit unintentional node ip changes, we probably should stop it.
<bigjools> I was all for doing a static-ish map
<bigjools> but everyone complained that it makes it harder to recover when decommissioning nodes
<lifeless> recover what ?
<bigjools> reclaim the IP
<bigjools> I wanted to use DHCP's mechanism to update a ddns when a lease was allocatewd
<lifeless> so I'm saying - can we totally eliminate that overhead
<lifeless> don't reclaim the ip, keep it for maas.
<lifeless> make the dns name totally static
<bigjools> it is kept - but we were talking about re-using it for a different node
<bigjools> if a new node enlists
<lifeless> sok
<lifeless> ok
<lifeless> so, when someone says 'delete it' that would mean 'remove it from dhcp static allocation'
<lifeless> fin
<lifeless> no DNS change at all
<lifeless> (in my happy happy little world :P)
<bigjools> lifeless: it's an interesting point that we could just keep the dns static but the hardware might change underneath that ip/host
<lifeless> right
<lifeless> changing hardware is a decommission+commission
<bigjools> jtv: ^
<lifeless> in a cloud world
<lifeless> you don't replace the motherboard and claim its the same machine, you migrate the workload off and kill the box
<lifeless> or you can have a special 'I'm replacing the MB' feature if needed
<bigjools> lifeless: actually I am not sure any of this changes anything
<bigjools> we still need to work out new IPs being allocated and stick them in the DNS
<bigjools> we can do that systematically of course
<lifeless> yes, but you can have one zone for all your nodegroups
<lifeless> and just replicate the entire thing
<bigjools> ew
<lifeless> seriously
<lifeless> how much data is needed to configure 1M forward nodes and 1M reverse for the ec2 DNS approach
<bigjools> that would be crazy imo, reloading a zone file would take a long time with 1m nodes!
<lifeless> well
<lifeless> two things, a) it would be very rare - DC reconfiguration rare
<bigjools> I disagree
<bigjools> it'd happen every time you added new hardware
<lifeless> and b) I didn't mean exactly one zone, I meant that you can split on IP classes rather than on nodegroup boundaries.
<lifeless> so you don't need the rfc2317 dance around
<lifeless> so each zone would be at most 256 rows.
<lifeless> (on the reverse side)
<bigjools> we want one zone per rack/group because it keeps the model sane
<lifeless> on the forward side, I'd expect one zone per nodegroup (matching the compute-1 in the ec2 example.
<bigjools> right
<bigjools> lifeless: I think that rvb was wanting to keep reverse zones in step with forward zones, it keeps things nice and simple
<lifeless> yes, I can see that
<lifeless> I don't see that its worth the complexity if we can make it entire static
<lifeless> thats a huge win
<bigjools> "static" :)
<lifeless> yeah, static
<lifeless> how often will you bring up a new nodegroup ?
<bigjools> it'll change every time new hardware is added
<lifeless> why? Surely it only changes when you allocate *room* for new hardware
<bigjools> unless we get cute and write the zone files out when setting the IP ranges on the nodegroup
<lifeless> that was my point
<lifeless> that DNS is entirely decouplable from commissioning/decommissioning
<bigjools> jtv: are you reading this?
<jtv> Yes, but I'm not familiar with reverse DNS so some of it is magic to me.
<bigjools> jtv: it's basically a "trick" to make a forward lookup work with a special domain
<jtv> Ah.  I had indeed missed that completely.
<bigjools> 4.3.2.1.in-addr.arpa is the forward domain for the reverse lookup
<jtv> Forward domain for reverse lookup?
<lifeless> bigjools: ITYM 3.2.1.in-addr.arpa
 * jtv despairs
<lifeless> bigjools: 4.3.2.1.in-addr.arpa is the record :)
<bigjools> yes -the zone is only notionally a reverse.
<bigjools> lifeless: right :)
<lifeless> jtv: its a normal lookup, but the client knows to switch things around
<lifeless> for the purpose of this discussion anyhow.
<jtv> What things?  The octets?
<bigjools> jtv: it gets complicated when the netmask is classless (ie doesn't sit on /8 octet boudaries)
<lifeless> jtv: yes, octets
<lifeless> the query to make to find the official hostname for the ip address 1.2.3.4 is 4.3.2.1.in-addr.arpa
<bigjools> so the reverse zone is just a special forward zone
<bigjools> anyway
<bigjools> jtv: we're talking about writing out static dns zones based on the declaration of IP ranges in the nodegroups
<bigjools> it would *vastly* simplify things
<bigjools> we'd still use the lease parser to set the hostname in the MAAS
<bigjools> but hostnames would now be something like 192-168-1-1.my.domain
<bigjools> lifeless: anyway the original question is not answered - do we care about the reverse zone being overly authoritative when there's a non-octet netmask boundary?
<lifeless> yes, because otherwise it won't work for adjacent nodegroups
<lifeless> you need to do rfc2317 if you have that sort of split.
<bigjools> right, this is what I expected
<jtv> Soâ¦ does this mean we won't bother with the hostnames any more?
<bigjools> jtv: we'd use the lease parsing script to set them
<jtv> And they wouldn't be user-definable any more?
<bigjools> right
<bigjools> not sure it helps at hyperscale
<lifeless> hosts moving ip address will confuse and break various things
<lifeless> in particular juju, which we care a lot about
<jtv> At scale, I see no point whatsoever to those user-definable hostnames.  I think they're really only useful for the seed cloud.
<lifeless> All the seed cloud discussion i've seen have had juju in the mix
<lifeless> and there, the hostnames aren't needed with
<bigjools> it may not even be worth worrying about for seeds
<lifeless> as juju handles all the resolving from service unit -> what to ssh to
<bigjools> the lease parsing script can just do a dns lookup to get the host name
<bigjools> and poke it in
<jtv> By the way, this lease-parsing script you're talking about, where will that sit?
<bigjools> the one that you wrote you mean?
<jtv> Well there's no script as such.
<bigjools> it's a Task, yeah
<lifeless> fwiw it might be a good idea to translate dynamic leases to static ones, when we're running dns
<bigjools> so s/script/Task/ everywhere
<lifeless> bah
<lifeless> dhcp
<bigjools> lifeless: we discussed that
<lifeless> that would avoid any chance of accidental ip changes
<bigjools> I think it's fine as-is, because the ip can be re-requested when the lease expires
<bigjools> hmmm although if there's a lot of dhcp traffic I'm not sure if that holds
<lifeless> what if a machine is off for maintenance and a sysadmin plugs into the lan temporarilty
<lifeless> how do you make sure they aren't holding that machines ip when the machine comes up
<bigjools> why does it matter if it changes when the machine is down?
<bigjools> it just gets a new one when it comes up
<lifeless> right
<bigjools> cloud, remember? :)
<lifeless> and all the charm associations that have memoised the ip address break.
<bigjools> if it's down it's not allocated
<lifeless> we may have terminology issues
<lifeless> I mean 'commissioned but powered off'
<bigjools> then it's still not allocated
<lifeless> e.g. by someone running 'sudo poweroff' before they plug more disk in.
<lifeless> really?
<jtv> Actually an allocated machine may also be off for longer periods.
<jtv> That's up to whoever allocated it.
<lifeless> I thought we had a more ec2 like model, where power on/off is orthogonal to allocate/deallocate.
<bigjools> I am assuming that if it's allocated it's on
<jtv> I'm not sure that's a safe assumption.
<lifeless> I expect that to be the general case.
<lifeless> but not the contra positive.
<lifeless> er
<lifeless> I mean, but not the entire story.
<bigjools> is that the negative? :)
<lifeless> the contrapositive is entirely different :)
<lifeless> <- sick dammit!
<bigjools> hehe
<bigjools> jtv: right, someone could conceivably reboot their allocated hardware
<lifeless> they will
<lifeless> for kernel upgrades
<lifeless> folk do this in ec2 as well
<lifeless> having the leases be dynamic post enrollment really unnerves me.
<bigjools> so we really do need to write static lease maps, or make the lease 100000000 years
<jtv> And at smaller scales where people have access to the hardware they allocated, and the hardware has identity.
<lifeless> of course if you're using non-managed dhcp
<lifeless> its a different story
<lifeless> but production -> maas to the wall
<bigjools> actually a long lease is also problematic
<bigjools> because it's harder to revoke
<bigjools> jtv: I am desperate for food, but can we have a call in a short while?
<jtv> Sure
<bigjools> let's chat about this with a little more bandwidth :)
<bigjools> I'll grab you in about 15-30m
<jtv> ok
<smoser> anyone want to review https://code.launchpad.net/~smoser/maas/vdenv-updates/+merge/115645
<smoser> shouldn't be anythign controversial there.
<bigjools> smoser: I'll rubber stamp it
<bigjools> can you set a commit message
<bigjools> we need to rip the cobbler stuff out of it soon, will any of it work still or does it now need special magic?
<smoser> it doesnt do anything cobbler specific now.
<smoser> assuming the interfaces of 'maas-import-isos' and 'dpkg-reconfigure maas' stay the same. it should continue to work.
<bigjools> cool
<jtv> smoser: maas-import-isos is to be replaced by maas-import-pxe-files, which is cobblerless, but for now it still uses the same config file and variables (where applicable)
<smoser> it really should retain the same name if at all possible
<smoser> or warn "legacy name, calling maas-import-pxe-files, update your scripts"
<jtv> The latter should be easy enough.
<smoser> ok. i just pushed one more change to that branch
<smoser> http://bazaar.launchpad.net/~smoser/maas/vdenv-updates/revision/764
<bigjools> gah I wish people would spell "existent" properly
<bigjools> it's even in the cmd line opts
<smoser> bigjools, i suppose i was wrong above. the zimmer build (which i've been recently working on) is not dependent on cobbler.
<smoser> i'll have to move some of the cobbler api stuff to just use the maas api
<bigjools> ok
<bigjools> we have a card on the kanban board to fix it
<bigjools> but if you do it, even better :)
<smoser> there is a client library now?
<smoser> or cmdline client i thought?
<smoser> hm..
<smoser> have to bother roaksoax tomorrow.
<smoser> in my vdenv, i boot a new node, it seems to be going trhough enlistment, shuts itself off, but maas doesn't know about it.
<bigjools> sorry smoser, was OTP
<bigjools> there's no command line client yet
<bigjools> but it's on the cards
<smoser> roaksoax, awake?
<roaksoax> smoser here!
<roaksoax> lifeless: scrolling back and hafl understanding the conversation :) , one of the issues we had with juju from the beginning was the fact that it used hostnames to address to nodes , which in our terms means having a DNS server
<rvba> Hey smoser.  FYI we are currently working on getting MAAS to work from the tree (we're doing this in the QA lab).  I'm sorry if I was too blunt the other day (saying it was unsupported, etc.) but it will indeed requires us to fix quite a few things to get it working. 2 reasons for that: a) the cobbler-removal work has changed things a bit (everything is unit tested but we need to test that everything works well
<rvba> together) b) like I said, we've always tested MAAS "for real" (i.e. with real nodes) from the package so this will require adjusting paths and stuff like that.
<roaksoax> lifeless: we don't care so much about what IP address it had, or if it changed it, as long as we can address a node via its hostname
<smoser> rvba, glad that you're getting it to work from tree.
<smoser> my feelings weren't hurt too bad. i stopped crying in a couple hours ;)
<roaksoax> rvba: indeed, but I wonder why aren't we doing it on packaging too
<rvba> smoser: cool :)
<smoser> roaksoax, http://paste.ubuntu.com/1100038/
<smoser> any ideas there?
<smoser> i get that when a node tries to register.
<smoser> maas is quantal
<rvba> roaksoax: mostly because once we have that, it will allow us to iterate more quickly.  Basically we will find some of the bugs more quickly.
<roaksoax> smoser: ah yeah, known issue, waiting for a SRU: 1024010
<roaksoax> smoser: ah yeah, known issue, waiting for a SRU: bug @1024010
<roaksoax> err
<roaksoax> bug #1024010
<ubot5> Launchpad bug 1024010 in maas-enlist (Ubuntu Precise) "[SRU] After Commission Action 2 no longer exists" [Critical,Fix committed] https://launchpad.net/bugs/1024010
<roaksoax> rvba: ok
<rvba> roaksoax: known issue indeed :)
<roaksoax> rvba: but if the removal is in place, by running stuff from trunk, then i think we probably need to start taking care of releasing it that way in packaging too
<rvba> roaksoax: we're in the process of getting it to work from the tree in the QA lab.  But we're not there yet.
<roaksoax> smoser: it is in proposed btw
<smoser> roaksoax, so how do i fix this?
<roaksoax> smoser: either install maas-enlist from precise-proposed, or you can patch maas temporarily
<smoser> how would i install maas-enlist from precise-proposed ?
<roaksoax> smoser: give me a sec i'm writing a patch
<smoser> any idea on why we we wouldn't have maas server just allow '2' ?
<smoser> and change it to '0'
<roaksoax> smoser: because they removed the after_commissioning_actions it used to exist, that was 2, because it didn't do anything but we were defaulting to that
<roaksoax> smoser: and after a conversation with rvba we agreed it made no sense to re-enable it, if we just needed to fix it as an sru
<roaksoax> smoser: but now that you mention it, the install cd does not get the updated package right?
<smoser> i dont know. i think that it would have been worth just allowing 2 as a synonym for 0
<smoser> and warning in logs.
<roaksoax> rvba: ^^
<smoser> well, roaksoax over time, there will be a new -updates install cd made
<smoser> but that package is coming from the rchive anyway
<smoser> so you'd get the neewer one (i think its being apt-get installed, right?)
<roaksoax> smoser: yeah
<rvba> Given how the code is structured, that would have been awkward to keep the value '2' (which corresponds to something that is unsupported yet).
<roaksoax> smoser: http://paste.ubuntu.com/1100061/
<roaksoax> rvba: so I have a patch for powering off machines with celery, it acutally works but i'm having a bit of trouble figuring out the tests (modifying them to not fail): http://paste.ubuntu.com/1100058/
<roaksoax> rvba: so if you have some input, it would be great
<smoser> if roaksoax's patch to maas listed there fixes the problem, then rvba and i have a different definition of "awkward"
<rvba> smoser: it seems more logical to fix the place which uses a value ('2') which doesn't exist anymore in the vocabulary rather that to patch how the vocabulary itself to have it store '0' when it is told to store '2'.
<rvba> roaksoax: looking.
<smoser> i think it seems like an unnecessary grasp at perfection resulting breaking of existing things.
<smoser> 2 is not even an int there.
<smoser> roaksoax, do i have the abilityto hack at the netbooted image at all to add -proposed there?
<smoser> that would give us an actual test of -proposed
<roaksoax> smoser: I don't know to be hones, but was thinking that maybe we can tell it to use -proposed thourhg the preseed
<smoser> roaksoax, https://lists.ubuntu.com/archives/ubuntu-installer/2009-August/000466.html maybe
<smoser> apt-mirror-setup	apt-setup/proposed	boolean	false
<roaksoax> smoser: yeah looks about right
<smoser> rvba, i admit i'm probably a little grumpy.
<roaksoax> setting it to true should do
<smoser> but i do think i'd rather live with a small wart than break some existing user, even if its only for a short time.
<rvba> smoser: I understand the concern (and feel the grumpiness).
<rvba> smoser: But I'm not sure I understand the problem: why don't we change the client (which uses '2' instead of '0') first, and then the server?
<rvba> I'm sure this is a stupid question.
<smoser> right. if the client does not get the update, but the server does, then you are broken.
<rvba> Right.
<smoser> other than this window right now, i dont think of a place where this issue should occur.
<rvba> roaksoax: I see why the tests are failing.  It is because you now skip the nodes for which power_type is WOL.
<rvba> roaksoax: and the API returns an error if the number of returned nodes (when calling 'stop') is 0.
<roaksoax> rvba: yeah i changed to not skip, and they still skipped :)., But the tests need to be changed in order to test celery outputs too
<smoser> roaksoax, can you confirm... the default of the 'default' option gives me a menu
<smoser> and that menu (i think) defaults to "maas enlist" after 20 seconds
<smoser> but its strange that "local boot" is highlighted
<roaksoax> smoser: yep, that's the behavior
<roaksoax> rvba: for instance, test_stop_nodes_stops_nodes needs ot be change dto be celery compatible right?
<rvba> roaksoax: definitely.
<rvba> roaksoax: that's the only test which fails with this: http://paste.ubuntu.com/1100106/
<roaksoax> rvba: alright, cool
<rvba> roaksoax: let me give you an example code to check celery's output.
<rvba> roaksoax: actually, the celery fixture is activated by default so you simply need to check the content of self.celery.tasks.
<smoser> roaksoax, http://paste.ubuntu.com/1100114/
<smoser> i modified the kickstart so i get the above preseed change. and verify that i now get the maas-enlist deb from -prposed
<smoser> (which is good)
<smoser> but then see the end of tha tlog
<smoser> did something die?
<roaksoax> smoser: check if maas-pserv is running
<roaksoax> smoser: there seems to be a rare race condition there
<roaksoax> smoser: thath causes maas-pserv no to start
<smoser> maas-pserv says its running (status maas-pserv)
<roaksoax> smoser: then check cobbler is running, and has the same IP address as maas server
<smoser> sudo cobbler list works.
<roaksoax> smoser: interesting, because the error log shows an xmlrpc error
<roaksoax> or, cannot create connectio
<smoser> roaksoax, well, i did dpkg-reconfigure maas
<smoser> watched it restart everything
<smoser> restarted apache2 too just for grins
<smoser> still see the issue.
<roaksoax> smoser: that's weird indeed. I can't see any other reason why cobbler wouldn't be responding to the xmlrpc
<roaksoax>  rvba if I have "print" statements within the tests, how can I effectively see them without them failing because the print is there
<rvba> roaksoax: hum, I confess I never tried to put print statement inside tests (not sure how nose will react to that).  I always use the debugger: put a breakpoint and the run an individual test: ./bin/maas test src/maasserver/tests/test_node.py:NodeManagerTest.test_stop_nodes_stops_nodes -s
<roaksoax> rvba: what debugger do you use?
<rvba> roaksoax: pdb.  Put a line like 'import pdb;pdb.set_trace' in the code.
<smoser> roaksoax, it would seem to me that maas-pserv is busted.
<smoser> status is lying.
<smoser> the pid is not present.
<roaksoax> smoser: yeah so it didn't really start then
<roaksoax> smoser: there is a race there
<smoser> i dont think its a race
<smoser> its not going to start
<roaksoax> smoser: there are situations on which it starts and on which it doens't
<roaksoax> smoser: i have been trying to troubleshoot that yesterday with allenap
<smoser> i dont think this is that
<smoser> http://paste.ubuntu.com/1100158/
<roaksoax> smoser: that it is
<smoser> thats not a race
<roaksoax> smoser: run this manually : sudo twistd -n --uid=maas --gid=maas --pidfile=/run/maas-pserv.pid --logfile=/dev/null maas-pserv --config-file=/etc/maas/pserv.yaml
<smoser> thats what i did
<roaksoax> smoser: does it run?
<smoser> http://paste.ubuntu.com/1100160/
<smoser> (same thing i just pasted you)
<roaksoax> ah lol
<roaksoax> smoser: ok, go to /usr/share/pyshared/twisted/plugins/maasps.py
<roaksoax> smoser: and add a "raise" before the "paas"
<roaksoax> and try to run again manually
<roaksoax> it should run normally
<roaksoax> it is a race somewhere were it is failing to import something.
<roaksoax> that appeared after allenap added the ftp stuff
<roaksoax> rvba: so I've never used pdb before. SO you insert the break point, how do you run the code with pdb?
<smoser> roaksoax, well, now its running (as in it doesnt crash immediately) but it connections to localhost:5243 still get connection refused.
<roaksoax> smoser: kill it, remove the pid in /run/maas.pid
<smoser> it removes that
<roaksoax> smoser: and start maas-pserv again
<rvba> roaksoax: you just run the test(s) normally but with the option '-s', for instance with "./bin/maas test src/maasserver/tests/test_node.py:NodeManagerTest.test_stop_nodes_stops_nodes -s"
<rvba> Or "./bin/maas test src/maasserver/tests/test_node.py"
<roaksoax> right i add the -s but it still finishes without giving me a console to debug it
<rvba> roaksoax: are you sure you put your breakpoint in the code that gets executed?  "./bin/maas test src/maasserver/tests/test_node.py" executes only the tests in test_node.py.
<rvba> And that should be "./bin/maas test src/maasserver/tests/test_node.py -s"
<roaksoax> rvba: yeah I put the breakpoint in the code, not the tests
<rvba> roaksoax: that's all right, as long as the code gets executed, you should get pdb prompt.
<rvba> roaksoax: care to share the diff?
<roaksoax> rvba: ah never mind, i had () missing after set_trace :)
<rvba> roaksoax: I see :)
<rvba> roaksoax: My fault, 'import pdb;pdb.set_trace()'
<smoser> roaksoax, so... how can i get past this?
<roaksoax> smoser: now after manually running it, you should be able to run it with upstarts (maas-pserv that is) and eveyrthing should e running normally
<roaksoax> smoser: i've been trying to figure what is wrong without success
<smoser> roaksoax, no
<smoser> after i run it, no 5243 is open on localhost
<roaksoax> smoser: i'm waiting for allenap's to finihs writing a branch to see if that got this fixed
<roaksoax> smoser: try to do a reboot mabe?
<roaksoax> rvba: but anyways, other than that, I'm changing the test in such a way that it no longer uses WOl, but virsh, http://pastebin.ubuntu.com/1100178/ but it requires the binary
<roaksoax> rvba: but the other wol tests, don't require the ether_wake binary
<rvba> roaksoax: you can patch PowerAction.run_shell to make it do nothing.  This way, you still tests that the task has been fired but you don't have to deal with the real consequences.
<rvba> roaksoax: see how it's done in test_get_effective_power_parameters_provides_usable_defaults (src/maasserver/tests/test_node.py): self.patch(PowerAction, 'run_shell', lambda *args, **kwargs: ('', ''))
<roaksoax> yep :)
<roaksoax> thanks
<realnorth_> I was wondering if anyone could help me out with this problem: http://askubuntu.com/questions/165545/maas-install-64-bit-client-nodes-doesnt-work
<realnorth_> basically I can't get 64 bit precise installed on client nodes
<realnorth_> the network boot only boots 32 bit
<roaksoax> realnorth_: hi!
<realnorth_> hey
<roaksoax> realnorth_: so the machines 1. enlist. 2. commission. 3. deploy?
<roaksoax> realnorth_: when you have MAAS installed, and you start enlisting machines, it will actually autodetect the architecture
<roaksoax> but for enlistment it will run the Ubuntu installer on i386
<roaksoax> once enlisted, you "Accept & Commission"
<roaksoax> they will commissioning using an ephemeral image
<roaksoax> is that the process you are following?
<realnorth_> I have the parent node all set up
<realnorth_> and then I network boot the other machines
<realnorth_> pxe boot
<realnorth_> they are bare with no OS on them
<realnorth_> it boots over the network
<realnorth_> and gives me like 6 options
<realnorth_> with a 20 second timer
<realnorth_> If I let the timer go out it'll install i386
<realnorth_> if I choose any other option it doesn't register with the parent node
<realnorth_> there is maas-precise-i386, maas-precise-i386-commisioning, maas-precise-x86_64, maas-precise-x86_64, ubuntu-precise-i386, ubuntu-precise-x86_64
<realnorth_> and I think there is a maas-enlist option
<realnorth_> the one that is default is local
<realnorth_> I don't think I tried the maas-enlist option
<realnorth_> but I mean there is like no documentation on what any of those options are or where they come from
<realnorth_> https://wiki.ubuntu.com/ServerTeam/MAAS/AddNodes#Installing_via_PXE_then_accepting_into_MAAS_dashboard
<realnorth_> I followed those directions and it did the i386 version
<realnorth_> not the x86_64
<roaksoax> realnorth_: the default is not local, it is maas-nelist, but in the UI it shows 'local'
<roaksoax> but it actually selects maas-enlist
<roaksoax> in order to deploy
<roaksoax> but anyways, you need to let the timer finsih by itsel
<roaksoax> and let that node register itself in MAAS
<realnorth_> yeah but at that time it selects the i386
<realnorth_> not the x86_64
<roaksoax> realnorth_: it doesn't matter
<roaksoax> realnorth_: it is not deploying ubuntu
<roaksoax> it is just enlisting
<realnorth_> okay
<realnorth_> when is anything installed on it
<roaksoax> realnorth_: once the machien enlists into 'MAAS', then you go to the MAAS WebUI, and clic on 'Accept & Commission'
<realnorth_> uhu
<roaksoax> that will (or should) start the machine again tfor the commissioning process
<roaksoax> realnorth_: once that process is done, then the machine will be in 'Ready' starte
<roaksoax> state* and you will be able to deploy Ubuntu
<roaksoax> when you deploy, it will install 64bit version
<realnorth_> ahhh
<roaksoax> realnorth_: are you looking into using juju
<roaksoax> ?
<realnorth_> yeah
<realnorth_> that's what I was planning on doing
<realnorth_> so I just boot them normally let it time out
<realnorth_> and then juju will handle installing the real OS right?
<realnorth_> after it has been commissioned and everything
<roaksoax> realnorth_: yeah, so 1. boot, let them time out for enlistment. 2. 'Accept&Commission' on WebUI, make sure it boots up again and let it commission (you'll need to configure the power management features correctly). 3. When machines are in 'Ready' state, you can use juju to deploy them
<realnorth_> okay
<realnorth_> what power management settings are you talking about?
<roaksoax> realnorth_: from the MAAS webui, you'll need to configure PowerManagement
<realnorth_> oh
<roaksoax> realnorth_: though in precise there are only few supported
<realnorth_> alright I know what you are saying now
<realnorth_> one last question
<realnorth_> to do the WOL do you need to open a port for every MAC on the MAAS parent?
<roaksoax> realnorth_: nope. The network cards should already be configured torespond to WoL requests
<roaksoax> realnorth_: MAAS simply sends a WoL packet to the node
<realnorth_> okay
<realnorth_> thanks you have been a real help
<realnorth_> I wish the Ubuntu docs had a bit better explanation of things
<roaksoax> realnorth_: yeah we are also looking to have better docs sometime in the near future :)
<realnorth_> thanks
<smoser> roaksoax, so i reboot
<smoser> no dice
<smoser> do you want to come in and poke around?
<roaksoax> smoser: sure
<roaksoax> rvba: finally it got it. Thanks for the help! Was cracking my head last night trying to get this test to work
<rvba> roaksoax: do you remember why we generate a default hostname if none is provided? (I'm asking you because it turns out you introduced that method: node.set_mac_based_hostname).  Now that we're going to change the hostname to be IP based I'm wondering if we can ditch that.  The only difference is that before the node will be assigned an IP, the hostname will be ''.  Would that be a problem?
<roaksoax> rvba: wait, are you pre-assigning an IP address to the enlisted nodes?
<rvba> roaksoax: no, the plan is to parse the DHCP lease file to get the IP address.  Then a node will always get the same IP from the DHCP server.
<roaksoax> rvba: right, is there an email thread where this is being discussed to address this
<rvba> roaksoax: the hostname will be ip-based, Ã  la amazon 192.168.3.2-maas.domain.
<rvba> roaksoax: yes, I'm implementing that precisely :)
<roaksoax> rvba: is this being discussed in an email thread?
<roaksoax> rvba: cause there might not be a need to parse the leases file
<rvba> roaksoax: my question is about that method: node.set_mac_based_hostname.  I'd like to remember why this was introduced, to assess if I can ditch it now that we're changing the hostnames to be IP-based.
<rvba> roaksoax: that's actually another question.  I'm working on changing the usage of node.hostname at the moment.
<rvba> roaksoax: see the discussion on the maas-dev list.
<rvba> roaksoax: more specifically, the threads with 'DNS' in the title.
<roaksoax> rvba: yeahc being the one having worked on Orchestra/Juju stuff at first, I think DNS was an issue
<roaksoax> but anyways
<roaksoax> to answer your question
<roaksoax> rvba: that method was added because during the enlistment process, we do not sent a hostname that we'd like the machine to be registered with
<roaksoax> rvba: however, we needed a hostname to be automatically determined
<roaksoax> rvba: at the moment, we decided it was simple enough to use a MAC address and generate the hostname based on that
<rvba> roaksoax: do you remember why exactly?  Is this only to have something to show in the UI?
<rvba> Because we didn't want to show the system-id?
<roaksoax> rvba: no because we needed the hostname to make it addresseable to juju
<roaksoax> rvba: juju *needs* hostnames
<rvba> roaksoax: indeed, so what I'm doing is going to replace that then.
<roaksoax> rvba: we cannot have a machine without a hostname, can we?
<rvba> Well, it will be without a hostname right after it's been added.
<roaksoax> rvba: well i think a hostname should be assigned on enlistment
<rvba> But as soon as the node will get an IP address, it's hostname will be set.
<rvba> its*
<rvba> So juju will be happy.
<roaksoax> rvba: right
<roaksoax> rvba: but you are giving the node a IP address on enlistment, commissioning, and deployment
<roaksoax> rvba: why complicate the process when nit should be kept simple
<roaksoax> rvba: IMHO, on enlistment, you should set the hostname. Simple as that
<roaksoax> rvba: whether it is based on IP, MAC or randomness, it should be set
<rvba> The plan is to have very long lease times so the same IP will be assigned to a node everytime it boots.
<rvba> time*
<roaksoax> rvba: right, so why determine a hostname only in the last boot, when it can be determined on enlistment
<roaksoax> cause as you mention, the idea is to keep
<roaksoax> a long time lease
<rvba> roaksoax: the DHCP server is in charge of assigning the IPs.  So we won't know why IP has been chosen until the node is booted up.
<rvba> s/why/which/
<roaksoax> rvba: exactly, and that is either on 1. enlistment, 2. commissioning. 3. deploying
<rvba> roaksoax: not sure I follow, 'enlisment' is simply about registering the node inside MAAS.  'commissioning' will be the when the node boots up for the first time so that's where the IP will be determined, once and for all.
<roaksoax> rvba: well, you already give an IP on enlistment, why would I need to do that on commissioning if you can get it early enough from enlistment
<roaksoax> rvba: i can't really recall whether ytou needed a hostname when adding a machine to maas as a requirement, whether it was enlisted or not
<roaksoax> I think if you didn't send one it would fail to enlist/add
<roaksoax> rvba: eitherway, the enlistment process should be able to tell MAAS the IP address that the client got
<roaksoax> enlistment already knows its IP, and should probably feed that back to MAAS to generate a hostname
<rvba> No the hostname has always been optional, precisely because we could generate a MAC-based hostname.
<roaksoax> IMHO
<rvba> roaksoax: I'm sorry but I don't see when the IP is given to MAAS during enlistment.
<rvba> s/when/where/
<roaksoax> rvba: we don't, we could
<roaksoax> rvba: either way there was thoughts of making enlistment/commissioning 1 step
<roaksoax> rvba: give me a sec and I'll give a better explanation of this
<rvba> roaksoax: we've got 2 types of enlistment, the one you're talking about where we have the IP and the manual enlistment where a user adds a node in the UI.  I'm not sure we want to ask the user the IP in this case, the mac address should be enough for MAAS to get going.  And in this case we will need to get the IP from the lease when the machine will be booted up.
<roaksoax> rvba: when you add a node from the UI, you can automatically select an IP for that
<rvba> roaksoax: how?
<roaksoax> rvba: give me a sec please
<rvba> sure
<roaksoax> rvba: btw.. from now on, are we going to enforce our own DHCP/DNS in mass without the possibility of using an external one?
<rvba> roaksoax: now, that's still going to be an option.
<rvba> s/now/no/
<roaksoax> rvba: ok so lets addres External DNS/DHCP server first
<roaksoax> rvba: So, during nelistment, the machine boots up and recieves a IP and a hostname from the external DHCP server. it uses both of those to enlist itself in MAAS
<roaksoax> then the hostname is set in MAAS for the enlistment
<roaksoax> which is required as it is the one given to juju
<roaksoax> and later on, juju uses the hostname in MAAS nodes to conectat the deployed nodes
<rvba> roaksoax: why do we need the IP in this case?
<roaksoax> rvba: we don't. juju doesn't really care what IP the server has
<roaksoax> rvba: it cares about the hostname
<rvba> roaksoax: so MAAS only needs to get the hostname then, not the IP.
<roaksoax> rvba: yes, that's in the case of running external DNS/DHCP
<rvba> Right.
<roaksoax> in the case of running one within, we need to make sure the node gets a hostname. We don't care what IP address is given to the node, because juju doesn';t care
<rvba> But since we need to write the DNS zone file, we have to get the IP.
<roaksoax> however, the issue being experienced by you guys is "How do I do DDNS"?
<rvba> files* even, forward and reverse.
<roaksoax> right, that's DDNS
<roaksoax> so your recommendation, instead of doing DDNS, is "let's parse the lease file"
<rvba> No.
<rvba> Robert's recommendation is: write a zone for each nodegroup with the full range of possible IPs, then use hostname derived from the IP addresses.  Just like amazon does.
<roaksoax> ok
<roaksoax> that;s fine
<roaksoax> rvba: but, does that mean admins won't be manually able to set their own hostname?
<rvba> roaksoax: indeed, they won't be able to do that.
<roaksoax> rvba: that's something I personally don't agree with
<roaksoax> rvba: if you have an external DNS/DHCP, you are allowing them to do that
<roaksoax> rvba: if you have an internal, you are giving them the option
<roaksoax> rvba: from my point of view, administrators will want to know what machine is what machine, and the only way to do that is by hostname
<roaksoax> that's me though
<roaksoax> my personal perception
<rvba> roaksoax: the main goal for maas is to be a provider for juju so I really think that's fine also.  Also, once you are playing with more than a handful of nodes, you won't want to customize the nodes' hostnames.
<roaksoax> maybe you do, maybe you dont
<roaksoax> but that's why there are 'name' constraints in juju
<roaksoax> yto allow us to select a particular node
<roaksoax> rvba: and that was very helpful in the ODS
<roaksoax> rvba: but anyways, these thouhgs of mine should go on the email response
<rvba> roaksoax: I'm not really sure we have a choice here, unless we want to manually rewrite the zone files each and every time a hostname changes.
<roaksoax> rvba: we don't really, once a node is enlisted we have a hostname and a zone written for it
<roaksoax> rvba: we should not be able to simply edit the hostname
<roaksoax> rvba: but anyways, the thing is that if in enlistment with external DNS/DHCP I already know the hostname
<roaksoax> rvba: I could determine the hostname in the enlistment process and send it back to MAAS
<roaksoax> rvba: that's what I wanted to do at first, but was decided to do it on the maas side
<roaksoax> so during enlist, we already know the IP address, and can simply say "This is my IP" and maas server can say "Enlistment, thanks, your IP will always the one you just got"
<rvba> And MAAS will have to write the zone files.
<roaksoax> rvba: or you can parse the leases file by MAC and add the IP, as in: DHCP gave 1.1.1.1 to MAC aa:bb:cc:dd:ee:ff , so on enlistment on the maas side "I got your enlistment request Aaa:bb:cc:dd:ee:ff", "the IP dhcp gave you is 1.1.1.1"
<roaksoax> rvba: if we are trying to avoid writing zones, then have no possibility of naming our own nodes
<roaksoax> which means we lose functionality
<roaksoax> IMHO
<rvba> I'm not trying to avoid writing zones files.  I've just added the ability to write zone files in MAAS yesterday :).
<rvba> But we're trying to keep it simple :).
<roaksoax> I personally think allowing us to set our own hostnames, is a good thing. In the hyperscale where we need things on depand and we are gonna run thoundsands of servers, that will be used on deman based on, lets say, usage needs, then we don't really care about hostnames
<roaksoax> rvba: but I personally think others will want to know the hostnames and what they have there and deploy accordingly
<realnorth_> not to interrupt your conversation but I have a really quick question
<realnorth_> can  you have maas boot a specific iso not just ubuntu precise?
<realnorth_> cobbler can but I was hoping for something simple through maas
<rvba> roaksoax: I understand your concerns but I think you should send an email to the list.  I'd like Robert to be part of the discussion.
<roaksoax> rvba: so, let me go through the emails, and give my thouhgs.
<roaksoax> rvba: heh yeah :)
<roaksoax> rvba: but to keep things simple, I think the hostname generation should be kept as simple as possible, and if you wanna change from MAC to IP "If MAAS is DHCP/DNS", then you would need to check that in the lease file for now
<roaksoax> rvba: because DHCP will already know what IP it gave to what MAC, so on enlistment, you already have the MAC then you just obtain the IP from that lease file
<roaksoax> and voala
<roaksoax> voila
<roaksoax> :)
<rvba> Well, that's the plan.  We will fetch the IP as soon as possible but if someone registers a new MAC which does not correspond to an assigned IP (because the machine hasn't been booted yet), then we will need for that machine to boot to get its IP.
<roaksoax> rvba: right, but when it comes to maas, we only need the first mac for the internal/deployment network
<roaksoax> rvba: which would be the "maas-management" network
<rvba> That's right but my point above is still valid.  That MAC might not be in the lease table.
<roaksoax> rvba: if I enlist into maas with X ethernet card, then the mac will be in the lease table
<roaksoax> rvba: that same X card will be the same used in commissioning and deployment, wouldn't it?
<rvba> roaksoax: yes, but maybe at enlistment time, that card is completely unknown, that's my point.
<roaksoax> rvba: right, it is uknown, but, does DHCP go: "I received a request from aa:bb:cc. I selecte ip 1.1.1.1 and granted it to aa:bb:cc"
<roaksoax> doesn't it do that?
<rvba> roaksoax: it does that when the machine boots up.  If the machine has never booted up yet, then the dhcp lease file does not contain that information.
<roaksoax> rvba: exatly, so on enlistment, the machine boots up, requests IP, DHCP server knows about it, then whne it sends the enlistment requirest, it grants it a hostname
<rvba> roaksoax: we're going in circles here.  I completely agree.  But when a user manually enlists a node by providing only its MAC, then, if the node was never booted up, we can't get its IP address.  We will need to wait for the node to be booted up (during commissioning) to get its IP.
<roaksoax> rvba: when you manually enlist, you can simply select one IP address from the pool and grant it automatically
<roaksoax> rvba: you manually nelist, you already know the MAC address, then you simply autoselect an IP address
<roaksoax> you already know the range, you already know which ones are in use, and which ones are free
<rvba> But you don't know which one the DHCP server will pick.
<roaksoax> rvba: we do not care which one it picks
<roaksoax> we just need one
<roaksoax> rvba: so its like "Hi I'm aa:bb:cc and I need an IP addres", "Hi aa:bb:cc, i have IP 1.1.1.1 free and i'm gonna give it to you" "thanks DHCP< now my hostname is blablabl.1.1.1.1-bablabal"
<rvba> roaksoax: Ok, so you want to do the DHCP request manually in MAAS. Now I get it :)
<roaksoax> rvba: yeah just check what IP addresses are free and select one from the pool and tell the DHCP "the ip for MAC aa:bb:cc will be 1.1.1.1"
<rvba> roaksoax: ok
<rvba> roaksoax: I need to run, thanks a lot for your insights; please reply to the thread 'Strategy regarding DNS and static DHCP leases' on the maas-devel MP.  I'd like this to be discussed by everyone.
<roaksoax> rvba: will do, but will do it internally only as I might mention private stuff
<rvba> roaksoax: ok
<smoser> http://paste.ubuntu.com/1100429/
<smoser> roaksoax, does that make any sense?
<roaksoax> smoser: i don;t recall seeing something like that
<roaksoax> smoser: but rvba did make some changes in the wsgi file that changed the behaviour of the avahi stuff
<roaksoax> which in fact might be the cause of this
<smoser> i'm running from ubuntu package
<roaksoax> smoser: /usr/share/maas/wsgi.py is the cause of that error
<smoser> roaksoax, ok. so i now have my ssystem that does not show the issue
<roaksoax> heh...
<smoser> the thing i did was disable the rc.local script
<smoser> maas-set-ip: http://paste.ubuntu.com/1100460/
<smoser> rc.local script: http://paste.ubuntu.com/1100461/
<smoser> so there is a race there.
<smoser> roaksoax, can you look at maas-set-ip and tell what i've done wrong there?
<smoser> or what i should not do ?
<roaksoax> smoser: i get this error when I run it : awk: line 1: regular expression compile failed (missing operand)
<roaksoax> *
<roaksoax> smoser: but other than that, nothing seems wrong
<smoser> really?
<smoser> roaksoax, can you explain that? it deost make sense. to me.
<smoser> i dont knwo what regular expression could be bad
<smoser> can you run it with 'sh -x' ?
<roaksoax> smoser: http://paste.ubuntu.com/1100500/
<smoser> so if you just run
<smoser>  awk '{gsub("*","");} $1 == key { print $2 }' key=maas/default-maas /etc/passwd
<smoser> does that give that error?
<smoser> roaksoax, ^
<roaksoax> yes
<roaksoax> smoser: oh your problem might be that then, it cannot connect to xmlrpc because you are using the worng password for it
<smoser> roaksoax, are you running bsd?
<smoser> or some form of aix?
<smoser> hm..
<roaksoax> lol it is a quantal image
<smoser> no its not quantal
<roaksoax> ubuntu@server-13819:~$ awk '{gsub("*","");} $1 == key { print $2 }' key=maas/default-maas /etc/passwd
<smoser> i call foul there.
<smoser> its 12.04
<roaksoax> awk: line 1: regular expression compile failed (missing operand)
<roaksoax> *
<roaksoax> smoser: http://pastebin.ubuntu.com/1100516/
<roaksoax> smoser: in my machine the error is not sohwn though
<smoser> dpkg -S `which awk`
<smoser> er.. i guess
<smoser> update-alternatives --display awk
<smoser> you have mawk.
<smoser> but ok. i'll fix anywy. its just a matter of putting '[*]' i think
<smoser> roaksoax, so i definitely think its a race condition based on running that maas-set-ip in rc.local
<roaksoax> may be indeed
<adam_g> https://bugs.launchpad.net/juju/+bug/1021861
<ubot5> Ubuntu bug 1021861 in MAAS "Transient error /w MAAS provider: Unknown operation: 'list_allocated'." [Undecided,New]
<adam_g> am i the only one hitting this? seems 25% of juju's commands are failing because of it
<realnorth_> anyone know of a way to boot an iso other than ubuntu?
<realnorth_> like for booting up a node
<roaksoax> realnorth_: While it is not supported, nor recommended, you could hack your way around it
<realnorth_> how?
<roaksoax> realnorth_: you'd have to import an iso and replace the profiles within cobbler
<realnorth_> for ubuntu?
<realnorth_> I would have to replace the ubuntu profile with the one I want booted?
<roaksoax> realnorth_: you would have to point the profiles to a different distro
<realnorth_> okay
<realnorth_> understood
<realnorth_> thanks
<lifeless> roaksoax: juju uses the address the provider supplies
<lifeless> roaksoax: maas provider can supply ip addresses
<lifeless> roaksoax: I was thinking about this overnight and there is no need for dns integration at all for the juju hyperscale story AFAICT
<roaksoax> lifeless: TBH, I do not know how things have changed from back when I did the initial orchestra/juju work, but at the time, juju (ensemble at the time) did not support the use of IP address and it was made very clear to me that it will never do that
<lifeless> roaksoax: the openstack provider shoves the ip address into the host fields.
<lifeless> roaksoax: and I have a patch I run here that uses ip addresses for ec2 to make devstack installs work properly.
<lifeless> roaksoax: so, there may be political issues. But concretely speaking, it works, and well, to just use IP.
<lifeless> roaksoax: in case you're thinking 'but using DNS protects from ip changes in ec2' or whatever - it doesn't: ec2 dns names are 1:1 with IP, if IP changes, DNS name changes too.
<lifeless> the only reliable mapping is instance id -> public,private hostname.
<roaksoax> lifeless: I see, well I don't have anything against juju addressing to the machines via ip rather than hostna,me, as I was pushing for that back then.
<lifeless> yah
<lifeless> so as an experiment you could try changing the maas provider locally to use the ip address
<roaksoax> lifeless: the only thing though, is the maas juju provider name constraints (as detailed on the email)
<roaksoax> use hostnames
<lifeless> I think we need to reevaluate some of the stack here
<lifeless> e.g.
<lifeless> is that sensible for a 10K install
<lifeless> if not, ok, so how do we do it for a 5 node install
<roaksoax> but I think that's merely knowing the name whithin maas and then communication can be done through IP
<lifeless> for a 5 node, I can imagine labelling the node in the MAAS UI, and letting the API say 'this named node', and yeah - after that, IP all the way.
<roaksoax> lifeless: yeah in the email I detailed my position about defining a ip based hostname works for hyperscale but not for small deployments where admins might want to set names to identify their machines
<roaksoax> lifeless: an yeah I agree, as the email mentioned, at ODS we used the named constraints to install certain components on certain machines and I think that's a feature that hsould be kept
<roaksoax> but the apprach can be changed to use IP address as communication and the hostname as simple labelling
<lifeless> roaksoax: you realise you went off-list ?
<roaksoax> lifeless: yes
<lifeless> robbiew: Hey, I'm around anytime
<robbiew> lifeless: cool...in an hour work for you?
<lifeless> sure
<realnorth_> is there a link for all the environment variables I can use?
<realnorth_> there doesn't seem to be a whole lot of information on what I can use or change
<realnorth_> or add
<lifeless> hi
<lifeless> its all meant to be via the UI
<lifeless> as MAAS runs as a daemon
<lifeless> is there something in particular you are looking for?
<lifeless> robbiew: just putting cynthia to bed; may be a little late as a result
<lifeless> robbiew: ok, so am ready when you are
<robbiew> lifeless: coolio
<robbiew> g+?
<lifeless> sure
<lifeless> invite winging its way to you now
#maas 2012-07-20
<realnorth_> anyone here have a link or know what environment variables I can feed juju?
<realnorth_> also is it possible to choose which server to add a charm to?
<melmoth> realnorth, i have been told that jitsu has a -deploy-to option
<melmoth> but i never try it yet
<smoser> i'm guessing just about everyone is sleeping, lifeless and bigjools at least. allenap maybe around?
<smoser> i hadnt' followed all the dns thread
<allenap> smoser: Hi there, what's up?
<smoser> but i thought that lifeless suggested doing ec2 like names, which i think is generally useful.
<smoser> but then others didn't like that because of "no human friendly hostname"
<smoser> wouldn't cnames fix that?
<smoser> and having dhcp hand out the hostname as the human-friendly-cname?
<allenap> smoser: Yes. One of the questions I had was: what do names with the IP address in add beyond just using IP addresses?
<allenap> rvba: ^ maybe you can help too?
<smoser> not much i guess.
<smoser> but if you add in the cname, then you have both, dont you?
<allenap> Yeah.
<allenap> For smaller clouds, having a nice name is more useful. For massive clouds it's a burden.
<smoser> random bit of info . i think that there was some dns service that just converted "v.x.y.z.somedomain.com' to v.x.y.z Ip address for you
<smoser> i tihnk ii think you'd just allow it as configurable.
<smoser> an dif someone sets the "hostname " for something then they get it.
<smoser> who cares.
<rvba> smoser: so a node would have an ec2 like name and a "human" CNAME then?
<smoser> yeah.
<smoser> well, that was my thought.
<smoser> reverse looking up its IP would get the ec2-like name
<smoser> but who cares.
<rvba> right
<rvba> The main problem with writing DNS config files is with reverse config files (because of the classful/classless pb) so yeah, that sounds like something we should definitely look into.
<smoser> if you allow the api to set the hostname of a node, then someone can easily take a DB that has "MAC:preferred-hostname" and populate maas from that.
<rvba> allenap: actually, we sort of mentioned that on the call this morning: having "two" hostnames: the ec2 like one and the manual one.
<rvba> But we thought about that on the DB level.  As smoser said, CNAME is exactly that at the DNS level.
<allenap> Yeah. We also mooted allowing nodes to specify their preferred hostname in the DHCP request and making that a CNAME of the "ugly" name.
<allenap> But that doesn't cope with collisions well.
<smoser> well, its easily fixable anyway.
<smoser> the only place that that has value is when maas is not running the dhcp/dns server
<smoser> and it assumes that something gave this system a relevant IP address based on its mac address.
<smoser> er.. relavant hostname based on mac
<smoser> and whatever that something is can then be just dumped/queried. and a tool then update the node once its in maas.
<rvba> allenap: collisions are not really a problem if we can write the reverse zone files using only ec2 hostnames.
<smoser> collisions are an issue if maas isn't serving the dhcp
<rvba> allenap: for each zone, we compute the superset classful network and we write the full reverse zone file for that with ec2-like names.
<allenap> rvba: I mean collisions of the "nice" names if done by dhcp; two nodes could ask for the same thing.
<rvba> Right, that's another problem.  I was thinking about the clasless network collision.
<rvba> Maybe the term "collision" is not really proper for that.
<allenap> I think my brain has had enough of that stuff for this week ;)
<rvba> smoser: that's a great idea I think.  Maybe there is something I'm missing but it definitely looks like the missing piece of the puzzle.
<roaksoax> allenap: howdy :) so I was looking at your recen changes and was wondeirng, wher's that maas-provision binary?
<rvba> smoser: I tried to summarize why I think your idea is great in the email I just sent.  We would have the full forward/reverse DNS config written using ec2-like names.  Then on top of that we could use nsupdate to add CNAME record to accommodate custom hostnames.
<rvba> records*
<smoser> rvba, thanks.
<smoser> we expose "friendly-hostname" setting in maas via api
<smoser> right?
<rvba> Yes we do.
<smoser> so my 'mac-addr:friendly-hostname' population script is easy.
<smoser> (ie, it takes a file full of lines with 'mac-addr friendly-hostname' and just iterates over it calling maas and setting the hostname for each node based on mac
<smoser> )
<allenap> roaksoax: It's a front-end for several commands. It will probably evolve into the front-end for all of MAAS (so that we no longer expose twistd or django-admin).
<rvba> smoser: yes, that would work.
<roaksoax> allenap: right, but where is it? do we need a wrapper for it (like we did with maas binary)? (see debian/extras/maas in packaging branch)
<rvba> smoser: hum, the key we use in the api is the system_id.
<roaksoax> rvba: that is indeed a good idea as I was mentioning to smoser
<roaksoax> but that means that servers wont have <rack>.<city>.<blahblah>.domain.com
<roaksoax> which might come handy
<roaksoax> when you do inventory
<roaksoax> that's why I think we should consult this with sysadmins
<rvba> Well, we might provide a way to set the custom hostname based on tags.
<rvba> So that would be possible.
<smoser> rvba, somehow i can surely get a node by its mac-id, no?
<rvba> Let me check.
<smoser> err.. get a system_id by its mac_id.
<roaksoax> rvba: right, there's various possiblities to approach the issues yes
<roaksoax> we need to consider invetory from now on
<roaksoax> and we need to consider Administrators, and Datacenters and the standards they use
<roaksoax> and provide a sokution based on that
<roaksoax> we can't simply impose them something we believe is better when they already follow standards
<rvba> smoser: no, system_id is the key really.  But that's something we could add very easily.
<smoser> rvba, well i think its essential, really.
<rvba> smoser: I completely agree.
<rvba> smoser: please file a bug.
<rvba> smoser: that's something really easy to add.
<rvba> The mac is the information that the outside world knows so it makes sense.
<roaksoax> allenap: btw is that part of the maas cli?
<allenap> roaksoax: It is, and it will need a wrapper. It's very simple though: python -m provisioningserver
<roaksoax> allenap: right, and what is the maas-cli stuff
<roaksoax> allenap: err apiclient
<allenap> roaksoax: Don't know :-/
<roaksoax> ok :)
<roaksoax> rvba: ping
<roaksoax> rvba: do you how how could you effectively run yui tests?
<rvba> roaksoax: the yui tests are run as part of the test suite but you can run only the yui tests.
<roaksoax> rvba: right, I'm looking towards running them in the actual yui3 source package from the archives
<roaksoax> rvba: so I was wondering how is it that you run them?
<roaksoax> and get the ooutput of them as?
<rvba> In MAAS, we use SST to do that (see src/maasserver/tests/test_js.py).
<smoser> rvba, bug 1027154
<ubot5> Launchpad bug 1027154 in maas (Ubuntu) "need way to get system based on mac address" [Undecided,New] https://launchpad.net/bugs/1027154
<rvba> smoser: ta
<roaksoax> rvba: cool thanks!
<smoser> anyone able to +1 this:
<smoser> https://code.launchpad.net/~smoser/maas/vdenv-fix-mawk/+merge/116067
<smoser> roaksoax, ^
<roaksoax> smoser: give me a sec :)
<roaksoax> smoser: done!
#maas 2013-07-15
<kentb> is there a 'proper' place to put dns forwarding info on a maas controller?  I have a dual-homed machine running cluster and region controller.  maas-dns & maas-dhcp are being managed on eth1 (private network) and eth0 is how I get out to the internet.  I found that if I edit /etc/bind/maas/named.conf.maas and add a forwarders option, my maas nodes can access
<kentb> the outside world.   However, if I make any changes to that interface on the maas web UI, the forwarding info gets wiped out. Thanks!
<kentb> "that interface" being eth1 for my dns dhcp management
#maas 2013-07-16
<bigjools> kentb: you need to edit named.conf.template instead
<AskUbuntu> How to setup a MAAS region controller with two interfaces? | http://askubuntu.com/q/320801
#maas 2013-07-17
<rubberneck> Are there editable preseed files created for each node somewhere?
#maas 2013-07-18
<bigjools> rubberneck: no, the preseed template is shared across all nodes
<rubberneck> bigjools: i guess a better question would have been is there a way to send kernel options to just certain nodes? I have some dell 1950s and they need a rootdelay option or they won't boot correctly.
<bigjools> rubberneck: you can use tags to achieve this
<bigjools> maas-cli maas tags new name='gpu' kernel_opts='rootdelay'
<bigjools> then add the tag to the relevant nodes
<bigjools> sorry, maas-cli maas tags new name='rootdelay' kernel_opts='rootdelay'
<bigjools> mis-pasted
<bigjools> http://maas.ubuntu.com/docs/tags.html
<rubberneck> bigjools: thank you
<bigjools> the doc doesn't mention the kernel_opts parameters, but the source code does :)
<AskUbuntu> how to deploy charm to a machine which is already installed with some other charm? | http://askubuntu.com/q/321479
#maas 2013-07-20
<AskUbuntu> dhcp server configuration in maas | http://askubuntu.com/q/322227
<AskUbuntu> Ceph Openstack HA installation with maas and juju but partially on physical servers and few on virtual machines | http://askubuntu.com/q/322235
#maas 2014-07-14
<lifeless> bigjools: so is there a blog post up about the pitfalls of celery (and why you're moving away from it?)
<bigjools> heh
<bigjools> hi stalker!
<lifeless> you know it
<bigjools> there's nothing wrong with it per se, it just doesn't do what we want
<lifeless> ok
<lifeless> what do you want? That it doth not do?
<bigjools> bi-directional comms and notifications that it's down
<bigjools> ie more than just a job processing system
<lifeless> interesting
<bigjools> because we're not directing generic jobs anywhere, we direct them to specific queues
<bigjools> we've NIHed bidirectional comms
<lifeless> RPC calls, or actual cooperating messages to long lived routines?
<bigjools> RPC
<bigjools> some will be long lived
<bigjools> but we need feedback on important stuff, like "did my power on command work?" :)
<blake_r> allenap: can I get a quick review for this, 1.6 upgrade fix, https://code.launchpad.net/~blake-rouse/maas/fix-1341001/+merge/226673
<allenap> blake_r: Sure.
<lazyPower> Is there a configuration guide that i've missed to proxy ppa requests since maas installs an apt-proxy on all of its hosts? i'm getting frustrated with having to remote in an dcomment out the apt-proxy on all of my maas nodes when testing juju charms that make use of a ppa
<bigjools> lazyPower: edit the squid config that maas installs to be more permissive
#maas 2014-07-15
<rvba> bigjools: did you QA the fix for https://bugs.launchpad.net/maas/+bug/1341001? (I can help with that if you didn't)
<ubot5> Ubuntu bug 1341001 in MAAS 1.6 "maas.tgt not rewritten on 1.5 -> 1.6 cluster upgrade" [Critical,Triaged]
<bigjools> rvba: I did not
<bigjools> bit sidetracked by other stuff today
<rvba> Okay, I'll do it now.
<bigjools> easy review karma: https://code.launchpad.net/~julian-edwards/maas/cluster-task--report-last-import-time/+merge/226782
<rvba> bigjools: approved
<bigjools> rvba: thanks
<bigjools> "make lint" always makes me chuckle
<rvba> allenap: hi, I see you've changed the RPC stuff so that exceptions are properly propagated.
<rvba> jtv: I'm still in the process of making sure it's repeatable but when I tried upgrading my 1.5.2 installation to trunk I got: http://paste.ubuntu.com/7797346/.  Any idea where this is coming from?
<jtv> rvba: yes, that's a known bug â fix is in review.
<rvba> Ah okay.  Ta.
<jtv> bigjools: I replied to your review.
<rvba> allenap: the error I'm getting on the cluster when a NoSuchNode is raised on the Region looks like this: 'Node with system_id=Node with system_id=unknown-system-id-dPaTUv could not be found. could not be found.'
<rvba> allenap: Looks like the code from paste.ubuntu.com/7797390/ is "applied" twice or something.
<allenap> rvba: Ah, thatâll be because youâve customised __init__ to munge the message.
<rvba> allenap: I see, the error is "recreated" on the other sideâ¦
<rvba> Right?
<allenap> rvba: Yep.
<allenap> rvba: You could instead customise __str__ (and __unicode__).
<rvba> allenap: that's precisely what I'm trying now :).
<rvba> jtv: lp:~jtv/maas/bug-1340896 is the branch that fixes the upgrade problem right?
<jtv> Yes.
<jtv> See the bug.
<jtv> You might actually have insights useful for the review discussion.
<rvba> Okay, I'll look at it in a bit.  I'm in the middle of doing some QA now and I'd like to get that sorted firstâ¦
<rvba> jtv: got the same upgrade problem with the fixes from that branch :/
<jtv> Hrrrh!?
<rvba> It's possible that my installation is brokenâ¦ I've been fighting with it all morning.
<jtv> Then maybe we need 3 migrations after all...
<jtv> But I did make it go through the migrations forwards and backwards with cluster interfaces present.
<jtv> (And I observed the changes in an SQL shell)
 * rvba switches to maas/1.6
<rvba> jtv: happy to do some more testing later (and have a look at that branch/bug) but right now, I just want to get through with the QA.
 * rvba backports fix.
<jtv> rvba: right 1.6 doesn't have those schema changes, so the problem won't happen there.
<rvba> Yep, that's why I'm using it.
<jtv> OK
<jtv> Would love to hear more if you run into that migration problem again â I did a similar experiment here and didn't have the problem.
<bigjools> rvba: did you look at why CI is failing?
<rvba> bigjools: no; still busy QAing the fix for https://bugs.launchpad.net/maas/+bug/1341001
<ubot5> Ubuntu bug 1341001 in MAAS 1.6 "maas.tgt not rewritten on 1.5 -> 1.6 cluster upgrade" [Critical,Triaged]
<rvba> bigjools: error installing: http://paste.ubuntu.com/7797564/
<bigjools> rvba: looks like a dependent package is broken
<bigjools> very hard to see which one though
<rvba> bigjools: job #215 failed with the error I pasted above.  job #214 failed because the cluster didn't connect to the region (I'm assuming this is somehow related to the problems Jeroen is working on);  but the dependent packages were exactly the same in the two runs.
<bigjools> rvba: could have been an upload in the interim.  Is this utopic or trusty?  they both fail
<rvba> bigjools: utopic.  Like I said, I retrieve the list of the packages that have been installed and there is no difference.
<bigjools> rvba: same versions?
<rvba> Yes
<bigjools> ok
<bigjools> weird then
<bigjools> very weird
<rvba> I'm diffing the lines:
<rvba> Get:1 http://archive.ubuntu.com/ubuntu/ utopic/main libc-bin amd64 2.19-4ubuntu1 [1169 kB]
<rvba> â¦
<bigjools> k
<rvba> I'm running a test on 1.6 (topic-adt-maas-manual).
<rvba> bigjools: the failure of the trusty job is different.  That job uses Trusty.  Seems like revision 2290 of lp:maas/1.5 broke the build (unless it's spurious).
<rvba> bigjools: 2290 is the backport of the nonce stuff.
<bigjools> rvba: fuck
<rvba> bigjools: I started another job, just to make sure the failure is consistent.
<bigjools> ok thanks
<bigjools> back it out if it fails again
<rvba> k
<rvba> bigjools: allenap: fwiw there is a card on the board (in the robustness lane) for the "Move power-related tasks over to twisted" task.
<rvba> And no, I'm not insisting ;).
<bigjools> it'll need breaking down, but that's ok
<rvba> bigjools: QA ok for the fix for https://bugs.launchpad.net/maas/+bug/1341001
<ubot5> Ubuntu bug 1341001 in MAAS 1.6 "maas.tgt not rewritten on 1.5 -> 1.6 cluster upgrade" [Critical,Triaged]
<rvba> bigjools: hum, the precise job also failed and the MAAS version didn't change.  Something might be wrong with the lab itself.
<bigjools> rvba: yeah, that would be my next suspect
<rvba> bigjools: I'm backporting the fix for https://bugs.launchpad.net/maas/+bug/1341001 to 1.6
<ubot5> Ubuntu bug 1341001 in MAAS 1.6 "maas.tgt not rewritten on 1.5 -> 1.6 cluster upgrade" [Critical,Triaged]
<bigjools> rvba: thank you
<bigjools> you're too kind :)
<rvba> allenap: I still get the repeated exception message when I customize __str__/__unicode__ on my exception class.
<allenap> rvba: Can you show me the code?
<rvba> allenap: just one secâ¦
<rvba> allenap: http://paste.ubuntu.com/7797848/
<rvba> allenap: http://paste.ubuntu.com/7797873/ wqorks
<rvba> works*
<allenap> rvba: Intriguing. I like the alternative constructor pattern. (Though donât inherit from BaseException.)
<rvba> allenap: right (I got rid of BaseException)
<rvba> allenap: where in the code does the exception recreation happen?
<allenap> rvba: See eb_massage_error in MAAS, or _massageError in amp.py.
<rvba> allenap: I'm confused, why is this in src/provisioningserver/rpc/testing/__init__.py ?  i.e. in what seem to be a testing utility?
<rvba> seems*
<allenap> rvba: Because itâs a testing utility that avoids mimics a remote call without having to do messy stuff with sockets and suchlike.
<allenap> s/avoids /
<rvba> allenap: but outside of testing, the error is also recreated on the caller's side right?
<allenap> rvba: Yes. eb_massage_error is meant to make call_responder behave more like a real RPC call.
<rvba> allenap: all right.  It feels a bit weird that we have to manually recreate that behavior in testing but okay..
<rvba> bigjools: the 1.5 (trusty) job passed.  It was indeed a problem with a node, the nonce stuff is okay.
<bigjools> cool
<bigjools> thanks for checking it out
<rvba> I'm running all the other jobs now.
<rvba> A run with 1.6 is in progress now.
<rvba> If this one passes, it will be time to release another beta package.
<bigjools> let me know
<rvba> bigjools: test passed (revision 2534 on lp:maas/1.6).
<bigjools> thanks
<jtv> Trusty CI is better now.
<jtv> Oh, that's 1.5.  :(
<jtv> The failure has changed.  Could it be that the CI needs to refresh its knowledge of the API?
<jtv> But then why did previous updates to the NGI succeed?
<jtv> Ahh, no, it didn't.
<rvba> jtv: looks like it's the parsing of MAAS' output that failed.
<rvba> jtv: no, I'm wrong, it's related to the recent addition of the NGI's name.
<jtv> Yes.  Question is, how?
<jtv> The API doesn't expect values for all of the form fields, does it?
<rvba> Depends on the form that handles the data.
<jtv> The field is taken directly from the model, and it's blank=True null=False.
<jtv> The form's cleaning provides a value if it's blank.
<jtv> Or at least, that's the theory.
<jtv> The parsing fails because _set_up_dhcp needs to check manually for an error return from _run_maas_cli, and doesn't.
<jtv> It receives error output and assumes it gets usable json.
<rvba> Right.
<jtv> The only hint we get from the CLI is "name" â which we don't specify, so that's why I wonder if there's a problem with that.
<jtv> I'm going to see if I can reproduce the problem with a simple API/CLI update.
<rvba> Should do the trick.
<jtv> Seems to work on the API...  :/
<jtv> class Action(Command):
<jtv>     uri = property(lambda self: self.handler["uri"])
<jtv>     def __call__(self, options):
<jtv>         uri = self.uri.format(**vars(options))
<jtv> ^ This is why we don't want code to be clever or "use the language to its full."
<jtv> That last assignment is the source of the error.
<jtv> Whoopee, right?
<rvba> What's 'options'?
<jtv> Parameter to __call__.
<rvba> No shit :)
<jtv> If you want to know more, and so do I, let's ask allenap...
<rvba> heh
 * allenap reads backwards
<jtv> allenap: the main thing is that bit of code (class Action(Command):) from the CLI.
<allenap> jtv: Whatâs going wrong?
<jtv> KeyError.
<jtv> In that self.uri.format line.
<allenap> jtv: Is it saying which key is missing?
<jtv> Yes.  It's ânameâ.
<allenap> So thatâs something in the URI template that Django/Piston generates.
<jtv> Where do they get the information for generating that URI template?
<allenap> It comes from the Piston handler class.
<allenap> resource_uri_template
<jtv> Then I think understanding dawns.
<allenap> See describe_handler in maasserver.apidoc.
<jtv> The CLI call isn't passing the name field, which is supposed to go in the URL path.
<allenap> jtv: That implies that the CLI doesnât know the name field is required. I think that parameters can be declared, but they need to be part of the URL path that a Django/Piston handler is registered with, or something like that.
<allenap> Which it should be :-/
<jtv> The CLI doesn't know that the name field is _not_ required.  :(
<jtv> So the URI template says "give me <name> here" and the CLI code tries to take <name> from its parameters dict, which doesn't have it.
<allenap> jtv: What does the URL look like?
<jtv> http://localhost:5240/api/1.0/nodegroups/{uuid}/interfaces/{name}/
<allenap> jtv: Why is the name optional?
<allenap> What behaviour does that produce?
<allenap> Without the {name} filled in, I think that should resolve to the NodeGroupInterfacesHandler. With {name} it should resolve to NodeGroupInterfaceHandler, i.e. the singular.
<jtv> The URL used to have the network interface name in it.  That's been changed to be the cluster interface name.
<jtv> So the request doesn't need to pass ânameâ as a field, only as part of the URL.
<allenap> jtv: Can you paste the error for me? I donât have the CI email any more :-/
<jtv> No point.  You get less useful information than this.
<jtv> The problem is that the CLI takes the positional URL parameters by name instead of positionally.
<jtv> No idea how to fix that.  :(
<allenap> jtv: Maybe the CI script needs to do `maas refresh` to get updated API descriptions?
<rvba> allenap: the CI use a new VM instance (to install MAAS) for each run.
<jtv> allenap: no, a refresh doesn't do it.  The problem is that the CLI takes the positional parameters from the dict of keyword arguments â which I suppose also spells trouble for any case where an identifying field is editable.
<allenap> jtv: Do you have some instructions so that I can reproduce this?
<jtv> allenap: ./bin/maas <profile> node-group-interface update <uuid> <name>
<jtv> Produces: ./bin/maas: error: u'name'
<jtv> Debugging shows that it's a KeyError while attempting to format that URI.
<jtv> To add to the fun, passing ânameâ as a named parameter doesn't help.
<allenap> jtv: It also believes that the name is required for bin/maas <profile> node-group-interface read
<jtv> Duh.
<jtv> The problem is that the CLI code passes that parameter around by name, instead of positionally.
<allenap> jtv: NodeGroupInterfaceHandler.resource_uri is still returning âinterfaceâ as a parameter. Not sure if that would explain everything, but itâs worth giving a go.
 * allenap gives it a go.
<jtv> Passing name and interface both doesn't change it for me.  :(
<allenap> https://www.irccloud.com/pastebin/vlInRrAc
<allenap> jtv: ^ try applying that, restarting, refreshing the cli.
<matsubara> allenap, jtv: fwiw, I filed https://bugs.launchpad.net/maas/+bug/1342117 for the cryptic error: u'name' failure
<ubot5> Ubuntu bug 1342117 in MAAS "CLI command to set up node-group-interface fails with /usr/lib/python2.7/dist-packages/maascli/__main__.py: error: u'name'" [Critical,Triaged]
<jtv> Thanks matsubara. We're just looking into it.
<matsubara> jtv, yep, I saw the backlog. Thanks. (And thanks for fixing the other nodegroup crash!)
<jtv> allenap: your paste (with a refresh) fixes the problem!
<jtv> I thought that part where these methods substitute a fixed string was only there for documentation..?
<allenap> jtv: Thereâs still an oddity where `â¦ node-group-interfaces list` requires a `uuid` argument. I think NodeGroupInterfacesHandler.resource_uri needs adjusting.
<allenap> jtv: Nope, waste not want not: it was repurposed for driving the CLI too :)
<allenap> jtv: Actually, and I donât know why, NodeGroupInterfaces.list() takes a uuid argument, which I think it probably shouldnât.
<jtv> allenap: IIRC NodeGroupInterfaces.list lists the cluster interfaces for one cluster â and so you need to tell it which cluster.
<jtv> Exceedingly small branch for review: https://code.launchpad.net/~jtv/maas/ngi-name-ci-breakage/+merge/226859
<dergrunepunkt> hi, I'm using maas + juju + virsh, installed as the doc said and it's working
<dergrunepunkt> I have the nodes installed and when they boot they show as "enlisted-node-XXX"
<dergrunepunkt> but I cant get juju to do the bootstrap
<dergrunepunkt> because every time I run "juju boostrap" it starts a VM and the operating system installation runs again
<dergrunepunkt> what I am doing wrong?
<schegi_> hi out there, have a maas lxc related question can someone help??
<rbasak> schegi_: nobody will know until you actually ask your question
<schegi_> thought it was more juju related, bootstraped a maas environemt and was wondering how to define the bridge device which is used if an service is deployed in a container.
<schegi_> asked in the juju channel
<dergrunepunkt_> why maas keeps reinstalling a node?
#maas 2014-07-16
<jtv> Thanks rvba.
<rvba> np
<bigjools> allenap: I was planning on doing a crappy TimerService derived service in pserv and making it do exactly as discussed in that email, do we need to pre-imp this further?
<allenap> bigjools: Try to avoid the crappy part, but otherwise that sounds fine.
<bigjools> haha
<bigjools> allenap: it's gonna call the func I landed yesterday every hour and then kick off a deferToThread() for the import
<bigjools> allenap: I still think we need a queueing mechanism for jobs
<allenap> bigjools: This is something Iâve discussed a lot with rvba. There should be only one job running per node at any time. For example, it doesnât make sense to have both an install and a shutdown job running concurrently...
<allenap> However, there are subtleties.
<bigjools> allenap: what about related jobs like this one?
<bigjools> unrelated, sorry
<allenap> If an install job is running and a âcheck the power stateâ question comes along, we can probably answer that question without waiting for the install to finish.
<bigjools> and I agree - but enforcement should be done in a different level
<allenap> bigjools: There should only be one import job at a time.
<bigjools> allenap: not talking about that per se
<bigjools> I mean total jobs across the pserv
<bigjools> if it's managing thousands of nodes, we could potentially have thousands of threads
<allenap> If we start hitting concurrency problems then we can add limits, but letâs get it working first.
<bigjools> all strangling CPU
<bigjools> I think we should consider this from the star
<bigjools> t
<allenap> Donât use threads too freely then :) Use the Twisted, Luke. In truth, many of those threads will be blocking at any one time.
<bigjools> well, thanks to the GIL :)
<allenap> Yes, but mostly because of IO.
<bigjools> simplestreams is not twisted
<allenap> I know, but itâs largely IO bound.
<bigjools> yes - it's a candidate
<bigjools> but it's no I/O bound when it builds the tarballs
<allenap> When you use deferToThread, the thread is obtained from a thread pool, to which we can apply limits.
<allenap> We can also create separate thread pools and defer tasks into them specifically.
<bigjools> that would be great
<allenap> The default thread pool is limited to 10 threads.
<bigjools> celery limited to total cores
<bigjools> we have to be absolutely sure that long running things like this don't interfere with power_on and suchlike
<bigjools> and for that we need a QA plan
<allenap> bigjools: If simplestreams is using tarfile, the Python module, then high CPU there could pose a problem to pserv. If itâs using /bin/tar then Iâm not going to worry.
<bigjools> I can't remember, let me look
<allenap> bigjools: twisted.internet.reactor.getThreadPool().adjustPoolSize(min, max)
<bigjools> it shells out
<bigjools> it loop mounts the extracted image
<bigjools> but will obviously hammer the disks
<allenap> bigjools: Thatâs not new though, and we could ionice it if itâs swamping things.
<bigjools> allenap: so that's adjusting the default pool size?
<bigjools> aye
<allenap> Yep.
<bigjools> allenap: too easy
<bigjools> what happens if you exceed that?  does it queue, or reject new thread requests?
<allenap> bigjools: It queues.
<bigjools> I wonder if we shouldn't set a limit on that too
<allenap> bigjools: If we subclass t.python.threadpool.Threadpool and override callInThreadWithContext we can add that behaviour.
<allenap> s/Context/Callback/
<allenap> biab
<bigjools> allenap: excelente!
<allenap> bigjools: Actually, more Twisted would be to use a DeferredSemaphore, say, to manage concurrency.
<rvba> allenap: I've got a branch up for review that I'd like you to review if you have time (it's tiny): https://code.launchpad.net/~rvb/maas/ipmi-power-state/+merge/226974
<rvba> I contains shell scriptâ¦ and my thinking is "shell script" â allenap ;)
<rvba> s/I contains/It contains/
<bigjools> allenap: can you have limits on those?
<allenap> rvba: Sure.
<rvba> Ta
<allenap> bigjools: Yep, you set the limit on the semaphore size when creating it. Weâd want to check the sem.waiting queue before calling sem.acquire() or sem.run(â¦).
<bigjools> allenap: excelente!
<jtv> Who wants to review my branch for the QA lab tests?  https://code.launchpad.net/~jtv/maas/qa-alt-ngi-name-fix/+merge/226991
<rvba> jtv: I'll take it
<jtv> Thanks.
<allenap> blake_r: Have you noticed any memory problems using tgt for iSCSI? Iâm using a laptop with 4GB memory to run a region+cluster for my NUCs but the boot import crashes with out-of-memory errors when registering images with tgt at the end.
<blake_r> allenap: no I have not in don't have anything with that low of memory
<allenap> blake_r: Hehe, yes, itâs somewhat underspecified. Itâs pink too.
<blake_r> allenap: unless the targets are mounted I wouldn't think it should take that much
<blake_r> allenap: the laptop is pink?
<allenap> blake_r: Yes, the laptop is pink. Itâs a cast-off. I have set a lot of swap but it doesnât care. Okay, thanks anyway, I may sniff around to see what I can discover.
<blake_r> allenap: Haha. Okay.
#maas 2014-07-17
<rvba> bigjools: jtv: I've got a couple of branches up for review if you're available: https://code.launchpad.net/~rvb/maas/event-types/+merge/227155 https://code.launchpad.net/~rvb/maas/event-model/+merge/227156
<dvr077> q
<dvr077> quit
<rvba> jtv: Based on the conversation I had with Julian this morning, I'm reworking the event-model branch.
<blake_r> rvba: quick question, why when a node is "acquired" it is marked allocated? shouldn't it be marked "Reserved". Reserved doesn't seem to be used at all.
<blake_r> rvba: shouldn't it be marked allocated, once the node is started
<rvba> blake_r: yeah, we've got a bunch of statuses that are not used at all.
<rvba> blake_r: we currently don't do that distinction (node acquired but not yes started / node acquired and started).
<rvba> s/not yes/not yet/
<blake_r> rvba: okay
<rvba> blake_r: We will probably clean that up as part of the robustness work / node lifecyle work.
<blake_r> cool
<blake_r> jtv: could you give this another look over, https://code.launchpad.net/~blake-rouse/maas/cloudbase-generate-winrm-cert/+merge/222650
#maas 2014-07-18
<rvba> allenap: I'm creating a small utility to send an event and create the event type if it doesn't exist.  How will that function (which will be called from the power-{on, off} tasks) get hold of the rpc_service object?
<allenap> rvba: Thatâs a good question :)
<allenap> rvba: I think we need to create a singleton to refer to all the services (or the application). I have some work towards that lying around...
<rvba> allenap: yeah, I was thinking about a singleton too.
<rvba> I'll make the utility take a rpc_service argument for now.
<rvba> We can figure out how to pass it around later.
<allenap> rvba: Yeah, that sounds good.
<rvba> Cool.  Ta.
<rvba> blake_r: reviewing your add-boot-type-to-node branch now.
<blake_r> rvba: oh, cool. thanks
<rvba> allenap: how can I easily unit test a method that uses other methods from the RPC region service?  Since I can't call call_responder directlyâ¦ is there a better option or will I have to patch all the RPC methods?
<allenap> rvba: See ClusterRPCFixture (er, which may not yet have landed). That wires up RPC calls to the cluster without needing sockets and whatnot.
<rvba> allenap: right, not landed yetâ¦
<allenap> rvba: Got time for a very quick review before you finish for the weekend? https://code.launchpad.net/~allenap/maas/rpc-cluster-client-service-ipv4-only/+merge/227358
<allenap> Just think, what a great way to finish a Friday.
<blake_r> allenap: i can do it
<rvba> allenap: nice!
<allenap> blake_r: What a wise man you are :) Thanks.
<rvba> allenap: thanks for the review!  Now, I need a review for https://code.launchpad.net/~rvb/maas/event-rpc-methods/+merge/227215 and https://code.launchpad.net/~rvb/maas/event-model/+merge/227156 and I'll be able to land this thing!
<allenap> rvba: Looking at that now.
<rvba> allenap: Thanks!
<allenap> blake_r: Do you mind taking another look at https://code.launchpad.net/~allenap/maas/rpc-cluster-client-service-ipv4-only/+merge/227358?
<blake_r> allenap: looking at it now
<allenap> Ta.
<blake_r> done
<allenap> blake_r: Thanks. Have a tip top weekend.
<blake_r> allenap: you to!
#maas 2015-07-13
<mup> Bug #1437280 changed: maas 1.8.0alpha7 two machines have same ip address <openstack> <uosci> <juju-core:Invalid> <MAAS:Expired> <https://launchpad.net/bugs/1437280>
<kiko> hey there
<kiko> sto: could you vote me up? :)
<kiko> http://askubuntu.com/questions/646278/how-to-ask-curtin-to-use-gpt-instead-of-mbr-with-maas/646839#646839
<rbasak> kiko: here, have an upvote :)
<kiko> thanks rba
<kiko> thanks rbasak
<mup> Bug #1474083 opened: Node disappeared after upgrade from 1.5 to 1.7 <MAAS:New> <https://launchpad.net/bugs/1474083>
<mup> Bug #1474083 changed: Node disappeared after upgrade from 1.5 to 1.7 <MAAS:Invalid> <https://launchpad.net/bugs/1474083>
<mup> Bug #1474083 opened: Node disappeared after upgrade from 1.5 to 1.7 <MAAS:Invalid> <https://launchpad.net/bugs/1474083>
<mup> Bug #1474083 changed: Node disappeared after upgrade from 1.5 to 1.7 <MAAS:Invalid> <https://launchpad.net/bugs/1474083>
<evilrob> so I've got a datacenter with several systems booting off PXE boot through dnsmasq.  I'm wanting to deploy openstack using canonnical's autopilot which uses maas.  How can I segregate maas and dnsmasq for netbooting in the same environment?
<wolverineav> so I've seen this behaviour periodically - if I leave the maas environment running over the weekend and then try to add a new node or release an existing, I run into this error of 'too many files open' : http://paste.ubuntu.com/11874322/
<wolverineav> there is an existing bug which is currently in fix-commited state, but I thought I should mention this so it helps in the fix.
<kiko> evilrob, I think you need to set up a separate VLAN; you can't have more than one DHCP listener on a segment
<kiko> wolverineav, this is a problem that has affected OIL at canonical; jhobbs is the one who is tracking it
<evilrob> kiko: that's what I'm thinking I'll have to do.
<evilrob> all of our production boxen have 2 interfaces.  I may just make dnsmasq only listen on the first
<CTWill> So I build a MAAS server and it does not seem to want to boot the pxe image on any of my clients
<CTWill> after much google'ing I am here and stuck
<CTWill> does anyone have some MAAS foo they can share?
<wolverineav> kiko: thanks for the info!
#maas 2015-07-14
<varunarya> Hi All, I just did a fresh install of ubuntu 15.04 server and maas using the documentation provided http://maas.ubuntu.com/docs1.7/install.html
<varunarya> When I create the admin user and go to the http://localhost/MAAS and click on login and enter the admin credentials I created then it does not log in and shows the login screen again
<varunarya> sorry if it is not the right forum. I am stuck in this login screen even after creating multiple admin users and using the corresponding credentials.
<rishi_> hiii
<rishi_> ping aybody
<rishi_> ctrl + c
<rishi_> ping anybody
<j^2> kiko: when you got some time can you ping me?
<kiko> j^2, okay, in 45 mins?
<j^2> yeah thatâll work, Iâve been evanglizing your software and found some feedback
<mup> Bug #1474417 opened: squid-deb-proxy does not refresh translation files often enough <maas (Ubuntu):Confirmed> <squid-deb-proxy (Ubuntu):Confirmed for mvo> <https://launchpad.net/bugs/1474417>
<kiko> maybe it shouldn't refresh translations ever, roaksoax? ^^
<j^2> kiko: we good to sync up?
#maas 2015-07-15
<mup> Bug #1474792 opened: maas installation bughit <MAAS:New> <https://launchpad.net/bugs/1474792>
<jogarret6204> hello.  having issue building a kvm instance using only maas.  Instance does not get DHCP using linuxbridge. tcpdump shows DHCP reply from maas being dropped.  All security is disabled.  help!
<jogarret6204> trusty 1404
<jogarret6204> maas 1.7.5
<mup> Bug #1474967 opened: MAAS 1.8.0+bzr4001-0ubuntu2 won't import boot resource images <MAAS:New> <https://launchpad.net/bugs/1474967>
<mup> Bug #1474967 changed: MAAS 1.8.0+bzr4001-0ubuntu2 won't import boot resource images <MAAS:Invalid> <https://launchpad.net/bugs/1474967>
<mup> Bug #1474967 opened: MAAS 1.8.0+bzr4001-0ubuntu2 won't import boot resource images <MAAS:Invalid> <https://launchpad.net/bugs/1474967>
<mup> Bug #1474967 changed: MAAS 1.8.0+bzr4001-0ubuntu2 won't import boot resource images <MAAS:Invalid> <https://launchpad.net/bugs/1474967>
#maas 2015-07-16
<Samir> HI pp
<Samir> hi
<mup> Bug #1475372 opened: MAAS misrepresents core count by including hyperthreading <cloud-installer> <landscape> <MAAS:New> <https://launchpad.net/bugs/1475372>
<mup> Bug #1435810 changed: Retried views are passed empty files <isolation-level> <MAAS:Fix Released by rvb> <https://launchpad.net/bugs/1435810>
<xoritor> can maas be installed on one or more of the openstack physical nodes?
<rbasak> xoritor: I find an odd question. Normally if using MAAS, you'd use MAAS to deploy openstack physical nodes.
<xoritor> yea, i am just limited in physical hardware
<xoritor> waht i have is fairly robust
<xoritor> especially for my need use
<xoritor> i have 5 servers  (12 cores, 128 GB DDR4, 4x1 Gbit, 2x40 Gbit, 1x256 GB SSD, 1x1 TB SSD, 1x2 TB PCIe)
<xoritor> ok can maas run in a VM then?
<xoritor> for just the 5 machines
<xoritor> i can give it 4 dedicated (passthrough) gigabit nics and 200 GB of ssd on an 8 core server, but I can not give it the whole server
<xoritor> and it can have 16 GB of ram
<xoritor> would that be enough "power" in a VM for dealing with 5 hosts?
<xoritor> can maas be run in docker?
<mup> Bug #1475410 opened: Third party drivers aren't loaded during commissioning or disk erasing <oil> <MAAS:New> <https://launchpad.net/bugs/1475410>
<mup> Bug #1475445 opened: Cluster disconnecting periodically - [ClusterClient,client] Failed to refresh power state <oil> <MAAS:New> <https://launchpad.net/bugs/1475445>
#maas 2015-07-17
<bhundven> having a problem adding my initial node to maas. I have the api url set to 172.16.0.52, and the internal interface of the cluster is 192.168.254.x/24. During the commission of the initial node, I see it trying to connect to http://172.16.0.55/... and failing. I believe this is due to the nat setup.
<roaksoax> bhundven: the nodes under MAAS should be able to access the region controller
<roaksoax> bhundven: so, dpkg-reconfigure maas-cluster-controller , and make it point to the address in the 192.168.254.x/24 range
<bhundven> right, but then I won't be able to access maas from the 172. side of the network.
<roaksoax> bhundven: you can
<roaksoax> bhundven: the region (webapp) listens in *all* networks
<bhundven> ah, ok!
<bhundven> roaksoax: thanks!
<bhundven> I shoulda checked netstat :P
<bhundven> roaksoax: ok. Sorry for my ignorance, but I'm learning fast here. I'm deploying the first node still, I've tried different ubuntu versions, but see the same thing in the cloud-init: /var/lib/cloud/instance/scripts/user_data.sh: 18: /var/lib/cloud/instance/scripts/user_data.sh: initctl: not found
<bhundven> then it says its powering off
#maas 2015-07-19
<mup> Bug #1453872 changed: Prodstack: bootstrap instance started but did not change to Deployed state  <MAAS:Expired> <https://launchpad.net/bugs/1453872>
<mup> Bug #1475977 opened: filenames for user-defined preseeds when booting using the debian installer are not correct. <MAAS:New> <https://launchpad.net/bugs/1475977>
#maas 2016-07-18
<mup> Bug #1603947 opened: ntp dhclient exit hook does not remove most servers <amd64> <apport-bug> <trusty> <MAAS:Triaged> <ntp (Ubuntu):New> <https://launchpad.net/bugs/1603947>
<mup> Bug #1603947 changed: ntp dhclient exit hook does not remove most servers <amd64> <apport-bug> <trusty> <MAAS:Triaged> <ntp (Ubuntu):New> <https://launchpad.net/bugs/1603947>
<mup> Bug #1603947 opened: ntp dhclient exit hook does not remove most servers <amd64> <apport-bug> <trusty> <ntp (Ubuntu):New> <https://launchpad.net/bugs/1603947>
<mup> Bug #1603947 changed: ntp dhclient exit hook does not remove most servers <amd64> <apport-bug> <trusty> <ntp (Ubuntu):New> <https://launchpad.net/bugs/1603947>
<BlackDex> Hello there, can a normal user add a machine via MAC? If not, how can we let a normal user do that.
<BlackDex> To prevent changing default settings etc..
<MonsieurBon> Hi guys. Is there a way to install maas < 2.0 on xenial?
<BlackDex> MonsieurBon: Nope, use 14.04
<MonsieurBon> seriously? there is only a beta version available on an LTS server OS? that's a bit disappointing
<BlackDex> Ubuntu LTS Server isn't released yet
<BlackDex> atleast not fully
<BlackDex> that will happen with 16.04.1
<MonsieurBon> I would call this released: http://www.ubuntu.com/server
<MonsieurBon> :)
<BlackDex> https://wiki.ubuntu.com/XenialXerus/ReleaseSchedule
<BlackDex> True, but an upgrade for example isn't supported yet
<BlackDex> from 14.04 to 16.04
<BlackDex> that will happen only when the PointRelease is done
<MonsieurBon> Ok, good to know
<BlackDex> maas currently is on 2.0rc2
<BlackDex> so i think it will be final during that time
<MonsieurBon> ok. Hope the same applies to juju. Because juju 2.0 seems far from stable to me :)
<MonsieurBon> thx for the clarification anyway
<virginie__> hi all, I am trying to boot ubuntu VM using iPXE with MaaS
<virginie__> without success. The error is "no bootable device"
<brendand> virginie__, have you configured the vm to boot from the network interface - is it a kvm vm?
<virginie__> brendand, I checked the box and the VM starts using iPXE but I got an error
<virginie__> Maas created a bootstrap vm but I have got this error
<virginie__> error 0x040ee119
<virginie__> the pb was that I have 3 different NIC and the iPXE boot on the wrong one. So I need to press F12 key to see boot menu and select the right interface
<virginie__> also for the minimum kernel, I had to set "trusty" rather than "no minimum"
<roaksoax> ;/win 6
<acovrig> is it possible to login to the provisioning environment?
<kiko> acovrig, yes
<kiko> acovrig, by provisioning, in maas-speak, do you mean deployment? or commissioning?
<acovrig> kiko: I netboot a new client, it boots to an ubuntu login prompt, how do I login to that?
<kiko> acovrig, I assume you uploaded SSH keys?
<acovrig> yes, and I tried logging in from that system as ubuntu and I think as root, what user should it be?
<acovrig> Iâm trying this from 16.04, it hung somewhere on the boot process for quite a while, so I tried 14.04 then it hangs on the ubuntu login after trying to conenct to a 169.254.*.* for a long time
<acovrig> I just let a 16.04 go for several minutes an eventually got the ubuntu login prompt; how long should I expect it to sit at this prompt before it shutsdown after enrollment?
<kiko> acovrig, as ubuntu
<kiko> acovrig, hmmm.
<kiko> acovrig, I don't know in what phase you are getting stuck. are you enlisting the machine, commissioning or deploying?
<acovrig> kiko: Iâm not sure either; from what I understand I can take a new piece of hardware, netboot it, then it will show up in the MAAS web interface so I can decide what OS and software to run on it.
<kiko> acovrig, correct. that first netboot is called "enlistment"
<acovrig> Itâs getting stuck when I netboot it. I believe itâs supposed to enlist in MAAS then shutdown
<kiko> acovrig, once enlistment concludes the machine shuts down
<kiko> acovrig, ah. so enlistment is not concluding. is it the same for all your machines or do some enlist while others hang?
<acovrig> all machines
<kiko> ok
<kiko> acovrig, can the machines reach the internet?
<acovrig> currenlty Iâm doing this in virtualbox on my macbook, but plan to use it to netboot and install MAAS on a physical server (because I have yet to have success installing from USB)
<acovrig> kiko: are they supposed to? the MAAS server has 2 nics, 1->internet, 2->internal network to the physical&VMs, but doesnât have a router on it
<acovrig> am I supposed to manually create FW rules and ipv4_forward rules on the MAAS server so the clients have internet?
<kiko> acovrig, enlistment requires packages be installed, which could come from a proxy. the simplest way to ensure it works is to give clients internet access.
<acovrig> Iâm a little confused on how thatâs supposed to work: do I need to configure the MAAS server to be a router or not use DHCP on the MAAS server and configure a router to give the netboot DHCP option?
<kiko> acovrig, configure the MAAS server to provide DHCP on the segment your clients are on, and have it be a router, or indicate the correct default gateway
<kiko> (if you have another router on the segment)
<kiko> acovrig, does that make sense?
<acovrig> I think so, by default, does it configure itself as the router in DHCP, so if I add the FW rules and enable forwarding in the kernel it should work then?
<kiko> acovrig, I believe it does advertise itself as the router unless you specify otherwise in the settings page
<kiko> acovrig, by FW rules I assume you are going to NAT the clients?
<acovrig> yea, my plan is to make an openstack cluster between 3 servers (for failover); from what I undertand I can have 1 network for internet (LAN) and a few other networks for cluster management
<acovrig> currently my thought was to install ubuntu server from my laptop to the 3 servers over an unused cluster network then convert one of them to be the MAAS controller
<kiko> right, that works fine
<kiko> but indeed, the machines need external access, so you need to set up NAT and routing
<acovrig> :D I got my first machine to show up in MAAS! I guess I missed something in the docs, or something needs to be specified more clearly in the docs. It booted much quicker too, I guess it was trying to conenct for a while during the boot process
<kiko> cool!
<kiko> acovrig, yeah, the process and the failure more is still sadly invisible
<kiko> acovrig, fortunately once it works it really does work
<acovrig> yea, unfortunately Iâve been testing this w/a VM nic bridged to a host VLAN that goes to the server room, and I donât think that will work, so I guess I need to untag my laptop on the cluter network to netboot the servers from my laptop :)
<kiko> heh
<acovrig> too bad virtualbox doesnât do _real_ network bridges
<mup> Bug #1604103 opened: maas doesn't install in xenial unprivileged container <MAAS:New> <https://launchpad.net/bugs/1604103>
<mup> Bug #1604111 opened: MAAS does not allow an interface to be "disconnected" and not have an associated (vlan, fabric) <MAAS:In Progress by blake-rouse> <https://launchpad.net/bugs/1604111>
<mup> Bug #1604103 changed: maas doesn't install in xenial unprivileged container <MAAS:Won't Fix> <https://launchpad.net/bugs/1604103>
<mup> Bug #1604128 opened: [2.0RC2] Unable to add a public SSH Key <MAAS:Triaged> <https://launchpad.net/bugs/1604128>
<mup> Bug #1604128 opened: [2.0RC2] Unable to add a public SSH Key due to lp1604147 <cdoqa-blocker> <MAAS:In Progress by allenap> <MAAS 2.0:New> <MAAS trunk:In Progress by allenap> <maas (Ubuntu):New> <maas (Ubuntu Xenial):New> <maas (Ubuntu Yakkety):New> <https://launchpad.net/bugs/1604128>
<mup> Bug #1604169 opened: maas login yields "ImportError: No module named 'maasserver'" <MAAS:New> <https://launchpad.net/bugs/1604169>
<mup> Bug #1604128 changed: [2.0RC2] Unable to add a public SSH Key due to lp1604147 <cdoqa-blocker> <verification-needed> <MAAS:In Progress by allenap> <MAAS 2.0:New> <MAAS trunk:In
<mup> Progress by allenap> <maas (Ubuntu):Fix Released> <maas (Ubuntu Xenial):Fix Committed> <maas (Ubuntu Yakkety):Fix Released> <https://launchpad.net/bugs/1604128>
#maas 2016-07-19
<mup> Bug #1604339 opened: Node names are hidden in nodes listing <ui> <MAAS:New> <https://launchpad.net/bugs/1604339>
<mup> Bug #1604375 opened: The action panel needs indenting from the edge of the screen <ui> <ux> <MAAS:New> <https://launchpad.net/bugs/1604375>
<ram_> Hi. I installed MAAS on one server. I have taken another two servers in which have the boot type as PXE. MAAS detected nodes. while doing commissioning it was failing. Could you please help me in this
<ram_> I saw some log files under /var/log/maas/ . http://paste.openstack.org/show/537405/
<ram_> http://paste.openstack.org/show/537405/ has log files output. could you guys please help me in solving this issue.
<roaksoax> ram_: /win 11
<roaksoax> ram_: what version fo MAAs are you using ?
<mup> Bug #1604393 opened: MAAS is not listing the storage disks <MAAS:New> <https://launchpad.net/bugs/1604393>
<ram_> roaksoax: Hi. I am using  MAAS Version 1.9.3+bzr4577-0ubuntu1 (wily1)
<roaksoax> ram_: you need to provide cloud-init logs. When you comission, click on the option to prevent power off, and ssh into the machine while commissioning
<roaksoax> ram_: and grab /var/log/cloud-init*
<ram_> roaksoax: OK. Thank you. I will try that now.
<acovrig> How do I tie a controller to a VLAN? the default untagged on fabric-0 is attached just fine, but when I add a VLAN for DHCP it doesnât let me add my controller as the rack controller when configuring DHCP
<roaksoax> acovrig: i dont quite understand what you are trying to do fully ?
<roaksoax> acovrig: you want to add a second rack controller ?
<acovrig> no, add a second network to my existing controller
<acovrig> so I have an interface for internet, and another interface for booting the MAAS machines (unless there is a beter/more best-practice method of doing this)
<mup> Bug #1604424 opened: [2.0 RC2] The proxy server could not handle the request  /MAAS/api/2.0/machines/ -- urllib2.HTTPError: HTTP Error 502: Proxy Error  <oil> <MAAS:New> <https://launchpad.net/bugs/1604424>
<ram_> roaksoax: Hi I tried as you said. I was unable to SSH to MAAS slave nodes. It has given permission denied(public key) Issue. In MAAS UI i added correct public key of MAAS server.
<ram_> roaksoax: http://paste.openstack.org/show/537431/
<mup> Bug #1604424 changed: [2.0 RC2] The proxy server could not handle the request  /MAAS/api/2.0/machines/ -- urllib2.HTTPError: HTTP Error 502: Proxy Error  <oil> <MAAS:New> <https://launchpad.net/bugs/1604424>
<mup> Bug #1604424 opened: [2.0 RC2] The proxy server could not handle the request  /MAAS/api/2.0/machines/ -- urllib2.HTTPError: HTTP Error 502: Proxy Error  <oil> <MAAS:New> <https://launchpad.net/bugs/1604424>
<acovrig> I seem to have this bug (https://bugs.launchpad.net/maas/+bug/1604128) however, Iâm running MAAS 2.0.0 (rc2+bzr5156)
<acovrig> installed from this PPA: http://ppa.launchpad.net/maas/next/ubuntu xenial/main amd64 Packages
<roaksoax> acovrig: the fix has been made available in Ubuntu
<acovrig> roaksoax: what do you mean?
<roaksoax> acovrig: that it is in -updates
<roaksoax> acovrig: but I'm guessing that your version number is different from what's in the archive which is preventing the update
<acovrig> so replace the ppm with maas/updates instead of maas/next?
<roaksoax> acovrig: i'll make a fix available in maas/next
<roaksoax> acovrig: no, I mean ubuntu archives, xenial-updates
<acovrig> oh, I see; apt-cache shows 2.0.0~rc2+bzr5156-0ubuntu1~16.04.2, should I apt-get install maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 ?
<roaksoax> acovrig: if you can try that and tell me if it works ?
<roaksoax> acovrig: i'm uploading a new version on the PPA too
<acovrig> sudo apt update && sudo apt upgrade doesnât show any updates
<roaksoax> acovrig: i'm uploading to the ppa nowish
<roaksoax> acovrig: should be aailable in ~30 mins tops
<acovrig> roaksoax: just curious, would apt-get install maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 solve my issue?
<roaksoax> acovrig: it should, but i've not tried it
<roaksoax> acovrig: i'd suggest for you to wait for the update in PPA
<acovrig> roaksoax: yea, Iâm planning on waiting
<mup> Bug #1604460 opened: In Node details->Machine output->Commissioning output the filename is hidden <ui> <MAAS:New> <https://launchpad.net/bugs/1604460>
<mup> Bug #1604461 opened: [2.0 RC2]  regiond log errors: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key" DETAIL:  Key (ip)=(**.***.**.**) already exist <oil> <MAAS:New> <https://launchpad.net/bugs/1604461>
<acovrig> roaksoax: apt update && apt upgrade is what I should be doing, right; I donât see the update yet
<acovrig> roaksoax: OK, it shows up now, update pending
<acovrig> roaksoax: I updated (2.0.0~rc2+bzr5156-0ubuntu2~16.04.2) and still get âInvalid SSH public key: 'PosixPath' object has no attribute 'path'â
<mup> Bug #1604465 opened: [2.0rc2] RackController.get_image_sync_status causes huge load on regiond process <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1604465>
<roaksoax> acovrig: are you truing to add via the webui ?
<acovrig> roaksoax: yes
<acovrig> roaksoax: I have a VM running ubuntu 16.04 and maas 2.0.0~rc2+bzr5156-0ubuntu1~16.04.1 and it accepted the keyâ¦
<roaksoax> acovrig:that's strange
<roaksoax> acovrig: so it accepted the key with  2.0.0~rc2+bzr5156-0ubuntu1~16.04.1 but not the one from PPA ?
<acovrig> the VM was installed from http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 (apt-cache policy maas)
<acovrig> roaksoax: I see the same version in apt-cache (http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64) in the non-working server, should I attempt to list all the dependencies to âcross-gradeâ to that âversionâ?
<acovrig> roaksoax: do you think apt-get remove maas && apt-get install maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 would work and keep my config?
<acovrig> (with an apt-get autoremove in between)
<roaksoax> acovrig: it should keep your config, let me test something though
<roaksoax> acovrig:  sudo apt-get install maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 python3-maas-client=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-cli=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-common=2.0.0~rc2+b
<roaksoax> zr5156-0ubuntu1~16.04.2 python3-django-maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-dhcp=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-proxy=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-dns=2.0.0~rc2+bzr5156-0ub
<roaksoax> untu1~16.04.2 maas-region-api=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-region-controller=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 python3-maas-provisioningserver=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-rack-
<roaksoax> controller=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2
<roaksoax> acovrig:  sudo apt-get install maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 python3-maas-client=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-cli=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-common=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 python3-django-maas=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-dhcp=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-proxy=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-dns=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-region-api=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-regio
<roaksoax> acovrig: ok, that sould work
<acovrig> roaksoax: Unable to locate package maas-regio and had to add maas-rack-controller=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 maas-region-controller=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 python3-maas-provisioningserver=2.0.0~rc2+bzr5156-0ubuntu1~16.04.2 but it accepted the install then; currently installing, then will try the key again
<roaksoax> acovrig: that's a paste issue
<acovrig> yea, figured as much; *sigh* _still_ installing lol
<acovrig> roaksoax: same issue, I tried from a browser that has never seen MAAS so itâs not a chache issue on the browser side; could the key be partially imported somwhere on the backend?
<roaksoax> acovrig: what if you manually restart maas-regiond and maas-rackd ?
<roaksoax> acovrig: actually, only maas-regiond
<acovrig> roaksoax: sudo service maas-regiond restart I presume?
<acovrig> same issue
<roaksoax> acovrig: weird
<acovrig> roaksoax: yea, Iâm starting to get the âdefying logicâ feelingâ¦
<roaksoax> acovrig: that;s the diuff between version
<roaksoax> https://launchpadlibrarian.net/273938668/maas_2.0.0~rc2+bzr5156-0ubuntu1~xenial1_2.0.0~rc2+bzr5156-0ubuntu2~16.04.2.diff.gz
<roaksoax> acovrig: actually I see where the issue is
<roaksoax> acovrig: let me fix it
<mup> Bug #1558314 changed: maas server is overloaded - twistd and postgres processes > 200% <oil> <MAAS:Invalid> <https://launchpad.net/bugs/1558314>
<roaksoax> acovrig: uploaded a new package. Let's see if that fixes it
<acovrig> roaksoax: apt update && apt-cache policy shows it wanting to upgrade to 2.0.0~rc2+bzr5156-0ubuntu2~16.04.2; update running now
<acovrig> O.o still get the same errorâ¦ is there a md5sum I can check somewhere to sanity-check that the updates are installing propperly?
<acovrig> roaksoax: if it means anything, the working system was installed from the ubuntu 16.04 server iso w/the MAAS option picked on boot, the non-working system was installed from the mini iso (netinstall)
<acovrig> roaksoax: I updated the VM (was working system) and now I get the same error...
<mup> Bug #1604513 opened: Redfish power support <MAAS:New> <https://launchpad.net/bugs/1604513>
#maas 2016-07-20
<oz_> hey guys does anyone know on maas 1.9.3 can i tag multiple vlans on nodes added in maas itself? i was not able to change the part in the networking config on nodes.
<mup> Bug #1604702 opened: Unable to add a 16384-bit ssh key to my account from web UI <MAAS:New> <https://launchpad.net/bugs/1604702>
<mup> Bug #1604702 changed: Unable to add a 16384-bit ssh key to my account from web UI <MAAS:New> <https://launchpad.net/bugs/1604702>
<mup> Bug #1604702 opened: Unable to add a 16384-bit ssh key to my account from web UI <MAAS:New> <https://launchpad.net/bugs/1604702>
<siva> Hi. I have taken 3 physical servers. On one server I installed ubuntu 15.10(Wily). And then installed MAAS. Configured the MAAS Cluster. Now I am trying to register my hardware with MAAS. MAAS done with PXE boot of its nodes and assigned IPs . When i was trying to ssh into the MAAS nodes, it was giving Permission denied (public key issue). In MAAS UI I added public key of MAAS installed server. And finally it was showing commission statu
<siva> Cloud-init v.0.7.5 started running and finished . I am using MAAS Version 1.9.3+bzr4577-0ubuntu1 (wily1).   Finally , the MAAS nodes have stuck with the info -> Cloud-init v.0.7.5 finished at wed, 20 jul 2016 13:01:51  +0000 Datasource  DataSource ConfigDriveNet [net, ver=2] [Source=/dev/sdb1]. UP 43.10sec.
<siva> I observed the Maas log files under /var/log/maas. I didnât get any error messages. But I didnât understand why commission is failing.
<siva> MAAS log info : http://paste.openstack.org/show/537405/
<siva> Could you please provide me the solution why MAAS commission failed.
<mup> Bug #1604738 opened: Dynamic subnet range disappears after edit and is replaced with "No IP ranges have been reserved for this subnet." <MAAS:New> <https://launchpad.net/bugs/1604738>
<acovrig> what debugging steps can I take, deployments are failing and Iâm not sure why
<kiko> acovrig, you'll need to tell us more about the failure first :)
<kiko> siva, hey there
<acovrig> kiko: thatâs what Iâm asking, all I can tell is it says âFailed deploymentâ, how can I learn more about the failure to troubleshoot
<kiko> acovrig, what are you deploying?
<acovrig> ubuntu 16.04
<roaksoax> acovrig: are you using MAAS 2.0 ?
<roaksoax> acovrig: get the installation log from the WebUI (at the bottom of the page)
<roaksoax> acovrig: and if you are using 2.0, look in: /var/log/maas/rsyslog/<machine-name>/../messages
<acovrig> roaksoax: the last thing I get is Node installation - âcloudinitâ running modules for final then Node changed status - From âDeployingâ to âFailed deploymentâ
<acovrig> I see a few âtimed out waiting for reply from <ip> (ntp)â which would explain it trying to hit a status point of 2012â¦
<roaksoax> acovrig: my guess is that it deployed successfully, but on the reboot, it either didn't boot from the disk, or it failed to reach maas after the the first reboot
<roaksoax> acovrig: can you pastebin the cloud-init outout + the install log from the wbeui ?
<acovrig> roaksoax: install log from the webui, where would the cloud-init output be? https://gist.github.com/acovrig/446fc9b616f72979c7995a5132fb607f
<roaksoax> acovrig: yeah, if it is not there, then that means cloud-init/curtin didn't send it back
<roaksoax> acovrig: also, can you pb the rsyslog ?
<acovrig> roaksoax: sure, currenlty I have the BIOS set to PXE boot first, should I not (I.E. have it boot from the HDD after deployment)?
<roaksoax> acovrig: no you don't, giving it a second look it seems something failed before the machine was actually rebooted
<roaksoax> Node changed status - From 'Deploying' to 'Failed deployment'	Tue, 19 Jul. 2016 15:16:52
<roaksoax> Node installation - 'cloudinit' running modules for final	Tue, 19 Jul. 2016 15:16:52
<roaksoax> acovrig: look at the timestamp
<roaksoax> acovrig: so the rsyslog should show why was that ?
<acovrig> roaksoax: OK, gist updated (https://gist.github.com/acovrig/446fc9b616f72979c7995a5132fb607f#file-rsyslog)
<acovrig> roaksoax: interesting: the rsyslog only goes to 13:53, not to 15:16 (time)
<acovrig> roaksoax: sudo grep 15:16 /var/log/maas/rsyslog/HOST-DELL-*/*/messages shows nothing O.o
<kiko> <acovrig> I see a few âtimed out waiting for reply from <ip> (ntp)â which would explain it trying to hit a status point of 2012â¦
<kiko> roaksoax, acovrig: ^^^ is this not relevant?
<kiko> could the times be skewed and failing for that reason?
<acovrig> yea, I watched the physical console a few times and saw it getting a 401 from my MAAS server because of it; what would cause it to not be able to reach ntp?
<kiko> acovrig, IIRC the node is trying to reach ntp.ubuntu.com
<kiko> is that reachable?
<acovrig> kiko: odd, Iâm deploying to 2 nodes now to see if it was just a timing issue; I SSHâd into one of them and it can ping the MAAS server (router for the nodes) but not ping google (8.8.8.8), but it can DNS dig google.com...
<kiko> acovrig, well, typically dig will work, as MAAS is the DNS server
<kiko> acovrig, it looks like routing (or NAT) is broken?
<acovrig> kiko: I have 3 FW rules on MAAS: -A FORWARD -i eno1.8 -o eno1 -m state --state RELATED,ESTABLISHED -j ACCEPT
<roaksoax> kiko: the NTP shouldn't be an issue release
<acovrig> -A FORWARD -i eno1 -o eno1.8 -j ACCEPT
<roaksoax> acovrig: what do you have under /var/log/maas/rsyslog/ ?
<acovrig> and -A POSTROUTING -o eno1.8 -j MASQUERADE
<roaksoax> acovrig: is it just completely empty ?
<kiko> acovrig, the MASQUERADE entry looks wrong
<kiko> acovrig, shouldn't that be -o eno1?
<kiko> oh hmm
<acovrig> roaksoax: no, I have my 2 hosts/date/messages and maas-enlisting-node/date/{messages,ip}
<kiko> roaksoax, right, but if there is no upstream connectivity, could things be failing?
<acovrig> kiko: the MAAS server is on VLAN 8 (tagged) to get to the nodes (untagged)
<roaksoax> kiko: yeah it could, but NTP shouldn't be a cause of failure
<acovrig> donât the nodes use a date to connect back to the MAAS server?
<kiko> acovrig, eno1.8 is external, I understand now
<kiko> acovrig, do the packets appear correctly masqueraded?
<kiko> acovrig, if you plug a laptop into the untagged vlan, can it get out?
<acovrig> kiko: good question, pending test
<kiko> roaksoax, sure, but NTP failing could be an indication that networking in general is failing :)
<acovrig> kiko, roaksoax: a windows 7 client canât access icanhazip.com from chrome, so routing must be broken
<kiko> acovrig, right. fix the networking first and then come back to me. :)
<acovrig> I updated the gist (https://gist.github.com/acovrig/446fc9b616f72979c7995a5132fb607f) with my iptables and ifconfig; not sure whatâs wrong...
<acovrig> sysctl net.ipv4.ip_forward is on...
<kiko> acovrig, check the input and output chains too, otherwise add logging and use tcpdump/wireshark to see what's happening
<acovrig> kiko: OK, not sure what was wrong, but fixed iptables and networking works, just started a deployment, weâll see what happens.
<kiko> acovrig, list out the rules you have now (iptables -L FORWARD -n, perhaps INPUT and OUTPUT too) and put it in a paste?
<acovrig> roaksoax: FYI: I couldnât get maas to accept an SSH key yesterday and was able to install 2.0.0~beta3+bzr4941-0ubuntu1 and now it accepts my key(s).
<kiko> acovrig, cool
<acovrig> kiko: looking at diff, I got my ifs flipped; I guess the guide I looked at used eth1 as internet and eth0 as lan and I didnât notice :/
<kiko> aha
<kiko> thanks
 * acovrig emits a squeel of excitement: âDeployedâ :D 
<acovrig> now to get juju and openstack to work :)
<kiko> acovrig, it worked? w00t!
<acovrig> kiko: yup, now I just need to find a good guide to get juju+openstack to work as a failover cluster for VMs
<roaksoax> acovrig: this should do : https://jujucharms.com/openstack-base/
<acovrig> roaksoax: ooh, thanks
<mup> Bug #1604901 opened: No way to retrieve complete event log through the cli <oil> <MAAS:New> <https://launchpad.net/bugs/1604901>
<kiko> acovrig, all a-ok?
<acovrig> kiko: with MAAS, yes, Iâm still working though juju/openstack âsolutionsâ
<kiko> okay
<G_Dog1985> do i need two server to run maas and openstack or can it be don by on mass
<G_Dog1985> done*
<mup> Bug #1604938 opened: node failed deployment "Exception: I/O operation on closed file." <oil> <MAAS:New> <https://launchpad.net/bugs/1604938>
<mup> Bug #1604962 opened: node set to "failed deployment" for no visible reason <oil> <MAAS:New> <https://launchpad.net/bugs/1604962>
<mup> Bug #1604987 opened: Event log should always include a reason why a node was marked Failed Deployment <oil> <MAAS:Triaged> <MAAS 2.0:Triaged> <MAAS trunk:Triaged> <https://launchpad.net/bugs/1604987>
<mup> Bug #1546760 changed: Re-attempt power on for initial PXE failure during node deployment to alleviate PXE failures <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1546760>
<mup> Bug #1605008 opened: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605008>
<mup> Bug #1605008 changed: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605008>
<mup> Bug #1546760 opened: Re-attempt power on for initial PXE failure during node deployment to alleviate PXE failures <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1546760>
<mup> Bug #1546760 changed: Re-attempt power on for initial PXE failure during node deployment to alleviate PXE failures <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1546760>
<mup> Bug #1605008 opened: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605008>
<mup> Bug #1605008 changed: juju2beta12 and maas2rc2:  juju status shows 'failed deployment' for node that was 'deployed' in maas <oil> <oil-2.0> <juju-core:New> <MAAS:Invalid> <https://launchpad.net/bugs/1605008>
#maas 2016-07-21
<mup> Bug #1605124 opened: When querying the node event log, domain and hostname are not preserved in prev and next URIs <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1605124>
<eeemil> Hi! I'm trying to set up a MaaS-cluster. I have access to a couple of servers within a network that currently are being DHCP:d along with a number of other servers that should _not_ be MaaS:d.
<eeemil> I'm thinking of installing an additional network interface and create a private 192.168.*.*-network for my MaaS-servers, and allow the servers to be internally DHCP:d on the private network.
<eeemil> Are there any drawbacks to this solution? The servers should still be publicly routable on the public network interface, right?
<eeemil> (On a complete other note: what the terminology for a MaaS-server? Is any machine managed by MaaS a MaaS-server? Or only the main server that manages all additional machines?)
<siva> Hi. I am using MAAS Version 1.8.3 (+bzr4053). MAAS failed to Commission. Log files link:  http://paste.openstack.org/show/538979/. Could you please provide me the solution
<siva> Hi. How do we set MAAS Commission time?
<siva> manually
<brendand> siva, what do you mean by commision time?
<eeemil> Could you _please_ try to keep this page up to date with current terminology and such? http://maas.io/get-started (Also, I'm still looking for answers to my questions asked 3.5 hours ago)
<siva> Hi. I am using MAAS Version 1.8.3 (+bzr4053). MAAS failed to Commission. Log files link:  http://paste.openstack.org/show/538979/. Could you please provide me the solution
<siva> How can we edit commission time manually?
<roaksoax> siva: hi! your log doesn't provide information as to why it failed to commisison. you need to provide the console log while the commissioning is happening. That said, however, 1.8 is no longer supported and it is not available on any PPA. 1.9 is still supported in the 1.9 series
<roaksoax> err 1.x series
<roaksoax> eeemil: -> http://maas.io/get-started 1.9
<roaksoax> eeemil: -> http://maas.io/get-started -> MAAS 1.9
<siva> brendand: Hi. For commissioning nodes , MAAS will take some default time. i.e, 10 min or 20 min. How can I change that manually?
<roaksoax> siva: you can't change the time it takes to commission
<brendand> siva, do you have reason to believe it might work if the timeout was longer?
<roaksoax> siva:Jul 21 16:22:35 maas maas.node: [ERROR] slave2: Marking node failed: Node operation 'Commissioning' timed out after 0:20:00. this is telling you that something went wrong
<roaksoax> siva: and it didn't finish commissioning
<brendand> siva, there's really no reason why it should take 20 minutes. commisioning should be very fast
<brendand> as it doesn't install anything
<siva> roaksoax: Ok, Thank you. I will use MAAS 1.9.
<siva> brendand : Thank you. may be something went wrong. I will try to resolve. If not better i will use MAAS 1.9.
<eeemil> roaksoax: are you telling me that the get-started link is for an old version? I'm running MaaS straight out of the ordinary repo for 16.04, wouldnt expect the documentation to be this inaccurate...
<roaksoax> eeemil: you are using MAAS 2.0rc2 -> RC
<roaksoax> eeemil: the get started is for 1.9
<roaksoax> eeemil: that will be updated once 2.0 is final
<roaksoax> eeemil: but I'll file a bug, since that's managed by the web team, not the maas team
<eeemil> Sure. However, one could question why 2.0 is in the LTS 16.04 release then. I skipped adding the dev ppa simply because I wanted to avoid this kind of hassle...
<mup> Bug #1605253 opened: [doc] maas.io/get-started refers to 1.9 and not 2.0 <MAAS:New> <https://launchpad.net/bugs/1605253>
<eeemil> Do you know of any 2.0 instructions on how to get started then, btw?
<brendand> eeemil, this is a bit more detailed but is definitely accurate: http://maas.ubuntu.com/docs2.0/install.html
<eeemil> Thanks!
<eeemil> Btw, I asked earlier if there are any drawbacks if all servers in my MaaS cluster has two network interfaces: one with public routable IP:s, managed by a dhcp I don't control, and one with private IP:s (192.168.*.*) which I dhcp through MaaS
<roaksoax> eeemil: not at all
<roaksoax> eeemil: if you are doing that with MAAS 1.9+ that is
<eeemil> roaksoax: how do you mean? Are there any drawback in doing it with MAAS 2.0?
<eeemil> I'm really confused by the version leap from 1.9->2.0. I've been reading documentation all week and always thought everything was stable (as I'm running LTS and not interested in bleeding edge-stuff right now). Are there any "changes from 1.9 to 2.0"-page? Will I be able to run Juju without hassle on MaaS 2.0?
<brendand> eeemil, juju 2, yes
<mup> Bug #1605312 opened: Unhandled failure in updating lease. django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key" <MAAS:New> <https://launchpad.net/bugs/1605312>
<mup> Bug #1605278 opened: Merge python-django 1:1.9.8-1 (main) from Debian unstable (main) <upgrade-software-version> <MAAS:New> <python-django (Ubuntu):Confirmed> <https://launchpad.net/bugs/1605278>
#maas 2016-07-22
<mup> Bug #1605432 opened: MAAS Bonding Defaults <MAAS:New> <https://launchpad.net/bugs/1605432>
<mup> Bug #1605476 opened: [2.0rc2] Changing DNSSEC validation does not trigger configuration file update <MAAS:New> <https://launchpad.net/bugs/1605476>
<mup> Bug #1605517 opened: Power mode keeps automatically changing from Wakeonlan to IPMI <MAAS:New> <https://launchpad.net/bugs/1605517>
<mup> Bug #1605538 opened: Too many "maas" logger names in use <MAAS:New> <https://launchpad.net/bugs/1605538>
<siva> Hi. I am using MAAS Version 1.9.3+bzr4577-0ubuntu1 (wily1). MAAS failed to commission . I am using Supermicro servers as MAAS nodes. log files info. http://paste.openstack.org/show/539663/
<siva> Could you please provide me the solution
<roaksoax> siva: first, it seems you have failures in downloading images
<roaksoax> siva: and seems something caused the DB to blow up
<roaksoax> siva: did you make any changes to the db directly? imported any type of images or such?
<roaksoax> siva: the first problem i see there is image importing
<roaksoax> siva: that it failed, maybe you are under a proxy or similar ?
<roaksoax> siva: also, exceptions.ValueError: http://archive.ubuntu.com/ubuntu/dists/wily-updates/main/binary-amd64/Packages.gz failed checksum. -> wily... i thjink that's about to EoL?
<virginie__> Hi there, I have a strange issue with my MaaS deployement. It appears that the MaaS is only retrieving CPU and memory information but nothing for the disk. this is a big problem as I can't deploy juju as the MaaS server can't see any disk on the Nodes
<virginie__> What I would like to know is if the bottstrap and controller node running into Vms can be a problem ?
<virginie__> thank you
<roaksoax>  virginie__ 1. what version of MAAs are you using ? 2. are there any logs errors in /var/log/maas/*.log ?
<virginie__> MaaS 1.9.3 coming with OPNFV installation.
<roaksoax> virginie__: the mahines you are trying to commisison, what type of machines are they ?
<virginie__> I found something in the maas.log
<virginie__> maas.lease_upload_service: [ERROR] failed to upload leases: 'str' object has n attribute 'mac'
<mup> Bug #1605756 opened: [ juju2 beta11 ] system show up in juju status as pending but there is no attempt to deploy in maas <oil> <oil-2.0> <juju-core:New> <MAAS:New> <https://launchpad.net/bugs/1605756>
<mup> Bug #1605794 opened: [2.0] migration error <MAAS:New> <https://launchpad.net/bugs/1605794>
#maas 2016-07-23
<mup> Bug #1605794 changed: [2.0] migration error <MAAS:Fix Released by andreserl> <https://launchpad.net/bugs/1605794>
<mup> Bug #1605794 opened: [2.0] migration error <MAAS:Fix Released by andreserl> <https://launchpad.net/bugs/1605794>
<mup> Bug #1605794 changed: [2.0] migration error <MAAS:Fix Released by andreserl> <https://launchpad.net/bugs/1605794>
<AK> Heyllo
<godleon> Hi all, is there any way to flush maas's DHCP address pool, I found the ip range I configured on maas was almost run out of after deploying OpenStack many times.
<x58> godleon: When you destroy your openstack (I assume a JuJu charm installation) it should automatically clear out all of the entries and start back over from the beginning.
<siva> roaksoax: Thank you Roaksoax
<godleon> x58: no.... actually maas(I am using version 1.93) did not do that. So once the ip  was ran out of, I have to reinstall maas for my next deployment.
<jo__> Hello, does anyone know if maas can be used for deploying Ubuntu desktops? I'm looking for something to do what PXE/kickstart does nicely for building CentOS desktops. I'm looking into maas and fai as options.
#maas 2016-07-24
<mup> Bug #1557626 changed: Region is not advertising RPC endpoints. <cdo-qa> <MAAS:Expired> <https://launchpad.net/bugs/1557626>
#maas 2017-07-17
<mup> Bug #1691434 changed: Can not apply stage final, no datasource found! Likely bad things to come! <MAAS:Expired> <https://launchpad.net/bugs/1691434>
<mup> Bug #1685807 changed: MaaS does not discover storage devices on Lenovo System x3650 M5 <MAAS:Invalid> <https://launchpad.net/bugs/1685807>
<mup> Bug #1685807 opened: MaaS does not discover storage devices on Lenovo System x3650 M5 <MAAS:Invalid> <https://launchpad.net/bugs/1685807>
<mup> Bug #1685807 changed: MaaS does not discover storage devices on Lenovo System x3650 M5 <MAAS:Invalid> <https://launchpad.net/bugs/1685807>
#maas 2017-07-18
<mup> Bug #1704993 opened: [2.x, UI, backend] MAAS dashboard shows maas registered containers that were removed <MAAS:Triaged> <https://launchpad.net/bugs/1704993>
<mup> Bug #1704993 changed: [2.x, UI, backend] MAAS dashboard shows maas registered containers that were removed <MAAS:Triaged> <https://launchpad.net/bugs/1704993>
<mup> Bug #1704993 opened: [2.x, UI, backend] MAAS dashboard shows maas registered containers that were removed <MAAS:Triaged> <https://launchpad.net/bugs/1704993>
<zeih> hi
<zeih> anyone around for some hints troubeshooting maas 2.2.0 on ubuntu 16.04. trying to commision ... ?
<zeih> logs and infos: https://paste.ubuntu.com/25118477/
<roaksoax> zeih: can you provide console logs ? or /var/log/maas/rsyslog/<machine-name>/<date>/messages
<zeih> roaksoax: sure. one moment
<zeih> roaksoax: rsyslog is empty. so it is not configured. I have no logs. Following the remote console of the server I can see that dhcp, pxe is scuccessful and the server boots with minimal 14.04.5 image. then the scripts run and checking
<roaksoax> zeih: ok, so i think the machiine is not connecting back to maas
<roaksoax> zeih: is /etc/maas/rackd.conf , maas_url option pointing to localhost ? or to an IP the machine can access to ?
<zeih> roaksoax: localhost
<zeih> maas_url: http://localhost:5240/MAAS
<roaksoax> zeih: change localhost with an IP the machines can connect to
<roaksoax> zeih: then restart maas-rackd
<zeih> route -n
<zeih> Kernel IP routing table
<zeih> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
<zeih> 0.0.0.0         10.100.40.1     0.0.0.0         UG    0      0        0 ens160
<zeih> 10.100.39.0     0.0.0.0         255.255.255.0   U     0      0        0 ens192
<zeih> 10.100.40.0     0.0.0.0         255.255.255.0   U     0      0        0 ens160
<zeih> Ubuntu24HighTowerMAAS# vim /etc/maas/rackd.conf
<zeih> Ubuntu24HighTowerMAAS# cat /etc/maas/rackd.conf
<zeih> cluster_uuid: b05e1736-ef58-4124-b5dc-00885c9b2976
<zeih> maas_url: http://10.100.40.203:5240/MAAS
<zeih> Ubuntu24HighTowerMAAS#
<zeih> done
<zeih> roaksoax: restart done
<roaksoax> zeih: ok, it should work now. the machine will pxe boot and contact the right ip
<zeih> roaksoax: okay, I give it a try.
<roaksoax> zeih: did it work ?
<zeih> roaksoax: it did not work
<zeih> same behavior then before
<zeih> any other idea?
<zeih> it seems like all works fine on the server but the result will not reported back to maas server
<zeih> so.. the action times out after 20min.. and goes to failed state
<zeih> roaksoax: I am stuck :(
<roaksoax> zei/win 4
<mup> Bug #1703984 changed: [2.2] Commissioning fail with local ntp server due to lldp <MAAS:Invalid> <https://launchpad.net/bugs/1703984>
<mup> Bug #1459432 opened: "could not serialize access due to concurrent update" errors in PostgreSQL logs <MAAS:Confirmed> <https://launchpad.net/bugs/1459432>
<agrebennikov> roaksoax, hi there, maybe you can quickly help me with the issue over here?
<agrebennikov> (I mean, maybe it isnt worth the bug)
<pmatulis> agrebennikov, what issue?
<agrebennikov> https://bugs.launchpad.net/maas/+bug/1459432
<agrebennikov> Andres just said it requires a new bug, but maybe I can fix it right away
<agrebennikov> unfrtunately I cannot use pre-release since it is supposed to be a prod cluster
<mup> Bug #1705090 opened: ValueError: number of bits must be greater than zero <cdo-qa> <cdo-qa-block> <MAAS:New> <https://launchpad.net/bugs/1705090>
#maas 2017-07-19
<mup> Bug #1705152 opened: Getting mDNS observation process failed error in regiond.log and commisioning not completing indefinitely <MAAS:New> <https://launchpad.net/bugs/1705152>
<mup> Bug #1705212 opened: Unable to find boot image when erasing disks when releasing <MAAS:New> <https://launchpad.net/bugs/1705212>
<mup> Bug #1705090 changed: ValueError: number of bits must be greater than zero <cdo-qa> <cdo-qa-blocker> <MAAS:In Progress by blake-rouse> <MAAS 2.2:Triaged by blake-rouse> <https://launchpad.net/bugs/1705090>
<mup> Bug #1705254 opened: [2.2, 2.3] RegionService can raise unexpected exceptions for an empty connection list <MAAS:Triaged by mpontillo> <MAAS 2.2:Triaged by mpontillo> <https://launchpad.net/bugs/1705254>
#maas 2017-07-20
<mup> Bug #1467119 changed: MAAS does not factor in BMC addresses to address availability <bmc> <ha> <MAAS:Fix Released> <https://launchpad.net/bugs/1467119>
<mup> Bug #1704212 changed: MAAS marks node 'Deployed' before sshd is up <cdo-qa> <cloud-init:New> <MAAS:Won't Fix> <https://launchpad.net/bugs/1704212>
<mup> Bug #1705418 opened: [2.x] rackd.log doesn't log that it is downloading images <MAAS:Triaged> <https://launchpad.net/bugs/1705418>
<cnf> morning
<mani123> Hello, facing issue while provisioning Dell R710 & PE 2950 through MAAS 1.9. servers shutdown automatically after PXE boot.
<mani123> is this realted to Dell BIOS/Firmware ?
<roaksoax> mani123: wlel, i would need more details to determine that. Do you have any console log ?
<mani123> roaksoaz: not in text format ...had a video of boot process..
<mani123> last line is stopping system V runlevel compatibility. and server shutdown.
<mani123> another shutdown message prints like "shuting down for poweroff "
<mani123> enabled wake on LAN ( IPMI/ ethernet interface ) and i can see maas user being created on IDRAC-6.
<mani123> with administrator.
<roaksoax> mani123: right, but if the machine is in deploying state, it pxe boots, it loads the ephemeral and does stuff
<roaksoax> mani123: so I would like to find out what of that process is not working
<mani123> loads the ephemeral and exchange root keys and then shutdown.
<mup> Bug #1705473 opened: [2.3] MAAS should pass netplan to both cloud-init and curtin by default <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1705473>
<mup> Bug #1705504 opened: on install MAAS has system warnings <MAAS:Triaged by danilo> <maas (Ubuntu):New> <https://launchpad.net/bugs/1705504>
<mup> Bug #1705504 opened: on install MAAS has system warnings <MAAS:Triaged by danilo> <maas (Ubuntu):New> <https://launchpad.net/bugs/1705504>
<mup> Bug #1705508 opened: [2.2]  rackd is not refreshing is commissioning information <MAAS:New> <https://launchpad.net/bugs/1705508>
<mup> Bug #1705508 changed: [2.2]  rackd is not refreshing is commissioning information <MAAS:New> <https://launchpad.net/bugs/1705508>
<mup> Bug #1705508 opened: [2.2]  rackd is not refreshing is commissioning information <MAAS:New> <https://launchpad.net/bugs/1705508>
<dannf> roaksoax: do you know if maas CI has ran w/ the freeipmi from proposed? looking for test results for the sru team
<roaksoax> dannf: no it hasn't, i'd need to test that
<roaksoax> but i wont be able to do that today
<roaksoax> dannf: but it worked fine with what is in -updates
<dannf> roaksoax: cool, thx. i can also unit test if there's hw i can login to
<mup> Bug #1705518 opened: [2.2] 100% cpu usage for 1 regiond process when having hundreds of images <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1705518>
#maas 2017-07-21
<mup> Bug #1705594 opened: [2.2] rackd errors after fresh install <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1705594>
<mup> Bug #1705654 opened: [2.2,2.3] DHCP IP addresses on HA rack controller causes incorrect fabric creation <MAAS:Triaged> <https://launchpad.net/bugs/1705654>
<mup> Bug #1705654 changed: [2.2,2.3] DHCP IP addresses on HA rack controller causes incorrect fabric creation <MAAS:Triaged> <https://launchpad.net/bugs/1705654>
<mup> Bug #1705654 opened: [2.x] DHCP IP addresses on HA rack controller causes incorrect fabric creation <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1705654>
<blizzow> I've got my maas server up and running and I have a couple machines PXE booting. How long do I need to wait for the machines to show up in the nodes list?
<blizzow> Sometimes a machine will show up in the nodes list and I hit commission. Then it just sticks in commissioning. The tasks just stay "pending"
<catbus> blizzow: usually a few minutes for me for new machine to show up.
<catbus> blizzow: Do the power parameters look good on the node page? Are they filled in after it's discovered/shown on the list?
<blizzow> catbus.for one of my hosts it looks alright but failed at testing.
<blizzow> Actually make that two hosts.
<blizzow> I'm also having trouble in this ...https://www.ubuntu.com/download/cloud/autopilot  step six says to run conjure-up --bootstrap-to myhost.  It pukes with -  "Exception: Could not determine LXD version"
<pmatulis> blizzow, for me, ~30 seconds to show up as 'New' (after booting for the first time) and ~2.5 minutes to be Commissioned
<mup> Bug #1705774 opened: Controller refresh fails when a static IP address is used <MAAS:New for mpontillo> <https://launchpad.net/bugs/1705774>
#maas 2017-07-22
<mup> Bug #1705792 opened: [2.2, 2.3] MAAS doesn't filter out progress bars, spinners, and counters from scripts making logs difficult to read <MAAS:In Progress by ltrager> <MAAS 2.2:Confirmed for ltrager> <MAAS 2.3:In Progress by ltrager> <https://launchpad.net/bugs/1705792>
#maas 2017-07-23
<mup> Bug #1681278 changed: bootstrap failure on MAAS <bootstrap> <maas> <juju:Expired> <juju 2.0:Won't Fix> <MAAS:Expired> <https://launchpad.net/bugs/1681278>
#maas 2018-07-16
<xygnal_> roaksoax: according to my co-worker, it was curtin_hooks.py that called the yum commands we spotted
<roaksoax> xygnal_: which I guess that was a customization since we don't have yum commands on our curtin_hooks ?
<xygnal_> roaksoax: there are no yum commands in our cutin config or the scripts it deploys
<xygnal_> which is very confusing because why would we see it as part of curtin hooks if we have NO configs that reference yum commands?
<xygnal_> ie:  if it was somehow put into the image before MAAS got a hold of it, why would it show in curtin hooks
<roaksoax> roaksoax@rivals:~/project/maas-image-builder/contrib/centos/centos7/curtin$ grep -sr yum *
<roaksoax> 1 roaksoax@rivals:~/project/maas-image-builder/contrib/centos/centos7/curtin$
<roaksoax> xygnal_: if you have an installation log to see, that'd be good
<xygnal_> and custom image instead of centos image has the same result?
<xygnal_> (it's centos but it's custom image type)
<roaksoax> xygnal_: yes, it should be (also, you should be able to add custom centos images with name=centos/custom1
<roaksoax> xygnal_: but a install log would be a better way to understand what's happening
<roaksoax> but from looking at the code, i don't see any commands we run inside curtin_hooks
<xygnal_> I'll see if he has that or we can re-create it to show you.
<xygnal_> explain about custom vs centos/custom?
<roaksoax> xygnal_: when you add a image with name='', it will be categorized as custom. When you add it as name=centos/customxyz, it iwll be categorized as "centos" images
<roaksoax> xygnal_: it wont bring any different behavior under the hood
<roaksoax> xygnal_: but for ui/api should better organize images
<xygnal_> interesting, ty.  feature that existed all of this time?
<roaksoax> xygnal_: yes, but IIRC, we fixed API validation that may have prevented from doing so
<sentinel-prime> wow there has been a lot going on today
<ebbex_> Anyone else here using swapfiles on the filesystem rather than a dedicated swap partition?
<sentinel-prime> i need to redo my settup
<mup> Bug #1782007 opened: [2.5] BMC Enlistment - When the machine commissionins/runs hardware testing, there are no cloud-init event stored <bmc-enlistment> <MAAS:Triaged> <https://launchpad.net/bugs/1782007>
<mup> Bug #1782007 changed: [2.5] BMC Enlistment - When the machine commissionins/runs hardware testing, there are no cloud-init event stored <bmc-enlistment> <MAAS:Triaged> <https://launchpad.net/bugs/1782007>
<mup> Bug #1782007 opened: [2.5] BMC Enlistment - When the machine commissionins/runs hardware testing, there are no cloud-init event stored <bmc-enlistment> <MAAS:Triaged> <https://launchpad.net/bugs/1782007>
<mup> Bug #1782009 opened: [2.5] Rack DNS config Failed to update DNS configuration. <MAAS:New> <https://launchpad.net/bugs/1782009>
<mup> Bug #1782025 opened: [2.5] Adding a machine with the non-PXE mac doesn't commission <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1782025>
<mup> Bug #1782035 opened: [2.5] BMC enlistment - When adding a machine with only IPMI credentials, these get overwritten by MAAS ones <bmc-enlistment> <MAAS:Triaged> <https://launchpad.net/bugs/1782035>
#maas 2018-07-17
<mup> Bug #1782060 opened: MAAS deployed Pods can be released <MAAS:New> <https://launchpad.net/bugs/1782060>
<mup> Bug #1682150 changed: [2.4] MAAS node memory detection problem : 0.0GiB <MAAS:Expired> <https://launchpad.net/bugs/1682150>
<mup> Bug #1782144 opened: maas-rack-controller install fails with nginx issue <MAAS:Incomplete> <https://launchpad.net/bugs/1782144>
<mup> Bug #1782144 changed: maas-rack-controller install fails with nginx issue <MAAS:Incomplete> <https://launchpad.net/bugs/1782144>
<bladernr> roaksoax, blake_r_ is there any way to get MAAS to use -proposed to boot the installer for deployments?
<bladernr> 4.15.0 is broken for ppc64el, however a fix is in 4.15.0-24 (proposed currently has 4.15.0-28
<roaksoax> bladernr: no
<bladernr> roaksoax, ok.  thanks
<mup> Bug #1782219 opened: Enlistment preseed does not render full repository configuration <MAAS:New> <https://launchpad.net/bugs/1782219>
<mup> Bug #1782220 opened: MAAS script runner should capture the environment in results.yaml <MAAS:New> <https://launchpad.net/bugs/1782220>
<mup> Bug #1782230 opened: [2.5a1, UI] Cannot disable pockets under the Ubuntu mirrors <MAAS:New> <https://launchpad.net/bugs/1782230>
<xygnal_> roaksoax: any chance of being able to only force the OS disk to be wiped on released?
<xygnal_> as opposed to non boot disks
<mup> Bug #1782234 opened: log_host and log_port not configurable <MAAS:New> <https://launchpad.net/bugs/1782234>
<mup> Bug #1782234 changed: log_host and log_port not configurable <MAAS:New> <https://launchpad.net/bugs/1782234>
<mup> Bug #1782234 opened: log_host and log_port not configurable <MAAS:New> <https://launchpad.net/bugs/1782234>
#maas 2018-07-18
<mup> Bug #1782363 opened: [2.5, UI] Upgrade from 2.4 pod refresh message needs improvement <MAAS:New> <https://launchpad.net/bugs/1782363>
<mup> Bug #1782365 opened: [2.5, UI] Refreshing a pod should tell you what will happen <ui> <ux> <MAAS:Triaged> <https://launchpad.net/bugs/1782365>
<mup> Bug #1782363 changed: [2.5, UI] Upgrade from 2.4 pod refresh message needs improvement <ui> <ux> <MAAS:New> <https://launchpad.net/bugs/1782363>
<mup> Bug #1782365 changed: [2.5, UI] Refreshing a pod should tell you what will happen <ui> <ux> <MAAS:Triaged> <https://launchpad.net/bugs/1782365>
<mup> Bug #1782363 opened: [2.5, UI] Upgrade from 2.4 pod refresh message needs improvement <ui> <ux> <MAAS:Triaged> <https://launchpad.net/bugs/1782363>
<mup> Bug #1782365 opened: [2.5, UI] Refreshing a pod should tell you what will happen <ui> <ux> <MAAS:Triaged> <https://launchpad.net/bugs/1782365>
<mup> Bug #1782409 opened: Adding a SRV DNS record crashes <MAAS:New> <https://launchpad.net/bugs/1782409>
<mup> Bug #1782413 opened: Rack controller has package dependencies on pxelinux and grub <MAAS:Incomplete> <https://launchpad.net/bugs/1782413>
#maas 2018-07-20
<mup> Bug #1782801 opened: [UI] Removing a specific cache set prompts user to remove all cache sets <MAAS:New> <https://launchpad.net/bugs/1782801>
#maas 2019-07-16
<gnuoy> Hi, 3+ hours ago I set off an import of centos images. Ever since the UI displays "Syncing to rack controller(s)". However the cli seems to have them marked as synced. Is there anything I can kick to persuade the ui that they really are there ?
#maas 2019-07-17
<gnuoy> Anyone know of a work around for images struck in "Syncing to rack controller(s)" state ?
<gnuoy> Stopping the rack svc then the region svc, waiting, starting region, then rack seems to have fixed it for me
#maas 2019-07-19
<mup> Bug #1835275 opened: Unable to apply netplan configuration in ephemeral environment <MAAS:Fix Committed> <MAAS 2.6:Fix Committed> <maas-images:Fix Released by ltrager> <systemd:Incomplete> <https://launchpad.net/bugs/1835275>
<ascott> Hello. I am new to maas and learning. Right out of the box though I'm having some trouble. Images are not downloading. Stuck on "Downloading 4%". I tried to contact you guys via a link but when I tried to read your privacy policy and ubuntu privacy policy, they are both 404, so I came here.
<ascott> are there currently any network outages? where are the privacy policies so i can read them before contacting you?
<ascott> maybe this is what is happening? https://askubuntu.com/questions/1153000/maas-image-import-not-syncing-with-conttroller-for-ubuntu-18-04-amd64-image
<ascott> so I just need to wait for a couple days.
#maas 2019-07-21
<mup> Bug #1828829 changed: MAAS can not change into PXE boot on Huawei  1288H V5 servers <MAAS:Expired> <https://launchpad.net/bugs/1828829>
#maas 2020-07-13
<mup> Bug #1887379 opened: [feature] Allow option for machines to automatically commission from the "enlisting" phase <MAAS:New> <https://launchpad.net/bugs/1887379>
<mup> Bug #1887379 changed: [feature] Allow option for machines to automatically commission from the "enlisting" phase <MAAS:New> <https://launchpad.net/bugs/1887379>
<mup> Bug #1887379 opened: [feature] Allow option for machines to automatically commission from the "enlisting" phase <MAAS:New> <https://launchpad.net/bugs/1887379>
<mup> Bug #1887384 opened: [feature] Allow user to provide separate storage location for images through GUI <MAAS:New> <https://launchpad.net/bugs/1887384>
<mup> Bug #1887250 changed: LXD machine failed testing <lxd> <MAAS:New> <https://launchpad.net/bugs/1887250>
<mup> Bug #1887436 opened: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
<mup> Bug #1887436 changed: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
<mup> Bug #1887436 opened: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
<mup> Bug #1887436 changed: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
<mup> Bug #1887436 opened: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
<mup> Bug #1887436 changed: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
<mup> Bug #1887436 opened: [feature] MAAS should populate the power parameter mac_address during enlistment <MAAS:New> <https://launchpad.net/bugs/1887436>
#maas 2020-07-14
<mup> Bug #1887454 opened: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1887454 changed: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1887454 opened: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1887454 changed: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1887454 opened: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1887454 changed: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1887454 opened: ceph-radosgw/0*  are  blocked,Services not running that should be: <email address hidden> <MAAS:New> <https://launchpad.net/bugs/1887454>
<mup> Bug #1878652 changed: "add user" button should be at the top of the page <ui> <MAAS:Expired> <https://launchpad.net/bugs/1878652>
<mup> Bug #1878652 opened: "add user" button should be at the top of the page <ui> <MAAS:Expired> <https://launchpad.net/bugs/1878652>
<mup> Bug #1878652 changed: "add user" button should be at the top of the page <ui> <MAAS:Expired> <https://launchpad.net/bugs/1878652>
<mup> Bug #1887533 opened: Can't add machine when IPMI Address using custom port <MAAS:New> <https://launchpad.net/bugs/1887533>
<mup> Bug #1887550 opened: [feature] MAAS should allow the user to add/edit virsh storage pools via the UI for KVM Pods <MAAS:New> <https://launchpad.net/bugs/1887550>
<mup> Bug #1887558 opened: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. <MAAS:New> <https://launchpad.net/bugs/1887558>
#maas 2020-07-15
<mup> Bug #1874840 changed: Add support for RHEV power driver <feature> <MAAS:Invalid> <https://launchpad.net/bugs/1874840>
<mup> Bug #1874840 opened: Add support for RHEV power driver <feature> <MAAS:Invalid> <https://launchpad.net/bugs/1874840>
<mup> Bug #1874840 changed: Add support for RHEV power driver <feature> <MAAS:Invalid> <https://launchpad.net/bugs/1874840>
<mup> Bug #1887667 opened: Provide capability to template machine names <MAAS:New> <https://launchpad.net/bugs/1887667>
<mup> Bug #1887699 opened: Add machine does not show all supported options <ui> <MAAS:New> <https://launchpad.net/bugs/1887699>
<mup> Bug #1887700 opened: API modifies power_address with special chars  <MAAS:New> <https://launchpad.net/bugs/1887700>
#maas 2020-07-16
<mup> Bug #1887797 opened: Impossible to delete zombie LXD VM <MAAS:New> <https://launchpad.net/bugs/1887797>
<mup> Bug #1887797 changed: Impossible to delete zombie LXD VM <MAAS:New> <https://launchpad.net/bugs/1887797>
<mup> Bug #1887797 opened: Impossible to delete zombie LXD VM <MAAS:New> <https://launchpad.net/bugs/1887797>
<mup> Bug #1887812 opened: maas-rack register does not work as expected in 2.8.1 MAAS snap <MAAS:New> <https://launchpad.net/bugs/1887812>
<mup> Bug #1887812 changed: maas-rack register does not work as expected in 2.8.1 MAAS snap <MAAS:New> <https://launchpad.net/bugs/1887812>
<mup> Bug #1887812 opened: maas-rack register does not work as expected in 2.8.1 MAAS snap <MAAS:New> <https://launchpad.net/bugs/1887812>
<mup> Bug #1878269 changed: Multiple servers both new and old are failing to fully enlist/commission <blocks-hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1878269>
<mup> Bug #1887833 opened: Enlisting a node sometimes requires manual entry of its MAC address <MAAS:New> <https://launchpad.net/bugs/1887833>
#maas 2020-07-17
<mup> Bug #1887877 opened: MAAS-RBAC inconsistent login behavior <sts> <MAAS:New> <https://launchpad.net/bugs/1887877>
<mup> Bug #1887812 changed: maas-rack register does not work as expected in 2.8.1 MAAS snap <MAAS:Invalid> <https://launchpad.net/bugs/1887812>
<mup> Bug #1887812 opened: maas-rack register does not work as expected in 2.8.1 MAAS snap <MAAS:Invalid> <https://launchpad.net/bugs/1887812>
<mup> Bug #1887812 changed: maas-rack register does not work as expected in 2.8.1 MAAS snap <MAAS:Invalid> <https://launchpad.net/bugs/1887812>
<mup> Bug #1887951 opened: Provide API to manage BMC users <MAAS:New> <https://launchpad.net/bugs/1887951>
<mup> Bug #1887955 opened: Add additional support for secure erase <MAAS:New> <https://launchpad.net/bugs/1887955>
<mup> Bug #1887955 changed: Add additional support for secure erase <MAAS:New> <https://launchpad.net/bugs/1887955>
<mup> Bug #1887955 opened: Add additional support for secure erase <MAAS:New> <https://launchpad.net/bugs/1887955>
<mup> Bug #1887955 changed: Add additional support for secure erase <MAAS:New> <https://launchpad.net/bugs/1887955>
<mup> Bug #1887955 opened: Add additional support for secure erase <MAAS:New> <https://launchpad.net/bugs/1887955>
<mup> Bug #1887996 opened: MAAS commissioning process should attempt to use LXD socket <MAAS:New> <https://launchpad.net/bugs/1887996>
<mup> Bug #1887996 changed: MAAS commissioning process should attempt to use LXD socket <MAAS:New> <https://launchpad.net/bugs/1887996>
<mup> Bug #1887996 opened: MAAS commissioning process should attempt to use LXD socket <MAAS:New> <https://launchpad.net/bugs/1887996>
<mup> Bug #1888002 opened: Wrong steps provided to add a rack controller <MAAS:New> <https://launchpad.net/bugs/1888002>
<mup> Bug #1888014 opened: PXE boot fails on Raspbeery Pi 4 missing recovery.elf/start.elf <MAAS:New> <https://launchpad.net/bugs/1888014>
<mup> Bug #1888015 opened: Not possible to add region controller <MAAS:New> <https://launchpad.net/bugs/1888015>
<mup> Bug #1888015 changed: Not possible to add region controller <MAAS:New> <https://launchpad.net/bugs/1888015>
<mup> Bug #1888015 opened: Not possible to add region controller <MAAS:New> <https://launchpad.net/bugs/1888015>
<mup> Bug #1888015 changed: Not possible to add region controller <MAAS:New> <https://launchpad.net/bugs/1888015>
<mup> Bug #1888015 opened: Not possible to add region controller <MAAS:New> <https://launchpad.net/bugs/1888015>
<mup> Bug #1888021 opened: Modify maas-run-remote-scripts to interfact with udev directly <MAAS:Triaged by ltrager> <MAAS 2.7:Triaged by ltrager> <MAAS 2.8:Triaged by ltrager> <https://launchpad.net/bugs/1888021>
<mup> Bug #1888021 changed: Modify maas-run-remote-scripts to interfact with udev directly <MAAS:Triaged by ltrager> <MAAS 2.7:Triaged by ltrager> <MAAS 2.8:Triaged by ltrager> <https://launchpad.net/bugs/1888021>
<mup> Bug #1888021 opened: Modify maas-run-remote-scripts to interfact with udev directly <MAAS:Triaged by ltrager> <MAAS 2.7:Triaged by ltrager> <MAAS 2.8:Triaged by ltrager> <https://launchpad.net/bugs/1888021>
