#maas 2012-06-18
<lifeless> bigjools: yo
<bigjools> g'day
<negronjl> hello all
<lifeless> bigjools: meet Juan, negronjl meet Julian, dev lead for maas.
<negronjl> hello bigjools
<bigjools> hi
<negronjl> bigjools: I am getting ready to do a MaaS demo @ Structure and I am having issues with TFTP
<negronjl> bigjools: TFTP daemon ( I believe tftpd-hpa ) is not running on boot so I started it and now I get TFTP unable to locate configuration file
<bigjools> is manage_tftpd set to 1 in /etc/cobbler/settings?
<negronjl> bigjools: checking ...
<negronjl> bigjools: yes
<bigjools> ok run "cobbler sync"
<bigjools> and it should write a config out
<negronjl> bigjools: ok ... brb ( setup is in the other room )
<negronjl> bigjools, lifeless:  well.... this is embarrassing, I had "cobller sync" in my restart-everything script ....
<negronjl> bigjools, lifeless: that did it by the way
<bigjools> nice :)
<bigjools> great
 * negronjl bangs his head against the wall a couple of times :/
<lifeless> you'll be glad to know cobbler is being nuked atm
<lifeless> so next version should have new and interesting failure modes.
<bigjools> with extreme pleasure
<negronjl> lifeless, bigjools: NICE!!!!!!!!
<negronjl> less banging on head
<negronjl> bigjools: ping
<negronjl> has there been a resolution to but 992075
<negronjl> sorry bug 992075
<ubot5> Launchpad bug 992075 in maas (Ubuntu) "Commissioning status persists with cloud-init 0.6.3-0ubuntu1" [Undecided,Confirmed] https://launchpad.net/bugs/992075
<negronjl> cloud-init-nonet killed by TERM signal
<negronjl> now I'm stuck there.
<bigjools> negronjl: sorry no idea, we need smoser to look at that
<negronjl> bigjools: thx ... I'll ping smoser tomorrow
<lifeless> negronjl: get some sleep :P
<negronjl> lifeless: yup. I think I will ... later all .. and thanks for the help
<rick1> I always get "httpd does not appear to be running and proxying cobbler" when I was running maas-import-isos, is there any hints?
<rvba> roaksoax: hi there, I've got 2 questions for you about the preseed files.
<rvba> roaksoax: first question: where exactly is the maas-enlist.preseed used?  It's used to create an "enlist" profile in scripts/maas-import-isos but then I don't see where this profile is used.
<rvba> roaksoax: second question.  I see you've divided the pressed files in templates and snippets and the snippets are included by the templates.  I see that most of the snippets (actually, all but one) are used only once and most of them are only one lines (or, at most, a few lines).  I was wondering if you would be against some reorganisation for these files.  The first reason, as described previously is that I
<rvba> don't really see the point in having a snippet when it's used only once.  The second reason is that the templating language we're planning to use (tempita) support inheritance (a la Django) but not inclusion.  What do you say?
<rvba> s/one lines/one line/
<roaksoax> rvba: 1. /var/lib/tftpboot/pxelinux.cfg/default
<roaksoax> rvba: when a system PXE boots, and it doesn't find itself in the PXE server, then it shows a PXE menu which the various options (cobbler profiles). The default to be run is maas-enlist
<roaksoax> rvba: 2. uhmmm what snippets are you referrering too. I think that ideally we would like to keep the concept of snippets because each prpovide a different capability that we might not want in the preseed
<roaksoax> rvba: smoser ^^
<roaksoax> err
<roaksoax> smoser: ^^ what are your thouhgts
<rvba> roaksoax: (1.) All right, I'll need to coordinate with jtv then because he's working on tftp right now.  Thanks!
<rvba> roaksoax: (2.)  I'm not suggesting dropping the notion of splitting up the templates into small bits.  I'd like to know if you prefer to keep the mechanism as is or move to a inheritance mechanism (a la Django templates).  I'm asking because only is supported by default by the template language we use :).
<rvba> s/because only/because only the latter/
<rvba> roaksoax: I'm referring to all of the snippets in contrib/snippets/.
<roaksoax> rvba: so the thing is that the concepts of snippets are good because they keep small pieces apart to do different things that can be reused when necessary
<smoser> i dont think anyone is specifically tied to the word "snippets"
<rvba> roaksoax: indeed.  What I'm proposing is somewhat similar.  It allows you to do the exact same thing but uses inheritance instead of inclusion.
<rvba> It's sort of done the other way around.
<roaksoax> rvba: right, but would the preseed be able to say
<smoser> without much information, i suspect that you'll still be rednering the preseed files with an object's ".preseed()" or something.
<roaksoax> rvba: right, but would the preseed be able to say "i want to inherit this feature for XYZ?"
<smoser> so in the end you'll have the same, dynamic result in that you can customize that stuff.
<roaksoax> smoser: right but a snippet can not only contain a dynamic result, but a static one
<smoser> well, foo.packages() can just return "some static package list"
<rvba> roaksoax: definitely, it would be done in a different way though.  Let me give you an example.
<smoser> (and then foo.packages() is used by foo.preseed())
<smoser> an example would be good. but rvba the key element is that you need preseed files, and even kernel command line to be customizable
<roaksoax> and "snippets" if they are kept
<smoser> snippets are a mechanism to get runtime evaluation
<smoser> and we need "pluggable runtime evaluation" in some mechanism.
<smoser> i'm not willing to argue over how that is implemented.
<smoser> s/willing/interested/
<rvba> smoser: preseed files will be customizable.  We've added a simple way for now and we're ready to make it more complex if we need to.
<rvba> smoser: 1. one will be able to add a new directory where templates will be looked up first before resorting to the default (maas provided) ones.
<rvba> roaksoax: 2. We've added a (simple) look-up mechanism: for instance if MAAS is looking up the template named 'enlist' it will first see if 'enlist_$nodearch_$nodesubarch_$hostname' exists.  Then 'enlist_$nodearch_$nodesubarch"â¦ etcâ¦ the 'enlist' and finally 'generic'.
<rvba> smoser: ^
<rvba> roaksoax: so, we would have a 'master' template: http://paste.ubuntu.com/1047323/ and then two child templates inheriting from it: http://paste.ubuntu.com/1047325/ http://paste.ubuntu.com/1047334/
<smoser> rvba, my concern with that is that it does not appear "dynamic"
<rvba> roaksoax: the results would be (respectively) http://paste.ubuntu.com/1047337/, http://paste.ubuntu.com/1047338/
<rvba> smoser: what do you mean by that?
<roaksoax> rvba: and would you be able to evaluate, within the tempaltes, whether a particular system is running precise, oneiric, or quantal? or things like that?
<smoser> by what you show there, ti see no way for me to extend 'top' to do a lookup into a hardware database and output specific information (such as kernel command lines) for that entity.
<smoser> (or actually do any sort of "dyanmic" nature)
<roaksoax> we also would lose the ability to be able to remove stuff from the preseed (or master template), or momentarily comment out. We would have to create another master template and then inherit
<roaksoax> TBH, I personally like the way how cobbler handles templating
<rvba> roaksoax: smoser these template will have access to as much context as you want: the node itself (node.architecture will give you the architecture), the release, etc.
<smoser> if what you have shown us there is the extent of the dynamic nature, then i think i can pretty definitively tell you it will end up not being good enough.
<rvba> smoser: this template system supports if/loops etc.
<rvba> And we can extend it to include python code directly even if it's rather ugly.
<roaksoax> rvba: we currently need importing of poython code
<smoser> i really could care less about "ugly"
<smoser> but thats fine.
<smoser> if we can execute loops, if, and python code, i'm fine with that.
<rvba> smoser: by ugly I mean less easy to test.  We'd rather keep the default templates clean and testable.  But for one-off customization, fine.
<smoser> fair
<rvba> roaksoax: are you referring to the base64 module?
<roaksoax> ya
<roaksoax> rvba: in orchestra we used to have our own custom python module to do stuff
<roaksoax> so I personally think it might be nice to have that ability in case is needed
<rvba> No problem, that's built-in.
<rvba> You can do that kind of stuff (http://paste.ubuntu.com/1047347/) so I'm sure there is all the flexibility we need.
<roaksoax> rvba: so basically, we could do something like: http://pastebin.ubuntu.com/1047349/
<roaksoax> ?
<rvba> roaksoax: well, no, that's the point (or the problem ;)).
<rvba> roaksoax: you would do something like that:http://paste.ubuntu.com/1047351/
<rvba> The master would look like that: http://paste.ubuntu.com/1047352/
<rvba> Basically, it defines 'slots' which are filled in by the child templates inheriting from it.
<roaksoax> rvba: so this is what happens: http://pastebin.ubuntu.com/1047353/
<roaksoax> right?
<rvba> roaksoax: exactly
<roaksoax> rvba: so in order for us to add/remove stuff, we have to write a lot of "code" rather than simply importing a snippet right?
<rvba> roaksoax: which (I think â but what do I know ;)) makes sense because the most specific stuff is then written in the child template.
<roaksoax> rvba: doesn't that make it non-easy to administer instead of having snippets that are easily plugable to the preseed?
<rvba> roaksoax: I don't agreeâ¦ hold on one sec.
<roaksoax> rvba: because in the current approach we have. 1. a preseed. 2. we can easily plug features to the preseed. 3. don't worry about anything
<rvba> roaksoax: We have the exact same feature here, hang on :)
<rvba> roaksoax: so I suppose your point is that you'll ahve to write the 'preseed' bit over and over again right?
<rvba> roaksoax: that's not the case, you'll put it in the master template.  And child templates will still be able to override it (or not).
<roaksoax> here we are kinda like 1. inherit the preseed. 2. manually add stuff 3. create another child used 1 and 2. 4. in order to add more stuff, we either go to 2, or 4, and create 5
<roaksoax> rvba: and that's the thing, can't we simply make all the snippet inheritable?
<rvba> roaksoax: what do you mean by 'make all the snippet inheritable'?
<roaksoax> rvba: that would be much easier. If we can inherit master, can't we just also inherit any other in the same file/temaplte?
<rvba> roaksoax: 'master' is the file name.  You can inherit from whatever file you want as long as it's available on the filesystem.
<rvba> roaksoax: in your example, the bit that is in {{def preseed}} would be put in the master because basically all child template will use this.
<roaksoax> rvba: right, but can't we have various masters?
<rvba> roaksoax: absolutely.
<rvba> We can.
<roaksoax> rvba: so we can have this then: http://pastebin.ubuntu.com/1047370/
<roaksoax> rvba: simply keep the snippet code as "snippet" and in the child file we just "include" the snippet as if it was a master
<roaksoax> rvba: so including 3 masters in a child means, 3 different parts together, right?
<roaksoax> that way, you kind keep the same functionality
<rvba> roaksoax: there is the problem, there is no 'include' tag.  'Inclusion' is done the other way around: by using inheritance.
<rvba> roaksoax: you last paste describe something that is not possible.
<roaksoax> rvba: right, but what I mean, maas.preseed --> inherit maas_proxy snippet code, inherit maas_disable_pxe snippet code
<roaksoax> etc
<roaksoax> rvba: what's the tag used for inheritance?
<roaksoax> rvba: how would you write a template that will inherit
<rvba> roaksoax: inherit
<rvba> roaksoax: you're right, in my examples, s/include/inherit/ ;)
<roaksoax> rvba: http://pastebin.ubuntu.com/1047380/
<roaksoax> would that be possible?
<rvba> roaksoax: yes, but snippets/maas_proxy would have to simply define something.  And that something would then have to be used in that file.
<rvba> roaksoax: I think it would be quicker if we could have a quick chat about this.  Note that I'm not again adding the beloved 'include' tag.
<rvba> roaksoax: can we have a quick chat in a g hangout?
<roaksoax> rvba: sure, but can you give me say 15 mins?
<rvba> roaksoax: sure.
<roaksoax> rvba: and would we be able to do this? http://pastebin.ubuntu.com/1047388/
<roaksoax> rvba: specially note the last part where we are adding stuff to a late_command line, rather than adding various late_commands. and we need to be able to that
<roaksoax> rvba: that's one of the issues that I see
<rvba> roaksoax: let me try something and I'll be able to reply to your question.
<roaksoax> brb
<roaksoax> rvba: sorry, i'm back
<rvba> roaksoax: all right, no worries.*
<roaksoax> rvba: ok so were you able to see whether the above works?
<rvba> roaksoax: it does not.
<roaksoax> smoser: you wnat it?
 * smoser reads backscroll
<negronjl> smoser:  I modified the image as you recommended ... I am now at the point where I get the error: cloud-init-nonet main process (344) killed by TERM signal.  Should I wait it out and then try to log in or is there something else I should be doing?
<negronjl> smoser: I'm in the node ... anything in particular I should be looking at?  While you get back to me I'll be looking around in that node
<smoser> negronjl, log into the system now.
<smoser> and you want to run the cloud-init code that would read the data from the server
<smoser> so... baically
<smoser>  /proc/cmdline has a url on it
<smoser> cloud-init reads that, saves it into /etc/cloud/cloud.cfg.d/91-something
<negronjl> smoser:  ok ... waiting on the system ( as I was waiting I tried it again )
<smoser> negronjl, ok. sorry.
<negronjl> smoser: no worries ... I appreciate you taking the time to help
<negronjl> smoser:  I see the /proc/cmdline output as well as /etc/cloud/cloud.cfg.d/91_kernel_cmdline_url.cfg
<negronjl> smoser:  the only thing i see out of all that that seems out of place is http://ubuntu-maas/MAAS/metadata for the metadata_url ( ubuntu-maas is resolvable to 127.0.1.1 )
<negronjl> smoser: not sure if that's correct or not
<smoser> negronjl, yeah, thats wrong.
<smoser> :)
<smoser> cloud-init should be more annoying in that case.
<smoser> can i 'look at that system at all?
<negronjl> smoser:  it's in an isolated network.  Can you give me an example of what it should look like so I can figure it out?
<negronjl> smoser: I will also try to put this set up online somewhere where we can see it
<smoser> h..
<smoser> can you pastebin /var/log/cloud-init.log ?
<smoser> and /var/log/cloud-init-output.log (if that is present)
<negronjl> I'll give that a shot ...
<negronjl> smoser:  it's worth mentioning that this worked fine the first two times that I tried it.
<negronjl> smoser:  that happened yesterday .....  I then lost my cobbler configuration ( i had to run cobbler sync ) ... it has never worked after that
<smoser> negronjl, so.
<smoser> i think what is wrong is that you need to run dpkg-reconfigure maas
<smoser> (which will re-set the value of the "maas-server")
<smoser> but how is ubuntu-maas resolvable?
<smoser> who is feeding that data, because that is what is wrong.
<smoser> maas told cloud-init to get some data from the url http://ubuntu-maas/MAAS/metadata .
<smoser> clearly, thats bad.
<smoser> :)
<negronjl> smoser: let me try that ...
<smoser> on the node, though?
<smoser> you said "resolvable"
<smoser> resolvable how?
<smoser> 'host ubuntu-maas' ?
<negronjl> ping ubuntu-maas
<smoser> what about 'host ubuntu-maas' ?
<negronjl> host ubuntu-maas ( 127.0.1.1 )
<negronjl> just ran dpkg-reconfigure maas ... trying again
<smoser> k. so who is serving 'ubuntu-maas == 127.0.1.1' ?
<smoser> yo uhave that in local dns somewhere?
<smoser> is maas serving dns ?
<smoser> where is htat coming from. because that is bad.
<negronjl> checking
<negronjl> apparently the maas-server is running a dns server that is serving that address ..
<negronjl> I believe it's via dnsmasq
<smoser> roaksoax, ^
<smoser> you know how that happened ?
<smoser> negronjl's maas-server is telling nodes that 'ubuntu-maas' (itself) can be reached at 127.0.1.1
<smoser> surprisingly that doesn't work for systems other than itself :)
<negronjl> I am grepping around to see if I find out where that is being configured
<roaksoax> negronjl: is this a clean install? what does /etc/hosts show?
<negronjl> roaksoax, smoser: i found it on /etc/hosts on the maas server
<negronjl> roaksoax, smoser: i just don't see how that affects the other nodes
<smoser> negronjl, roaksoax this would seem to be a significant issue
<roaksoax> negronjl: so when you dpkg-reconfigure maas you are using ubuntu-maas right?
<smoser> and quite possible the root of a lot of people's issues
<smoser> as the installer (i think) is going to write a 127.0.1.1 entry for 'hostname'
<smoser> and then maas is going to
<smoser> a.) tell other systems "talk to me by my new hostname, $HOSTNAME"
<roaksoax> smoser: so my thinking is that negronjl used as 'network address' ubuntu-maas instead of the IP address.
<smoser> *and*
<smoser> b.) (via dns) my new hostname $HOSTNAME == 127.0.1.1
<smoser> what do you mean, roaksoax ?
<roaksoax> smoser: since dnsmasq is serving on the localnode, it reads /etc/hosts and founds that ubuntu-maas is 127.0.1.1, and that's why it gets serveed that way
<negronjl> smoser, roaksoax: I commented out the offending /etc/hosts entries
<smoser> what do you mean " negronjl used as 'network address'"
<roaksoax> negronjl: what did you input when you did 'sudo dpkg-reconfigure maas'
<smoser> ok..
<smoser> so i dont know if that is going to fix your problem
<roaksoax> smoser: on sudo dpkg-reconfigure maas, it asks for an IP address which will be the address that it iwll hand out to the clients
<negronjl> smoser, roaksoax: dpkg-reconfigure maas =>  I used 192.168.1.2 ( my network address )
<roaksoax> smoser: that address is set on /etc/maas/maas_local_settings.py
<roaksoax> smoser: as DEFAULT_MAAS_URL
<negronjl> checking /etc/maas/maas_local_settings.py
<smoser> negronjl, so after you make that change
<negronjl> that file still says http://ubuntu-maas/
<roaksoax> smoser: if, by default, you do not set that in DEFAULT_MAAS_URL, it will try to guess the hostname/ip of the node
<roaksoax> negronjl: that's weird
<smoser> then the 'url=' parameter that you saw on the kernel command line should serve you data with a new-and-iproved maas_url
<roaksoax> negronjl: can you add -x to the postinst script to see what's going on please?
<negronjl> I can manually change the DEFAULT_MAAS_URL to the correct value but, It's worth finding out how it got that way to begin with
<roaksoax> negronjl: ahhh I see
<roaksoax> negronjl: is this a clena install?
<negronjl> no idea .. i just got the hardware from the Montreal office
<negronjl> so I don't think so
<negronjl> smoser, roaksoax: proc_cmdline: http://paste.ubuntu.com/1047565/ 91_kernel_cmdline_url.cfg: http://paste.ubuntu.com/1047566/
<roaksoax> negronjl: yeah
<roaksoax> negronjl: i know that the issue is
<roaksoax> negronjl: so they *manually* edit /etc/maas/maas_local_settings.py to use ubuntu-maas instead of the IP address because I'm guessing they also had some kind of external DNS
<negronjl> roaksoax: what should i set that value to then ?
<roaksoax> negronjl: the IP address
<negronjl> roaksoax: doing it now ...
<roaksoax> negronjl: then restart apache
<roaksoax> negronjl: and just in case a sudo cobbler sync
<smoser> roaksoax, well, one other option.
<smoser> is to actually set 'ubuntu-maas' to resolve correcctly
<smoser> which should be acheivable via editing /etc/hosts and restating dnsmasq
<negronjl> that's what I was thinking
<roaksoax> yes
<roaksoax> that too
<roaksoax> but dpkg-reconfigure maas only accepts IP addresses
<smoser> ah
<negronjl> roaksoax: I put the right address in dpkg-configure maas
<smoser> (well, that is kind of silly, but roaksoax and i have argued about that before)
<negronjl> roaksoax: It just did _not_ update the values on maas_local something or other .py
<roaksoax> negronjl: yes, but the checking mechasnims checks for a regex that matches an IP address
<negronjl> roaksoax: ahh .. I get it now
<roaksoax> negronjl: and since DEFAULT_MAAS_URL = ubuntu-maas instead of DEFAULT_MAAS_URL = X.Y.Z.A , then it didn't replace it
<negronjl> we need to change it to an IP on maas_local_???.py
<roaksoax> negronjl: no if you set the IP in /etc/hosts
<roaksoax> (the correct IP)
<negronjl> roaksoax: maybe i missed something but, i thought that, when you dpkg-reconfigure maas, there was a regex that would look for an IP ( presumably in /etc/maas/maas_local_settings.py ) and change it ... that would still be broken if I were to leave "ubunt-maas" in /etc/maas/maas_local_settings.py and just have it resolve properly in /etc/hosts
<negronjl> that would solve _my_ issue but, it wouldn't solve the root issue ... the regex is not cathing the value to change
<negronjl> brb
<negronjl> back
<roaksoax> negronjl: right, becuase in the first place, there shouldn't be a hostname instead of an IP address. that hostname mas changed manually, not by packaging
<roaksoax> negronjl: hence breaking it
<negronjl> trying again
<roaksoax> negronjl: packaging on a clean install sets an IP address. If we reconfigure, it will be changed correctly. However, since someone manually changed from a IP to a hostname by directly editing the config file, then dpkg-reconfigure does update the value correctly as it checks that an IP address is there
<negronjl> roaksoax: I see ... a suggestion would be that, on reconfigure of any kind, ignore what is there and just replace it ... my two cents.
<negronjl> roaksoax: but I understand the issue.
<negronjl> roaksoax: and it is working correctly now
<negronjl> roaksoax, smoser:  Thanks for all of the help.  I'll keep poking at this and hopefully won't need to bother you guys anymore
<smoser> roaksoax, if someone edits the config file, then maas should respect that on dpkg-reconfigure
<smoser> and if they used a dnsname, it should accept that too
<roaksoax> negronjl: yet. could please file a bug? :)
<roaksoax> smoser: I agree it should not specifically check for an IP to be set, but IMHO I think we shouldn't rely on DNS
<roaksoax> smoser:becuase of the issue we just saw
<smoser> well, it did the rigth thing.
<smoser> (which was to do what the human told it to)
<smoser> its fine to not use dns by default
<roaksoax> ack
<kurt_> Can you guys tell me if PXE boot from a kvm client to another bridged VM maas server on a separate host is supported?
<kurt_> ie. should the kvm client PXE boot from the VM maas on another host?
<kurt_> with the current version
<kurt_> or is that a bigjools question?
<smoser> kurt_, it really just depends on the networking i think.
<smoser> i'm not an expert on pxe, but the pxe broadcast must get from the booting-system (or kvm instance, or whatever) to the pxe server, and then the traffic also get back.
<kurt_> I have both interfaces bridged and connected via a dedicated switch
<kurt_> so they are on the same network
<smoser> then i would suspect it to work.
<kurt_> but, you think that functionality for PXE boot for kvm is there then?
<smoser> well, kvm pxe boot works, yes.
<smoser> and if the networking is all set up correctly, then it has no way to *not* work.
<smoser> :)
<negronjl> smoser, roaksoax: how do I get oath token for juju to work on maas
<smoser> i think in th emaas ui somewhere?
<smoser> (you cna get it from the DB for your user-id)
<negronjl> smoser: thx
<smoser> kurt_, did i answer your question?
<kurt_> yes, mostlyâ¦but I wasn't able to get it to work before...
<kurt_> bigjools was having me look at vender?
<kurt_> vendev
<kurt_> or something like this?
<kurt_> I was wondering why he had me going down this path - what advantage it had over straight PXE boot
<negronjl> I have each node ( the HP microservers ) enabled for WoL, I also have them as WoL at the maas-server but they don't start up .... am I missing anything here
<negronjl> ?
<negronjl> smoser, Daviey, roaksoax ^^
<smoser> you are missing something
<smoser> specifically https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/981461
<ubot5> Ubuntu bug 981461 in ifupdown (Ubuntu Precise) "wol broken on HP ProLiant N40L (Broadcom tg3 driver)" [High,Triaged]
<smoser> wake on lan on those is not reliable unless you're really nice to the driver.
<negronjl> smoser:  this demo will not be very impressive at all
<smoser> you just have to be really nice, negronjl
<negronjl> smoser: I will have to "juju bootstrap"  then, find out which machine has been allocated to it and power that machine on ...
<smoser> in the happy path, everything is fine. Daviey or roaksoax might remember more details
<negronjl> smoser: thx
 * smoser wonders if negronjl was being sincere in his 'thx'
<smoser> in the purely happy path, though, i think it is fine. but if you ever are in the point where you're hard power-offing a mchine, it ends up foobarred.
<negronjl> smoser: of course ... it's all data that's needed ... not very happy data but, data nonetheless
<smoser> its not actually terribly odd for wakeonlan to not really be reliable.
<roaksoax> negronjl: they didn't get you the pdu?
<roaksoax> i thought they were getting one
<negronjl> i do have a pdu
<negronjl> actually ... I have two ... none of them arrived in working condition
<roaksoax> really? I'm thinking that 1 might be the one that didn't work at all
<roaksoax> the other one is the new one that should be working
<roaksoax> negronjl: you will have to configure it though
<roaksoax> negronjl: and can't you just do this? juju bootstrap --constraints maas-name=natty
<negronjl> i have tried with them all ... WoL is not happening
<roaksoax> negronjl: there's a process for wol to work. 1. manually start machine and let it enlist. 2. Accept & Commission and manually start the machine and let it commission
<negronjl> Have you guys tried the workaround on bug 981461
<ubot5> Launchpad bug 981461 in ifupdown (Ubuntu Precise) "wol broken on HP ProLiant N40L (Broadcom tg3 driver)" [High,Triaged] https://launchpad.net/bugs/981461
<roaksoax> negronjl: 3. juju deploy will now work
<roaksoax> err wol will now work
<negronjl> roaksoax: i did 1 and 2 ... the machines now show up as ready
<negronjl> but, when I try juju bootstrap ... nothing happens
<roaksoax> negronjl: pastebin /etc/cobbler/power/power_ether_wake.template
<negronjl> k .... give me just a sec
<negronjl> roaksoax: http://paste.ubuntu.com/1047848/
<roaksoax> negronjl: is that the network you are using? /usr/bin/powerwake -b 192.168.1.255 "$power_address"
<negronjl> roaksoax: yes .. that is the broadcast address ( 192.168.1.0/24 )
<roaksoax> negronjl: what if you manually try to WoL a server?
<negronjl> roaksoax: give me a command and I'll run it.
<roaksoax> negronjl: powerwake -B 192.168.1.255 <MAC> of one of the ready servers
<negronjl> roaksoax: trying
<roaksoax> negronjl: melmoth and alexmoldovan had the same issue last week and the "solutions" above fixed the issue. (allowing them to shutdown cleanly without disconnecting them from the power outlet at all)
<negronjl> roaksoax: I'll try that too ... I'll report back in a few minutes ... I also have the Ubuntu Contributing Developer meeting now.
<roaksoax> k
<burnbrighter> negronjl: out of curiosity, why are you sending to the broadcast address - won't that broadcast that command to all nodes? I don't understand the purpose.
<negronjl> burnbrighter: not an expert on this but, the magic packet goes out on the wire and the mac address ( included in the command ) will the pick it up and _turn itself on_
<burnbrighter> right, but don't you want to direct it to the right IP addr?
<negronjl> burnbrighter: i _think_ it works by putting the packet on broadcast ... haven't seen it done by direct ip
<lifeless> burnbrighter: http://en.wikipedia.org/wiki/Wake-on-LAN#Magic_packet
<burnbrighter> kk, thnx
<lifeless> can't use IP
<lifeless> IP isn't available for a powered off machine that uses DHCP, because it isn't holding a lease.
<burnbrighter> gotcha, thnx for the clarification
<negronjl> we need a better solution for the WoL thing ... is this an HP issue ?
<lifeless> negronjl: have you spoken to daviey or the folk that helped mark with this setup ?
<lifeless> negronjl: I'm worried that we're getting close to the wire, when the setup *was* working :(
<negronjl> lifeless: I have ... I have random and intermittent issues ... it may work now and it may not .... frustrating
<negronjl> lifeless: at this point, I am trying to get the WoL part done so I can move on to the juju part of this.
<lifeless> negronjl: cool; let me know if you need more help from $wherever.
<lifeless> negronjl: I presume it needed a rebuild / reconfiguration or some such ?
<roaksoax> burnbrighter: the reasing why the  -B BCAST.address was added is because if the server has multiple network interfaces, it won't send the WoL packets through the right interface
<negronjl> lifeless: I'm not even sure what it needs at this point.
 * negronjl feels like a good kick is in order :)
<roaksoax> negronjl: your best bet would try to make one of those PDU's work
<negronjl> roaksoax: forgive me if this is a stupid question but, how would a pdu help me whith WoL
<negronjl> s/within/with
<roaksoax> negronjl: i meant your best bet would try to make use of the PDU so you don't worry about WoL
<roaksoax> as WoL is unreliable
<negronjl> roaksoax: will MaaS be able to control the PDU ?
<roaksoax> negronjl: MAAS does not control anything in power, cobbler does
<roaksoax> negronjl: but yes, is it an APC PDU?
<roaksoax> it is just matter of doing a couple minor changes
<roaksoax> negronjl: making use of a fence-agent to contorl the PDU in cobbler would be relatively easy
<negronjl> roaksoax:  it is APC PDU
<roaksoax> negronjl: so yeah, should be fairly easy to use with a fence-agent (from fence-agents package)
<roaksoax> negronjl: it will just telnet the APC PDU telling it "turn on outleat where machine 'oneiric' is"
<burnbrighter> roaksoax: is not cobbler going away.  Bigjools is making that happen now?
<roaksoax> burnbrighter: it is
<roaksoax> burnbrighter: the current version still uses a subset of cobbler
<roaksoax> err, a few pieces of cobbler
<burnbrighter> kk
<negronjl> roaksoax: Let me try to set up the PDU first ... if the process is simple enough, would you help me out with it
<roaksoax> negronjl: yes, of course
<negronjl> gotta go get some food ... I'll be back in a minute
<roaksoax> k
 * negronjl is back
<negronjl> trying one last time the whole WoL thing.
<negronjl> roaksoax: if you're around .... I'll take you up on the pdu thing for maas
<negronjl> lifeless: you around ?
<roaksoax> negronjl: is it configured?
<negronjl> roaksoax: all nodes are ready
<negronjl> roaksoax: is what configured ?
<roaksoax> negronjl: pdu
<negronjl> roaksoax: no
<negronjl> roaksoax: didn't want to do anything without talking with you first
<negronjl> roaksoax: should I flush the entire environment or, is it ok to leave the nodes as they are ( Ready )
<lifeless> negronjl: hi, yes, OTP just now, whats up ?
<roaksoax> negronjl: leave them as they are for now. Get the PDU configured first and make sure it is in working condition
<negronjl> lifeless: nm ... roaksoax is helping me.
<negronjl> roaksoax: ok ... I'll ping you when done ...
<roaksoax> negronjl: and that you can assign the outlets to specific machines
<roaksoax> negronjl: so just for now, assign 1 outlet and call it the same name what a node in maas would be called
<roaksoax> i.e. oneiric
<negronjl> roaksoax: got it
<roaksoax> negronjl: and in the meantime, sudo apt-get install fence-agents
<roaksoax> and the command to turn on the oulet would be something like: fence_apc  -a <IP of APC> -l <user> -p <password> -n <name of outlet, or outlet ID> -o on
<negronjl> lborda: are both of the PDUs working ?
<lborda> negronjl, no
<negronjl> lborda: want to make sure that I don't spin my wheels on a PDU that may not be working
<lborda> negronjl, one is broken
<negronjl> lborda: hmmm... why ship it then
<negronjl> lborda:  ok
<lborda> negronjl, no idea, I told them not to ship... :P
<negronjl> lborda: the one that needs the adapter ... is that the one that works ?
<lborda> negronjl, yes
<negronjl> lborda: ok .. let's see if i can get the adapter on it ... it'b being difficult
<lborda> negronjl, yeah... good luck
<lborda> negronjl, we got the pdu with no adapter... do you have one?
<negronjl> lborda: now I'm confused ...
<negronjl> lborda:  There is a PDU with a label that say "Broken PDU" ... that one ( oddly enough ) requires no adapter for me to use
<negronjl> lborda: the other one requires a power adapter ( it's a locking plug ).  There is an adapter here but, it doesn't fit.
<negronjl> lborda:  Do you have any idea which one of these PDUs is the "good" one ?
<lborda> lborda, the "broken pdu" is broken
<negronjl> lborda: ok
<negronjl> lborda: ... and I don't have an adapter for the "good" one.
<negronjl> so ... no PDU then .
<negronjl> ... and WoL is not working either ... this is just GREAT :(
<negronjl> roaksoax: You had mentioned that there is a process by which this works.  Here is what I have done so far ( in case I am just doing this wrong ): ...
<negronjl> - Turn machines on.
<negronjl> - They show up on Maas Web UI as Declared
<negronjl> - Edit the node ( change power to Wake on LAN )
<negronjl> - Accept and Commit
<negronjl> ---- machines do their thing and shut down
<negronjl> - They show up as "Ready" on the MaaS Web UI
<negronjl> roaksoax: Is that the right thing to do so far ?
<roaksoax> negronjl: they have WoL working by defaulkt, you do not have to manually tell them so
<negronjl> roaksoax: ok ... noted ....
<negronjl> roaksoax: now what should I be doing ...
<negronjl> roaksoax: I tried going to the maas ui, getting the token, putting it in envirnonments.yaml and juju bootstrap
<negronjl> roaksoax: juju bootstrap --constraints maas-name=quantal ( my node 1 )
<negronjl> roaksoax: according to maas, the unit get allocated to admin ( my username ) but, WoL doesn't do anything.
<negronjl> roaksoax: if I turn the machine on manually that doesn't get me any further ... juju status still hangs.
<negronjl> roaksoax: atm I am trying again without telling maas that the nodes are WoL ... unless you have any suggestions
<roaksoax> negronjl: ok so isn't quantal deploying?
<roaksoax> negronjl: it needs to install ubuntu, install juju, etc etc
<negronjl> roaksoax: let me try this again.
<negronjl> roaksoax: once the machines are in "Ready" status ... should I be doing anything else ?
<roaksoax> negronjl: when a machine is in ready status it is ready to use, they do not have ubuntu on them
<roaksoax> negronjl: so if youjuju deploy, it will take time for it to install ubuntu on it
<negronjl> roaksoax: ok ...
<roaksoax> after it installs, it will reboot
<roaksoax> and continue the process as if it was a cloud image for juju purposes
<negronjl> roaksoax: when i juju deploy --constraints maas-name=quantal the node doesn't WoL ... I have to start it up manually
<roaksoax> that is, install juju and appropriate deps, use a charm (in case of deployment) etc
<negronjl> roaksoax: I will see if it deploys ( even without WoL )
<roaksoax> negronjl: ok, so that means that the machine probably wasn't shutdown cleanly
<negronjl> roaksoax: k... how do we shut these down cleanly
<negronjl> roaksoax: I can't log into them
<roaksoax> negronjl: add an SSH key in juju
<roaksoax> and you'll have to deploy again
<roaksoax> err
<roaksoax> SSH key in MAAS
<negronjl> trying that ...
<negronjl> roaksoax: am I supposed to add the SSH key _before_ I accept & commission the machines or can I do it now that the machines are "Ready"
<roaksoax> negronjl: i can't recall TBH, as there was a bug on it, but for you to not do that, edit "/var/lib/cobbler/kicstarts/maas.preseed"
<roaksoax> negronjl: and set a password there
<roaksoax> for the ubuntu user
<negronjl> roaksoax: ok ... doing that
<negronjl> roaksoax: done ... should I cobbler sync ??
<roaksoax> nope, not necessary
<negronjl> roaksoax: ok ... trying this thing again ...
<roaksoax> it will take time to deploy though
<negronjl> roaksoax: k.  I have a monitor/keyboard attached to the node so I can see if it fails anywhere
<roaksoax> negronjl: ok cool
<roaksoax> brb
<negronjl> roaksoax: the PDU is out of the question now ... I'll keep working on this and will ping you back when I get stuck again ( which I'm sure will be soon ).
#maas 2012-06-19
<negronjl> This officially sucks ..
<negronjl> now that I have the MaaS WoL thing more or less worked out, juju is not working...
<negronjl> juju bootstrap turns the machine up but, there is no IP address to that machine ... and I can't log in to the node either.
<negronjl> Does anyone know which kickstart is used to load the machine up when it is in the "Ready" state ?  I need to add a password to the ubuntu user there.
<negronjl> I need to take a break from this for a few minutes ... I'll be back in a few
<bigjools> negronjl: don't do that, juju gives the ubuntu user your public ssh key
<negronjl> bigjools: don't do which part ... I've done so many things that I am going crazy and losing track.
<bigjools> <negronjl> Does anyone know which kickstart is used to load the machine up when it is in the "Ready" state ?  I need to add a password to the ubuntu user there.
<negronjl> bigjools: I am having issues with juju now ( got past the WoL issues )
<negronjl> bigjools: after the nodes are "Ready" I can't get juju bootstrap to finish
<negronjl> bigjools: I am able to do "juju bootstrap" or "juju bootstrap --constraints maas-name=quantal"
<bigjools> negronjl: how do you know it's not finished?
<negronjl> bigjools: I have a monitor hooked up to the node
<negronjl> bigjools: it WoL the system, installs everything that needs to be installed ( system, juju, etc. )
<negronjl> bigjools: after i end up @ the login promt, I try "juju status" but, I get "no route to host"
<negronjl> bigjools: i can't even ping the node.
<negronjl> bigjools: I was wondering how to get inside of the node ( the juju bootstrap node ) to see if I can figure out what is happening
<bigjools> are you relying on zeroconf for name lookup, or dns on the maas server?
<negronjl> dns on maas server
<bigjools> and is it correct?
<negronjl> it is ... when I "host <insert node name>" it replies correctly
<bigjools> can you ssh ubuntu@host?
<negronjl> no ... "no route to <nodename>"
<negronjl> it doesn't seem to be a DNS issue as I cannot get to the node by address
<negronjl> I am working on 192.168.1.0/24 and I have checked every IP ...  the node doesn't appear to have an IP
<bigjools> it doesn't sound like a juju or maas problem
<bigjools> did the dhcpd assign an ip?
<negronjl> bigjools: according to the DHCP ... no
<negronjl> bigjools: The node doesn't appear to be asking for one
<negronjl> bigjools: That's why I was asking for a way to get in the node so I can ( with the monitor/keyboard ) check things out.
<lifeless> negronjl: are you running precise? is that what was used to do the demo ?
<negronjl> lifeless: yes and yes
<lifeless> ok cool
<lifeless> uhm
<lifeless> bigjools: ip allocation will be dhcp on each node right ?
<lifeless> negronjl: have you got tcpdump / wireshark running on the dhcp server, see what comes in ?
<negronjl> lifeless: yes but, when I do "juju bootstrap"  I don't see the node asking for an address
<lifeless> ok, even though it powers up ?
<negronjl> lifeless:  yes .. the node powers up and it appears to install correctly ... it then reboots and it leaves me at the login prompt.
<lifeless> during the install, it must do networking - do you see dhcp requests during that period ?
<negronjl> I then do "juju status" and I get a "no route to host"
<lifeless> If you don't, it suggests that the wireshark config is wrong, if you do, but don't later, it tells us either that a) dhcp is being more persistent than usual :P[unlikely] or b) dhcp isn't trying on the reboot.
<lifeless> negronjl: you could try using arp to locally associate the address used during the install with the node, and see if it responds to that via ping/ssh
<negronjl> ahh ... I could try that lifeless
<negronjl> lifeless:  give me a minute to try it out
<lifeless> sure
<lifeless> also check you see DHCP for the install phase.
<negronjl> lifeless: i will
<bigjools> I have had it up to HERE with ubuntu problems already today
<lifeless> welcome back
<negronjl> bigjools: amen to that :)
<bigjools> three reboots
<bigjools> and it's 11:40am
<negronjl> just so we are all on the same page .. I am monitoring DHCP traffic at the maas server with tcpdump -i eth0 -n port 67 and port 68
<negronjl> I just did a "juju bootstrap"  ( leaving maas to pick a node for me )
<negronjl> I see the DHCP requests from the node and the responses from the DHCP server
<negronjl> hah .. i found something
<negronjl> i found that the IP that the node got differs from what's expected.
<negronjl> ie:  the node's name is natty so, when i do "host natty"  I get "192.168.1.103" but, when I look at the tcpdump output, I see that the address is 192.168.1.168
<negronjl> I cannot ping 192.168.1.103 but, I CAN ping 192.168.1.168
<negronjl> I CANNOT ssh into 192.168.1.168 though ... it asks me for password but, I don't have one
<bigjools> is cobbler set to manage DNS and DHCP?
<bigjools> it is supposed to set the same IP in both
<bigjools> if it doesn't then things will break like this
<negronjl> cobbler is managing DNS but, not DHCP
<bigjools> there's your problem
<negronjl> there's one of my problems ...
<bigjools> :)
<lifeless> optimist :P
<bigjools> ha
<negronjl> lol
<lifeless> why isn't cobbler managing DHCP ?
<negronjl> this is the way the setup was sent to me
<negronjl> but, even if I change that ... i still have the problem that I cannot log int
<negronjl> s/int/in
<lifeless> well
<lifeless> so cloud-init may not have run
<lifeless> because its not got the right ip now
<negronjl> ok ... let's try to fix the DNS/DHCP issue first i guess....
<lifeless> definitely
<negronjl> I have a dd-wrt router in the mix acting as DHCP ... let me pull that out of the equation...then I will need a bit of help fixing the DHCP in the maas server ...
<negronjl> brb ...breaking things :)
<bigjools> /etc/cobbler/settings
<bigjools> manage_dhcp: 1
<lifeless> negronjl: the whole thing should be its own standalone network, with (I believe) a local archive mirror and everything
<negronjl> bigjools:  checking /etc/cobbler/settings now
<negronjl> lifeless: i have the local mirror thing working.
<lifeless> negronjl: or at worst, with the maas server itself attached on one side to the maas cluster LAN,a nd on the other to your network.
<lifeless> again, AIUI, second hand knowledge ;)
<negronjl> oh wow ... according to /etc/cobbler/settings .... the maas server is managing both dns and dhcp ( manage_dhcp: 1 manage_dns: 1 )
<negronjl> this doesn't seem right ....
<bigjools> check that your node has got an entry in the dhcpd config
<bigjools> is it using dnsmasq or isc?
<bigjools> it should map a mac to an IP
<negronjl> bigjools: ok .. let me check ...
<negronjl> let's start with this ... the maas server should have a static ip address
<negronjl> right now it doesn't
<negronjl> it is assumes that it will be 192.168.1.2 ( I am working with a 192.168.1.0/24 network )
<negronjl> I should probably fix the maas server first
<lifeless> yup
<lifeless> if its dual attached, it can be dhcp on the external side
<lifeless> but the cluster side has to be pretty fixed ;)
<negronjl> lifeless: the maas server is a laptop with a broken screen :/
<negronjl> and one interface only
<lifeless> no wifi? or wifi only ?
<lifeless> negronjl: it arrived broken?
<negronjl> lifeless: it did although I have a vague recollection of a conversation with mrussell ( Mark Russell ) that it may have been broken before shipment to my house.
<negronjl> maas server has a static ip now
<negronjl> now checking dhcpd ( /etc/dhcp3 ) configuration
<lifeless> cry for help sent
<negronjl> bigjools: in MaaS ... what handles the DHCP server ( I see maas-dhcp package installed but, don't really know where the config resides )
<lifeless> its lacking all technical info about where we are at; wrong audience for that I think.
<negronjl> lifeless: thx ... well said .. cry for help
<bigjools> negronjl: cobbler does everything
<bigjools> maas is just driving it
<negronjl> bigjools: ok .. so earlier you had mentioned for me to make sure that I had something in dhcp config ... where should I be looking ?
<bigjools> dhcp config is written by cobbler
<bigjools> so we need to see if it wrote it right
<bigjools> so, just looking for that MAC to IP mapping so the node gets the IP address we expect
<negronjl> bigjools: I'll start a node to see how it behaves now
<bigjools> ok
<negronjl> hah ... i have to laugh because I don't want to cry ... the maas server just froze ... rebooting it
<bigjools> you are seriously unlucky
<negronjl> bigjools: what handles dhcp and dns here ...
<bigjools> negronjl: it's in settings
<negronjl> I'm really not trying to ask stupid questions ... I am just trying not to assume anything at this point
<bigjools> so you can tell cobbler to drive isc dhcp or dnsmasq
<bigjools> it's dnsmasq out the box IIRC
<negronjl> it appears to be dnsmasq
<negronjl> bigjools: where is the config for dnsmasq ?
<bigjools> /etc/dnsmasq.conf?
<negronjl> i may have to rebuild this environment .. ( maas-flush and all ).  I think that having two dns/dhcp servers in the network ( cobbler and the router ) may have screwed the config on the nodes
<bigjools> very likely
<negronjl> rebuilding now ... it'll take a few minutes
<negronjl> bigjools: now pxe booting is not working .... TFTP open timeout
<bigjools> yay
<negronjl> bigjools: i have cobbler using the system's tftp ... should I change it so cobbler uses it's own instead ?
<negronjl> system's tfpt => tfptd-hpa ( /etc/init.d/tftpd-hpa )
<bigjools> that should be ok
<negronjl> the system's ?  it was working before so it seems to be a config issue somewhere in cobbler
<bigjools> you must have been use pxe on the router before
<negronjl> apparently
<bigjools> using*
<bigjools> I don't know what's best here, try flipping and see
<negronjl> ok
<negronjl> nope ... I think I was getting further with the router ...
<negronjl> maybe try having the router pass the pxe and dhcp and have the maas server handle dns
<negronjl> bigjools: ^^
<bigjools> I don't know what's best, sorry
<bigjools> you need roaksoax perhaps
<bigjools> if he is up
<lifeless> negronjl: its what, 8pm for you ?
<negronjl> lifeless: it is
<lifeless> negronjl: ok; I'm going to break for a late lunch in a minute
<lifeless> negronjl: whats the network layout you have in use?
<negronjl> lifeless: flat network ( 192.168.1.0/24 )
<lifeless> negronjl: e.g. nodes + maas server on one switch, where does your router come into it, and so on.
<negronjl> lifeless:  router is 192.168.1.1 ( gw and dhcp )
<negronjl> lifeless: 192.168.1.2 -> maas-server
<lifeless> negronjl: you haven't turned dhcp off ?
<lifeless> on the router that is
<bigjools> are all the necessary daemons running on the maas box?
<negronjl> lifeless:  i did turn dhcp off on the router but, pxe stopped working
<negronjl> bigjools: they are
<lifeless> negronjl: ok, leave it off please :)
<lifeless> negronjl:  you won't have your router at Structure :>
<bigjools> dnsmasq, dhcpd, bind, tftpd
<bigjools> and tgtd
<negronjl> lifeless:  the router was shipped here with the rest of the stuff so, yes .. i could
<negronjl> bigjools: no dhcpd nor bind ... dnmasq instead
<lifeless> negronjl: oh, it was? See, bad assumptions R us :)
<negronjl> bigjools: tftpd-hpa for tftp
<bigjools> ok
<bigjools> so we need to work out why the node cannot see a pxe server (ie dhcpd)
<bigjools> firewalled?
<negronjl> bigjools: straight cable
<lifeless> negronjl: are the nodes plugged into the router, or a switch/hub ?
<negronjl> lifeless: switch
<negronjl> here is the setup
<negronjl> one switch where all of the nodes and maas-server are connected
<negronjl> maas server is a laptop running the above mentioned daemons
<negronjl> that is all
<lifeless> the router is plugged into the switch?
<negronjl> 9 nodes ( HP microservers ) one switch and one laptop.
<bigjools> can you get a pxe test client?
<negronjl> when I was using the router ... it was also plugged in the switch to provide dhcp/dns
<lifeless> ok, cool.
<lifeless> so, default switch environment is to forward everything, pxe should work fine there.
<lifeless> negronjl: got wireshark up? can you get the DCHP OFFER contents, that should have the pxe metadata in it.
<negronjl> lifeless: pxe was working fine with the router but, I believe that the nodes were getting their IPs from the DHCP but not updating the DNS.
<negronjl> I believe the DHCP was on the router but, DNS was on router AND maas server ...
<negronjl> not sure but, I really believe that there was a conflict between the router's dns/dhcp and the maas-server's dns/dhcp
<lifeless> sure, so we now have dhcp and dns both set to managed in cobbler
<lifeless> and wol is firing up the node
<lifeless> and its failing to PXE boot ?
<rick1> execuse me, does anyone know the MaaSslave's default username and password ?
<lifeless> rick1: your ssh key should be copied into the ubuntu user for you by cloud-init.
<negronjl> lifeless:  with all the wires running across my dining room table, I may have found something ... let me try again .. give me just a second
<lifeless> rick1: If I remember correctly.
<rick1> lifeless: so, I can just login to it?
<lifeless> ssh yes
<lifeless> it won't have a password at all
<rick1> lifeless: Thanks, I'll try
<negronjl> lifeless:  pxe booting is now working off of the maas server ( no router involved )
<negronjl> I am going to continue deploying to see if things are better now
<lifeless> negronjl: what was wrong?
<roaksoax> negronjl: that machine has been broken for ages and that's the same one used at the ODS demo
<bigjools> rick1: the user is "ubuntu"
<negronjl> network loop
<rick1> bigjools: oh, Okay
<roaksoax> negronjl: use the dd-wrt router for DNS/DHCP
<roaksoax> negronjl: don't use maas
<lifeless> roaksoax: ohhai!; I'll let you steer :)
<negronjl> roaksoax: i see now that by using the maas-server for dns/dhcp i can't spoof the archives ...
<negronjl> roaksoax: I'll switch it back ... give me just a second
<negronjl> roaksoax: do I need to turn dns/dhcp off in cobbler now that the router is back in the picture ??
<roaksoax> negronjl: yeah don't use maas-server as DHCP/DNS because you'll have to manually assign what IP addresses the clients will get throught DHCP
<roaksoax> negronjl: yes you do
<roaksoax> negronjl: the dd-wrt should already e confogured for DNS/DHCP so you shouldn't really have any issues there
<rick1> lifeless: It shows Connection reset by peer, It must be something wrong with the PXE installation.... always shows commissioning on the WebUI
<negronjl> roaksoax: just to be sure .... manage_tftpd and manage_dns should both be set to 0 ( and not 1 ) on /etc/cobbler/settings ??
<roaksoax> negronjl: manage_dns manage_dhcp
<roaksoax> negronjl: we *do* manage_tfptd
<negronjl> roaksoax: sorry yes .. those two
<negronjl> roaksoax: manage_dns and manage_dhcp
<negronjl> roaksoax: they are both set to 1 on /etc/cobbler/settings
<roaksoax> negronjl: I can't recall (i'm zombie already), but you should be able to dpkg-reconfigure maas-dhcp and disable DNS/DHCP
<lifeless> roaksoax: negronjl: I'm going to take that lunch break. Will check in in ~40
<negronjl> lifeless: thx
<negronjl> roaksoax: I'll do that ..
<roaksoax> lifeless: provecho :)
<negronjl> roaksoax: did dpkg-reconfigure but, I don't get the option to disable it.
<negronjl> roaksoax: trying again ( i disabled it directly in /etc/cobbler/settings ) at this point, i am just trying things out
 * roaksoax checks
<roaksoax> negronjl: try purging maas-dhcp
<negronjl> roaksoax: will do
<negronjl> roaksoax: not sure if you are still around but, I have a question....
<roaksoax> negronjl: shoot
<negronjl> roaksoax: After the node is in "Ready" state ... can I then run "juju bootstrap" ?
<negronjl> roaksoax: or is there anything that I have to do between Ready and juju bootstrap
<roaksoax> negronjl: ready state are ready to be used
<negronjl> roaksoax: so, juju bootstrap would be ok at that point ?
<roaksoax> negronjl: yes,
<negronjl> roaksoax: thx ...  I think I should be good for now ... thank you VERY much for all you patience and help during this.
<roaksoax> no worries
<roaksoax> negronjl: we've been there
<roaksoax> :)
<lifeless> hows it looking guys?
<rick1> Still stuck on "Commissioning" state
<lifeless> rick1: I don't know the states all *that* well, but I think that means 'trying to turn it on and install it'
<rick1> lifeless: yes
<lifeless> rick1: are you using wake on lan or a power board or something? or just reaching down and pushing the power button ?
<rick1> after maas slave PXE booting, scrolling of many screen of messages, stay on "cloud-init-nonet killed by TERM signal" for a while, then jump to login screen
<rick1> lifeless: yes, I set to wol
<lifeless> bigjools: / roaksoax: ^ any thoughts?
<bigjools> no idea why that TERM happens, I think we went through all the usual suspects
<bigjools> so I'd have to defer to roaksoax or smoser who have more knowledge of cloud-init than I
<negronjl> I have had so many issues today that I may be able to help with this.
<negronjl> rick1:  would you pastebin the contents of /etc/maas/maas_local_settings.py please ?
<rick1> ok, wait a moment
<negronjl> rick1:  I am looking for the value of: DEFAULT_MAAS_URL
<negronjl> rick1:  In my case ( out of many ), cloud-init was not working correctly because I had an issue where DEFAULT_MAAS_URL was not properly set to the IP address ( not the hostname ) of the maas-server
<rick1> negronjl: DEFAULT_MAAS_URL = "http://10.20.0.1/"
<negronjl> rick1:  also make sure that the time on your nodes ( all of them ) as well as the maas-server is the same .. weird .. I know but, it helps
<bigjools> oauth FTL
<negronjl> bigjools:  do I have to upload my SSH key to the maas server before i do anything with it ?
<rick1> negronjl: oh, you are right, the time is different
<negronjl> bigjools: juju status ( after a successful juju bootstrap ) gives me Invalid SSH key
<bigjools> negronjl: no, unless you plan on starting nodes manually in the UI
<negronjl> rick1:  put them all on the same time
<negronjl> bigjools: I get Invalid SSH key ....
<bigjools> negronjl: yeah that happens if it's still booting before its run the preseed that adds the user data
<bigjools> if the console is at the login prompt then it's not booted properly
<bigjools> wrong image or failure to get metadata
<negronjl> bigjools: hmmm... how do I avoid that ?  Did I do something wrong here ?
<bigjools> negronjl: the clock problem?
<bigjools> perhaps
<negronjl> bigjools: nope ... all nodes are time synced
<negronjl> bigjools: maas-server as well
<bigjools> is it at login prompt?
<negronjl> bigjools: yup .. .and I can't log in to figure what may be wrong ...
<bigjools> ok is there anything in maas-server syslog from the node?
<negronjl> Test WP failed, assume Write Enabled
<negronjl> Asking for cache data failed
<negronjl> Assuming drive cache: write through
<rick1> negronjl:now time is synced, but still the same problem.
<negronjl> Test WP failed .....
<negronjl> rick1:  .... 10.20.0.1 is your maas server right ?
<rick1> negronjl: yes
<lifeless> negronjl: is juju bootstrap running for you now ?
<negronjl> lifeless: it does but, I cannot juju status ( invalid ssh key )
<lifeless> check your environments config is right and so forth ?
<negronjl> rick1:  dpkg-reconfigure maas ... make sure your maas server ip is correct ...
<negronjl> rick1: sudo cobbler sync
<negronjl> rick1:  try again
<bigjools> negronjl: watch the console during boot, see if anything else looks bad
<rick1> negronjl: ok
<rick1> negronjl: still the same. and I found a bug report about this. https://bugs.launchpad.net/ubuntu/+source/maas/+bug/992075
<ubot5> Ubuntu bug 992075 in maas (Ubuntu) "Commissioning status persists with cloud-init 0.6.3-0ubuntu1" [Undecided,Confirmed]
<negronjl> rick1:  for me, it was a simple case of /etc/maas/maas_local_settings.py having the wrong DEFAULT_MAAS_URL
 * bigjools adds that to the FAQ
<rick1> negronjl: :), I'm go out and take a break, thank you VERY much
<negronjl> rick1:  no worries .. just paying it forward ... I have been helped a LOT
<bigjools> please consider amending or adding to the https://answers.launchpad.net/maas/+faqs
<negronjl> bigjools, lifeless: good news:  I am now able to ssh ubuntu@<node_ip>
<negronjl> bigjools, lifeless: bad news:  the dhcp address associated with the name does NOT match what MaaS thinks it should be
<negronjl> bigjools, lifeless: MaaS and Juju think that my node is 192.168.1.101 but, my node got an address of 192.168.1.127 ... I have to check the DHCP server to see what may be going on but, I will have to do that tomorrow ... I am calling it a night ...
<bigjools> negronjl: what is dhcp serving?
<negronjl> bigjools: checking that now ... 'cause as far as I know ...the router should be the _only_ DHCP  running and it has static dhcp clients
<bigjools> negronjl: mac address mismatch then
<bigjools> check static config vs what maas has
<negronjl> bigjools: checking that too but, gotta be honest ... I'm pretty tired ... I'll be checking that and then calling it a night
<negronjl> what do you mean ... what maas has ?
<bigjools> maas stores macs for the nodes
<bigjools> and tells cobbler what to write
<bigjools> so if you're now doing your own static config then it's probably different
<negronjl> bigjools: do tell ... the router's DHCP/DNS seems to have all of the right MAC addresses ...
<bigjools> how do you know?
<negronjl> bigjools: can you tell me where to find what maas-server has ?
<bigjools> look in its UI
<negronjl> bigjools: that seems to be right as well
<bigjools> are you sure the DHCP server on the router is assigning the IP then?
<negronjl> bigjools: checking again
<bigjools> also, in some dhcp setups you need to explicitly say that pxe boots can have the same IPs as regular boots
<negronjl> done for tonight ... I'll fight some more with this tomorrow
<negronjl> thank you very much for the help all
<lifeless> sleep well
<bigjools> nn
<SpamapS> life	o/
<SpamapS> lifeless: o/
 * SpamapS wonders what happened to irssi there
<lifeless> o/
<lifeless> you hit tab and typed too fast
<lifeless> and triggered its 'paste detection'
<lifeless> which, FTW, bypasses all character evaluation.
<SpamapS> ahhh
<SpamapS> probably the bursty ssh
<SpamapS> actually I'd swear its only bursty when I'm in tmux
 * czajkowski hugs jtv 
<jtv> czajkowski: all for that little email?  *blush*
<jtv> And hi to you too :)
<czajkowski> jtv: yes you at least replied!
<czajkowski> simple things in life realy
<jtv> :)
<flacoste> roaksoax: have you seen the question from Brian G. Peterson on the maas-devel list? he's having problems with the avahi boot story
<flacoste> is the image supposed to be working?
<flacoste> is the error he's encountering the same than you did when you tested with matsubara?
<roaksoax> i havent
<roaksoax> i will address it in a bit
<flacoste> roaksoax: thanks!
<jtv> Daviey: where on the TFTP path do the enlistment kernels live?
<jtv> Or do we use the same as install/commissioning?
<Daviey> jtv: wherever makes sense to you.
<Daviey> enlistment kenel == install kernel
<Daviey> comissioning kernel is sperate
<jtv> OK, I'll use the install kernel then.  Thanks.
<jtv> Daviey: do we know yet exactly what kernel options we need to append for enlistment?
<rvba> jtv: I think it's the options of maas-enlist in /var/lib/tftpboot/pxelinux.cfg/default
<rvba> jtv: am I right?
<Daviey> roaksoax: do you have those to hand?
<jtv> rvba: not sure I understand you correctly.  Are you referring to the maas-enlist profile in Cobbler?
<rvba> jtv: no, I'm referring to the line " append initrd=..." next to the "LABEL maas-enlist" bit in the file /var/lib/tftpboot/pxelinux.cfg/default
<jtv> Ah, thanks.
<jtv> Unfortunately I don't seem to have that bit there.
<rvba> jtv: http://paste.ubuntu.com/1049268/
<jtv> Whoa that's a lot bigger than what I've got
<roaksoax> Daviey: the kernel options
<roaksoax> Daviey: yeah those are appended automatically by cobbler
<roaksoax> err
<jtv> Weird stuff in there...  multiple locale options
<jtv> Yeah, talking to me, I know.  :)
<roaksoax> some of them are appended automatically
<roaksoax> jtv: some of the ones appended in the default file come from maas-import-isos and some from the commissioning
<jtv> From the commissioning?  But this is for a node that isn't even ready for commissioning yet.
<roaksoax> jtv: the default ones are: append initrd=/images/precise-i386/initrd.gz ksdevice=bootif lang=  text  hostname=precise-i386 domain=local.lan suite=precise
<roaksoax> jtv: when you import an ISO it creates the file that appends a few kernel args
<jtv> We need a way to organize generation of this stuff I guess.
<roaksoax> jtv: when you run maas-import-isos, it appends other required kernel args
<roaksoax> jtv: and when you run maas-import-ephemerals, it adds different kernel args
<roaksoax> jtv: so we just need a default, and we need to be able to add and modify them as needed
<jtv> That's all plumbing we don't have yet, I think.
<roaksoax> jtv: the addition of new custom kernel args is done by both, maas-import-isos and maas-import-ephemerals. So we should be able to do similar way to add more kernel args. However, we should be able to also add kernel args per node and as needed
<jtv> But we're driving this from the inside out, rather from the outside in as Cobbler does.
<jtv> In Cobbler, the scripts make this stuff up because they need to create profiles and that's where this stuff gets stored.  We're driving it from the database, independently from the import scripts.
<roaksoax> jtv: right, so import a netboot or server ISO into cobbler and check what args are added by default :).
<roaksoax> jtv: should be similar to http://pastebin.ubuntu.com/1049288/
<jtv> Have to organize this in some way that makes it reusable.
<rvba> roaksoax: another question for you: right now, a node, once the install is done, calls "wget "http://$http_server:$http_port/cblr/svc/op/nopxe/system/$system_name" -O /dev/null".  We've already created the equivalent method (to disable PXE boot) on the metadata API.  The trouble is that the call to this method needs to be oauth-authenticated.  I'm afraid I'm going to have to expose that method anonymously.  Do
<rvba> you have a better idea?
<rvba> roaksoax: I'm asking you because I'm not sure what the limitations are exactlyâ¦ could we use a tool with oauth support instead of wget?
<jtv> We have some code that we were hoping to extract into a library at some pointâ¦
<jtv> Based on the juju code.
<jtv> src/apiclient/maas_client.py
<roaksoax> rvba: can't we just do similar way to how cloud init does it?
<rvba> roaksoax: what would that mean exactly?
<roaksoax> rvba: hold on, have an idea
<roaksoax> rvba: ok, so for cloud-init, MAAS gives cobbler the MAAS_PRESEED which is a base64 blob right?
<rvba> roaksoax: yes
<rvba> roaksoax: in the cobbler-less world, this is not base64 encoded any more, but obviously the same data is present in the preseed context when it gets rendered.
<roaksoax> rvba: ok, so even better then, in the template, we can just get the auth credentials and hand them to curl
<roaksoax> rvba: unless ytou want to store them in base64. So in the template, we just get the variable where that blod is stored, decode it, separate the credentials and give them out to curl
<rvba> roaksoax: are you sure that curl supports oauth?
<roaksoax> rvba: weren't we using it at the beginning for enlistment? before we decided not to?
<rvba> roaksoax: I /think/ we were using a python script.
<roaksoax> rvba: if it does not, then we need it to be unauthenticated
<roaksoax> rvba: unless we get a python script that runs the credentials
<roaksoax> rvba: which might be a PITA
<rvba> roaksoax: why would it be a PITA?
<roaksoax> rvba: downloading a script of late_command and executing it
<roaksoax> rvba: another thing we could do would be do something we used to do with juju/orchestra
<roaksoax> rvba: which was basically have a base64 blob that decoded into a .py file which was run
<roaksoax> rvba: it should allow us to import python-oauth
<roaksoax> rvba: but it is even more prone to errors
<rvba> Indeed.
<rvba> And less testable.
<roaksoax> indeed
<roaksoax> that's the same issue with wgetting a python script and having to do the same thing
<roaksoax> rvba: so I think we'd have to have it unauthenticated for ease
<rvba> Yep, I'm afraid that's the only simple solution.
<rvba> Note that we will have kinda the same problem when exposing the preseed files.  The preseed contains authentication creadentialsâ¦ and I'm afraid we will have to expose it in the open.
<rvba> roaksoax: because that preseed url will be passed as a kernel argument (url=...) and I suppose we don't have a way to use some kind of authentication when fetching that file do we?
<roaksoax> rvba: right, but the preseed should only be obtainable by the PXE client machine and not by simply browsing the server http
<roaksoax> rvba: or only exposed on the network which PXE is happening
<roaksoax> rvba: and not that I know of.
<rvba> roaksoax: how can we prevent someone browsing the server to fetch that file?
<rvba> roaksoax: I mean how can we tell the request comes from the legitimate client machine?  Using the IP would be a start but kinda fragile.
<roaksoax> rvba: i have seen websites on which for instances, if you access to http://www.test.com/files/ it does not list the files, however, if you access test.com/files/file1.xyz it is downloadable
<roaksoax> rvba: it is probably just hidding
<roaksoax> it
<rvba> Preventing directory listing is not a proper protection.
<rvba> Besides, this is an API so there is no directory listing anyway.
<roaksoax> yeah, never mind me then
<roaksoax> err don't mind me :)
<rvba> roaksoax: anyway, I think we will concentrate on having full feature parity with cobbler without making things worse w.r.t. security.  And then we will iterate on that.  Thanks for your help.
<roaksoax> welcome
<negronjl> ... and back again with this
<negronjl> roaksoax: 'morning.  have another question for ya
<negronjl> roaksoax: yesterday you had mentioned that we should be using the router as DHCP/DNS
<negronjl> roaksoax: If we are using the router as DHCP/DNS should we turn dnsmasq off on the maas server ?
<negronjl> roaksoax: or is the router acting as a relay ( DHCP/DNS relay that is ) ?
<roaksoax> negronjl: when you remove maas-dhcp dnsmasq will no longer be running for cobbler
<roaksoax> sudo cobbler sync should update the settings to not run it
 * roaksoax brb
<negronjl> roaksoax: ok ...thx
<jtv> Daviey, roaksoax: actuallyâ¦ should the kernel options for enlistment be based on the âephemeralâ kopts or on the âinstallâ kopts?
<negronjl> I am now getting a Bad archive mirror error
<negronjl> roaksoax, Daviey: i need help with the mirror ... it doesn't seem to be working.  I get "Bad archive error" on the screen.  On syslog ( on the node ), I get mirror does not have any suite symlinks and mirror does not support the specified release (precise) but, when I run the wget command manually ( wget -q http://archive.ubuntu.com/ubuntu/dists/precise/Release -O - | grep -E '^(Suite|Codename):' I do get Suite: precise .... this st
<negronjl> opped working when I stopped dnsmasq BTW to allow the router to answer DHCP requests.
<negronjl> Daviey, roaksoax:  nm ... i figured it out
<Daviey> spoofing archive.ubuntu.com to the wrong ip address?
<negronjl> Daviey: not really sure but, I just edited /etc/dnsmasq.conf and commented out all of the dhcp parts and it seems to be working now
<Daviey> negronjl: wait, on the router?!
<negronjl> Daviey: the router is the DHCP/DNS
<negronjl> Daviey: I disabled dhcp on dnsmasq on the maas server
<Daviey> negronjl: hmm, who installed dnsmasq on the maas server?!
<negronjl> Daviey:  no idea .. it was like that when I got it.
<negronjl> Daviey: I still need dnsmasq running on maas server.  If I don't, the nodes complain about a bad archive mirror ...
<Daviey> hmm
<negronjl> Daviey: so I just commented the dhcp parts of /etc/dnsmasq.conf and it seems to work better.
<Daviey> negronjl: the router should handle all dhcp/dns..
<negronjl> Daviey: now I am to the point where juju bootstrap works ( still without WoL ) but, when I do juju status, the command hangs
<Daviey> or at least it did, when i left it
<negronjl> Daviey: the router does
<Daviey> juju status should hang, until the boostrap node is ready, no?
<Daviey> negronjl: WOL should work, only when a node is cleanly shutdown
<Daviey> negronjl: i did suggest attempting to switch out the power control 'card' in the APC to the other one.
<Daviey> The one which you can plug into the wall, has a duff management card.
<Daviey> I hoped you could just swap it out.
<negronjl> Daviey: I could try that.  I should open both PDUs and see about that
<negronjl> Daviey: that would still leave me with the juju status problem.
<Daviey> negronjl: juju status should hang until the ootstrap node is ready
<Daviey> is it ready?
<negronjl> Daviey:  it is at the login prompt ( has been like that for 10 minutes already ). Nothing is happening.
<negronjl> Daviey: juju status is still hanging
<Daviey> negronjl: okay, then this wouldn't seem to be a maas problem as such.. but a juju problem
<Daviey> have you logged in to see what the heck is going on?
<Daviey> zookeeper poorly?
<lifeless> negronjl: Daviey: how is it looking?
<negronjl> lifeless, Daviey:  I am trying again.
<negronjl> lifeless:  I have the MaaS part working ( no WoL yet ).  juju bootstrap is working but, juju status hangs
<negronjl> lifeless: at the same time, Daviey recommends i open the PDUs and take a part out of one to put it into the other to see if the PDU works.
<Daviey> I haven't seen the new one, but if they are the same model.. that makes sense.. the old model had a duff management computer
<Daviey> WoL is fragile at best, but handled with care.. and a dry run on stage to prepare them.. should leave them in an adequate state
<negronjl> Daviey:  I'm trying to open the PDUs right now ... re: WoL, I am trying to see if I can reproduce it to where I can consistently have WoL working ... so far .. no luck.
<Daviey> (WOL only works if cleanly shutdown)
<negronjl> Daviey: within the maas process, how do you go about cleanly shutting them down when maas controls the entire process.
<Daviey> negronjl: doing a dry run, turning them on by hand.. then briefly pressing the power button to shut them down.
<Daviey> (brief press)
<negronjl> Daviey: WoL is now working ....
<negronjl> Daviey: waiting on bootstrap node ( connecting monitor to it )
<negronjl> lol ... now I get Invalid SSH key ...
<Daviey> :S
<lifeless> Daviey: seen that before ?
<Daviey> lifeless: no
<lifeless> SpamapS: roaksoax: ^ ?
<Daviey> It's not clear to me if it is the juju ssh key, or the maas ssh key failing
<Daviey> if there is no maas ssh key, then it could be a hint that juju/zookepper has failed.
<Daviey> (or not yet online)
<SpamapS> lifeless: sorry I missed the context
<Daviey> negronjl: can you confirm that there is an accurate ssh key in MAAS ui?
<Daviey> attacking the problem by allowing maas to inject the ssh key AND juju inserting the same key makes much sense.
<lifeless> SpamapS: we've now gotten everything seeming to run up, but juju status is whinging about the ssh key being invalid
<SpamapS> raw ssh works?
<Daviey> juju would be using the same ssh key as the laptop, so i can't imagine raw ssh would work
 * Daviey wonders if negronjl is using the laptop, rather than his own machine.
<SpamapS> worth a try with ssh -v
<Daviey> i'd rather negronjl validates the ssh key in MAAS ui first.
<SpamapS> ok I still dont' know what I'm being asked. :)
 * Daviey suspects it is null
<negronjl> Daviey: I am using the ssh key in the maas server
<negronjl> i have NOT put any ssh key in the maas server UI
<Daviey> if the database has been reset since i last had my mits on it, it will be empty.
<Daviey> you need to copy and paste the public key into the maas ui, under Pref's
<negronjl> Daviey:  ok ... let me try that ... brb
<Daviey> ... my hunch is that juju/zookeeper is borked.. and you can't get into it, as no key has been injected.
<Daviey> negronjl: on a side note, did you update the local apt mirror?
<negronjl> Daviey: no ... this is an isolated network
 * lifeless hopes thats good :)
 * negronjl too .  It has the same archive as it did when it was "working"
<Daviey> well, if it were me.. i'd have updated tto precise release.. curently it's pre-release archive
<Daviey> negronjl: well, juju seems fragile in that snapshot.
<Daviey> i'd hope that precise-release is more stable :)
<lifeless> negronjl: so, you're adding the ssh key to maas and trying again I guess?
<negronjl> Daviey: this experience has me second-guessing everything that i do fearing i may break more things ...
<negronjl> Daviey: Do you think I should update the mirror ?  If so,  how ?
<Daviey> negronjl: i don't blame you at all.
<Daviey> negronjl: Lets see the current status.. juju destroy-enviroment .. add you key to the MAAS ui
<Daviey> (your key == the laptop key)
<Daviey> do a juju bootstrap.. so at least you can sanely get into the box
<Daviey> then, if we hit the same problem again.. SpamapS will be a better person to help debug why juju has failed.
<Daviey> (sorry SpamapS!)
<negronjl> Daviey: did that already .. waiting on the node to come to life
<Daviey> negronjl: cool
<SpamapS> Daviey: no worries, I'm good at ignoring other peoples' problems
<Daviey> hah
<Daviey> negronjl: if i am not mistaken, the public key should be http://pb.daviey.com/ITE8/
<negronjl> Daviey: it is
<Daviey> negronjl: for /what to do/, get inspiration from /home/ubuntu/charms/timed-deploy.sh
<Daviey> utils.py is really helpful
<negronjl> Daviey: i have.  It is useful indeed.
<Daviey> negronjl: timed-deploy.sh was an effort to leave it in an install loop, bootstrap, deploy, teardrop
<Daviey> teardown
<SpamapS> hah
<SpamapS> freudian slip?
<SpamapS> the deploy does indeed bring tears to some peoples' eyes :)
<Daviey> SpamapS: one of my servers is called teardrop, so automagic hangs :)
<Daviey> s/hangs/hands
<Daviey> damn, i need to EOD
<negronjl> Daviey: thx for all of your help :)
<Daviey> negronjl: I won't go just yet, but my typing suggests i should :)
<negronjl> Daviey: ah .. ok  ... well thanks for your help anyway ... and stick around to help some more :)
<negronjl> The bootstrap machine is done.... juju status still hangs
<negronjl> I was able to ssh into the box
<SpamapS> negronjl: status may be hanging on zookeeper starting up
<Daviey> with the ssh key in MAAS aswell?
<SpamapS> sshd comes up before zk
<negronjl> i see that /var/log/cloud-init-output.log has something about juju-admin:  error: unrecognized arguments: --constraints-data=......
<Daviey> right, but if the ssh public key came from the metadataservice,he should be able to directly ssh now.
<Daviey> of crap
<negronjl> SpamapS: zookeeper is running
<SpamapS> negronjl: I bet you have an old juju
<SpamapS> negronjl: dpkg -l juju
<negronjl> node:  504
<negronjl> maas server: 539
<negronjl> that is not right :/
<SpamapS> yeah but that should support --constraints-data
<SpamapS> oh wait
<SpamapS> node 504
<SpamapS> yeah thats broken
<SpamapS> negronjl: apt-cache policy juju
<SpamapS> where did that come from?!
 * SpamapS saw 504 and thought 540
<Daviey> SpamapS: hapen to know off the top of your head what precise juju paa is?
<Daviey> i want to rule out that it isn't usin that
<SpamapS> juju should have 531
<SpamapS> precise rather
<SpamapS>      0.5+bzr531-0ubuntu1.1 0
<SpamapS>         400 http://archive.ubuntu.com/ubuntu/ precise-proposed/universe amd64 Packages
<SpamapS>      0.5+bzr531-0ubuntu1 0
<SpamapS>         500 http://mirrors.kernel.org/ubuntu/ precise/universe amd64 Packages
<negronjl> SpamapS: this does appear to now be a juju issue
<negronjl> SpamapS: I see that "juju status" is trying to connect to the node but, it just hangs there .
<negronjl> SpamapS: on the node, I do see juju.agents.machine and juju.agents.provision running
<negronjl> SpamapS: I am also able to ssh into the node ( ssh ubuntu@<node> )
<negronjl> SpamapS: any thoughts?
<SpamapS> negronjl: apt-cache policy juju
<SpamapS> negronjl: wherever that 504 came from, thats your problem
<SpamapS> negronjl: its well before constraints landed, and your client is incompatible with that version entirely
<negronjl> SpamapS: damn ... that means that I have to refresh my mirror
<SpamapS> perhaps a very out of date mirror on the node?
<negronjl> SpamapS: I need help doing that
<SpamapS> negronjl: yeah get that mirror up to date, *OR* point your machine at it, and use the same client version
<negronjl> SpamapS: got it. apt-mirror correct ?
<SpamapS> no idea
<SpamapS> I mean, thats likely
<negronjl> SpamapS: no worries ... I found a script that does it for me ....
<SpamapS> Daviey: ^^ can you help negronjl get the mirror updated?
<Daviey> cool
<negronjl> Daviey: I found a create-mirror.sh script in the maas-server.
<negronjl> it seems to create the mirror
<Daviey> negronjl: that is the one we used, yes.
<negronjl> Daviey:  running it now ... it needs to downlod 5.5G of data so, it will take a while ...
<Daviey> cool
<SpamapS> 5.5G? ugh
<SpamapS> negronjl: hopefully you're not doing that on conference wifi ;)
<negronjl> SpamapS: nope ... this is an isolated network
<negronjl> SpamapS: I am downloading everything that I need into the maas server .
<negronjl> SpamapS: and spoofing archive.ubuntu.com so I can server it all locally.
#maas 2012-06-20
<burnbrighter> bigjools: I saw you updated bug 992075 (this is kurt_ btw  - I finally registered a real nick)
<ubot5> Launchpad bug 992075 in maas (Ubuntu) "Commissioning status persists with cloud-init 0.6.3-0ubuntu1" [Undecided,Confirmed] https://launchpad.net/bugs/992075
<bigjools> hi
<burnbrighter> hi there
<burnbrighter> you say this is expected per bug 1015223
<ubot5> Launchpad bug 1015223 in cloud-init (Ubuntu) "cloud-init-nonet main process killed by TERM signal" [Low,Triaged] https://launchpad.net/bugs/1015223
<burnbrighter> however, the behavior I see after that message is the system (VM in my case) boots to local disk then thus bypassing PXE boot
<burnbrighter> (after the timeout period)
<burnbrighter> this has been a real problem with my set up
<burnbrighter> and yes, the clocks have been checked in my particular case
<bigjools> pxe booting is before that message so it can't be bypassing it
<burnbrighter> I'll try again.  I found a debug option to turn on in maas_local_settings.py
<burnbrighter> How does this happen...
<burnbrighter>   File "/usr/lib/python2.7/dist-packages/maasserver/api.py", line 420, in create_node
<burnbrighter>     raise ValidationError(form.errors)
<burnbrighter> ValidationError: {u'mac_addresses': [u'Mac address 00:0c:29:91:61:a8 already in use.']}
<bigjools> burnbrighter: you're adding a new node and its mac address is already defined for another
<burnbrighter> bigjools: thanks.  It was an old error message.
<burnbrighter> bigjools: how goes the cobbler replacement stuff?
<bigjools> it goes :)
<bigjools> a little tricky but getting there
<burnbrighter> good stuff.
<burnbrighter> that bug about ubuntu and WOL on HP servers - could that apply to other types as well, or only observed on that particular model?
<bigjools> no idea
<bigjools> I expect it'd be news if so
<burnbrighter> kk
<rvba> frankban: Hey Francesco, I've got a review for you if you're available: https://code.launchpad.net/~rvb/maas/pxe-off-preseed/+merge/111225
<frankban> sure rvba, on it
<rvba> ta
<rvba> frankban: please add allenap as a second reviewer when you'll be done with it.
<frankban> rvba: ok
<cheez0r> hey maas guys, I know that Cobbler is deprecated, but I'm trying to access the web interface from a MaaS node- but the http://<host>/cobbler path just gives me a list of files, not the Cobbler WebUI.
<cheez0r> Any tips on what path I can use to access the Cobbler WebUI and/or how I can enable the web interface for Cobbler from the MaaS-installed version?
<cheez0r> apt-get install cobbler-web fails due to broken packages
<roaksoax> rvba: so I';m just about to sent a diff to debian for the yui packaging changes
<roaksoax> rvba: if it gets accepted, I'll upload a yui3.5 source package that contains the change in installation method
<roaksoax> Daviey: smoser ^^ just FYI.. this is what's blocking me from getting a maas package ready
<rvba> roaksoax: cool.  Do you know how long it could take to be rejected/accepted?  I suppose you can't know that in advance but based on previous experience maybe you have an estimate.
<roaksoax> rvba: so I'm just submitting a debdiff for them to review. If they accept is up to the maintainer to upload the package. once they do, I'll manually sync it over Ubuntu (yui2.8). This should be days
<czajkowski> rvba: allenap jtv you each have a mail from me :)
<roaksoax> rvba: so I'm just submitting a debdiff for them to review. If they accept is up to the maintainer to upload the package. once they do, I'll manually sync it over Ubuntu (yui2.8). This should be daysom debian
<roaksoax> this should be a day or so
<rvba> All right.
<roaksoax> rvba: now, to get yui3.5 would bempretty simple with the new install directories in ubuntu
<Daviey> roaksoax: ok, if it doesn't get accepted by Monday, we'll carry a diff.. if it's not making progress. deal?
<roaksoax> Daviey: ok, so this is what I would do. If they don't accept the changes, I'll update yui to 3.5 and start carrying a diff (for which we would have a yui source package instead of yui3.5 source package)
<roaksoax> Daviey: my plan was to first make them accept changes in tt 'yui' source package, so then we can upload a 'yui3.5' source package, that won't have conflicting installation paths
<roaksoax> rvba: ^^
<rvba> czajkowski: I'm not sure round robin is an appropriate method here.  Some questions will require knowledge that only some of us have.
<Daviey> roaksoax: that sounds like a winner
<rvba> roaksoax: sounds good to me too.
<rvba> roaksoax: btw, do you need me to fix up the usage of raphaeljs now or can it wait ~1 week?
<roaksoax> rvba: it can wait... either way it is a tiny js lib
<czajkowski> rvba: right but also posting to the list isnt ideal either so if yours does suit, then tell me and I'll find someone else but the questions do need to be answered.
<roaksoax> rvba: I'll, however, upload a new librapahel to archives
<roaksoax> rvba: so is ready to use when you do so
<rvba> roaksoax: cool.
<roaksoax> rvba: oh btw.. now yui source shipts a build/* and a api/* dir. I';m installing build under /usr/share/javascripts/yui/<ver>/ and api inside that dir, makes sense?
<czajkowski> rvba: trying to be fair, I had 3 questions, so you each got one.
<czajkowski> I've spoken to mrevell about this and agreed it there as we are trying to support the users and questions
<rvba> czajkowski: I completely agree that we should make an effort to answer questions.  I'm just not sure the round robin method is appropriate.
<czajkowski> well posting to the list means it goes into the unknown
<czajkowski> and there is no accountabilty for when it will be answered and by whom
<rvba> czajkowski: I'll have a look at "mine" though.  At first glance it just looks like something for the packaging guys.  but I guess this will be true for most of the questions ;).
<czajkowski> rvba: ok, what are would you prefer ?
<rvba> roaksoax: you mean that the 'api' directory will end up in /usr/share/javascripts/yui/<ver>/api ?
<roaksoax> rvba: yes
<rvba> roaksoax: sounds good to me.
<rvba> czajkowski: I think we (the red squad and the server team) should commit to answering to one question every day.  But we should pick the questions.  For instance, for "my" question, I'll have to poke someone from the server team.
<roaksoax> rvba: cool then
<rvba> czajkowski: note that appreciate the fact that you're trying to get the questions answered.  We should probably help more with that â¦ and I agree we probably need to be "pushed" a little ;).
<rvba> frankban: time for another review? https://code.launchpad.net/~rvb/maas/use-reverse/+merge/111240
<czajkowski> rvba: fair enoughm you can find someone to answer it, but at present it's been sitting there
<frankban> sure rvba
<rvba> ta
<czajkowski> I spoke to Daviey and said I'd ask as a filter to spread the questions to people
<rvba> czajkowski: I guess my whole point is that round robin is probably not the appropriate filter then.  Even if it's a filter, I give you that :).  Deployment questions involving images, boot sequence etc can be better handled by the server team.  Questions related to how to do manual stuff with nodes, how to customize things (other than images & stuff) are best for the red squad.
<rvba> czajkowski: does that make sense?
<czajkowski> rvba: right, but again, if you can't answer it, ask someone else to answer, it. At present all the questions are not being answered or are falling to one person.
<czajkowski> rvba: I can roughly work out which one goes to a person, but am trying to be fair here as well so one person doesnt get all the easy ones and not one person gets lumped with them all either.
<rvba> czajkowski: fair enough.
<cheez0r> Daviey: are you around?
<Daviey> cheez0r: today, i am mostly asquare.. but at least i am here.
<cheez0r> Cool- mind if I PM you for a sec?
<Daviey> rvba / czajkowski : I entirely agree that having a single marshal is better than 'best effort'.. Round-robin until czajkowski fully appreciates where best to push particulars, makes most sense.
<Daviey> ... really, just having someone on the hook to make sure the issues is resolved is key.
<Daviey> cheez0r: sure
<Daviey> smoser / roaksoax: would you be able to help out cheez0r?
<cheez0r> They've been providing assistance on occasion so far
<Daviey> He has dual nic in the bootstrap node..
<Daviey> Sounds like zookeeper isn't coming up
<Daviey> ... and no ssh key sent to the node... could be contacting metadata service issue
<cheez0r> I'm actually about to go through decommissioning and recommissioning all of my nodes except the MaaS node
<cheez0r> let me work through that and then try a fresh juju bootstrap and see where we get
<Daviey> super
<cheez0r> I'll also do the SSH key workaround- I published it to http://askubuntu.com/questions/147714/invalid-ssh-key-error-in-juju-when-using-it-with-maas
<smoser> cheez0r, after you've insalled a node, and can ssh into it, then i'd like to see what we can figure out
<smoser> i'm fairly certain that the node is not seeing the metadata service
<Daviey> ugh
<Daviey> I agree with smoser.
<cheez0r> well part of my issue today is time; I've only got about an hour and a half before I have to take off.
<cheez0r> anyhow let me get started rebuilding the MaaS, back in a bit.
<cheez0r> Okay, all of the nodes are in the process of commissioning. Here's an overview of my current environment: MaaS Node is dual-homed, eth0 is dom0 network, 192.168.1.1/24, eth1 is public, IP masquerade is configured for outbound connections from dom0 network, maas-dhcp is installed and both it and the maas package have been dpkg-reconfigured.
<cheez0r> Steps done so far: 11 nodes added to MaaS WebUI GUI and rebooted. User has SSH public key populated in both ~/.ssh/id_rsa.pub and the MaaS WebUI.
<cheez0r> Unusual configuration aspects: IP masquerade; dnsmasq is configured for static IP mappings for all 11 nodes, and /etc/hosts on MaaS node has entries for static IP address to hostname mapping for all 11 nodes that match their DNS Name in Cobbler/MaaS.
<cheez0r> One more note: This is HP BladeServer hardware that doesn't support WOL very well, so I've been manually booting the nodes.
<cheez0r> All 11 nodes are done with commissioning and are now off.
<smoser> cheez0r, the nodes got through the commissioning without issue or help?
<cheez0r> they're all in Status: Ready right now.
<cheez0r> My next step would be to juju bootstrap and then manually power on the node it picks to host the juju system.
<cheez0r> (and yes, got through commissioning with nothing other than adding them to MaaS and power-cycling the box.)
<Daviey> so.. that suggests MD is working.
<cheez0r> should I proceed?
<Daviey> yeah
<cheez0r> bootstrap completed, I see the node allocated in MaaS.
<Daviey> smoser: ^
<Daviey> so, lets see how the install goes.
<cheez0r> server is now powering up; takes 5-8 minutes, damn HP hardware
<Daviey> heh
<smoser> bootstrap completed.
<smoser> meaning juju says so?
<cheez0r> yes.
<cheez0r> meaning juju can contact maas and allocate a node.
<cheez0r> so now once the machine is booted up, I should be able to run juju status and get a status response from juju.
<cheez0r> It just started the PXE boot.
<cheez0r> okay, the machine is now booted to a login screen.
<cheez0r> ran juju -vvv status; accepted the SSH key warning, and now the server is refusing to accept the client.
<cheez0r> oh, wait.
<cheez0r> it actually worked!
<cheez0r> SWEET!
<cheez0r> so now I've deployed mysql and just booted the node maas picked for it.
<Daviey> \o/
<cheez0r> Daviey: it was your hostname questions that pointed me in the right direction
<cheez0r> the hostname of the MaaS node was resolving to the public IP.
<burnbrighter> When api keys are correctly installed in environments.yaml, and you run in to:
<burnbrighter> Bootstrap aborted because file storage is not writable: The supplied storage credentials were not accepted by the server
<cheez0r> I fixed that and it seems to have fixed my SSH key propagation issue.
<burnbrighter> any ideas?
<Daviey> cheez0r: thought it might be something like that
<Daviey> super
<cheez0r> and on that note I'm out of time for today; we'll have to pick up tomorrow sometime. Thanks for the help Daviey and smoser!
<Daviey> cheez0r: speak soon!
<burnbrighter> cheez0r: I'm looking at your solution here: http://askubuntu.com/questions/147714/invalid-ssh-key-error-in-juju-when-using-it-with-maas
<burnbrighter> what needs to be done after the maas.preseed is updated?
<cheez0r> burnbrighter: re-commission the nodes and boot them.
<cheez0r> When they come up, the ssh key should work.
<cheez0r> err, the password should work.
<cheez0r> also read the note below the answer- the cause of my lack of SSH key propagation seems to have been a hostname issue with the MaaS node.
#maas 2012-06-21
<burnbrighter> cheez0r: thanks.  I started over.  We'll see how it goes this time.
<burnbrighter> cheez0r: ut?
<burnbrighter> bigjools: how exactly should the /etc/hosts file look for the primary maas node?  There seems to be some confusion around this.
<bigjools> you don't need one, cobbler updates your dns
<burnbrighter> I see two entries - 127.0.0.1 for localhost - the usual
<burnbrighter> then I see 127.0.1.1 for my host's name
<burnbrighter> (maas01)
<burnbrighter> cheez0r experienced ssh problems in juju because of his dual-homed set up - which is what I have
<burnbrighter> and I continue to see problems with commissioning
<burnbrighter> sometimes it works, sometimes it doesn't
<burnbrighter> and, cobbler updates internal dns, how about external dns?
<burnbrighter> like when you need to update packages and such or maas-import-isos?
<burnbrighter> my external dns doesn't seem to work
<bigjools> it updates the dns server you configured on it, which is probably your local dnsmasq
<burnbrighter> if I try to update resolvconf/resolv.conf/base or something else, that mucks with the required 127.0.0.1 entry
<bigjools> yes, resolv.conf defers to dnsmasq locally
<burnbrighter> anything in resolvconf overwrites resolv.conf
<burnbrighter> and if you add something to base in resolvconf, it messes up the external dns
<burnbrighter> so how can you get around that?
<burnbrighter> am I missing some basic configuration?
<burnbrighter> or are these known issues
<bigjools> you don't need to change this stuff
<bigjools> if the box is multi homed then make dnsmasq serve on all addresses
<burnbrighter> do you have hints on how I would go about doing this?
<bigjools> I don't know without looking but I'd start with "man dnsmasq" :)
<burnbrighter> isn't a dual-homed system for the maas server one of the common configurations, out of curiosity?
<burnbrighter> I mean, how are people deploying them mostly
<bigjools> most people will use bind for real deployments - dnsmasq is not proven in larger environments
<bigjools> and in fact we are removing support for dnsmasq in the next revision of maas
<burnbrighter> interesting, ok
<cheez0r> burnbrighter: the trick is to ensure that you have two hosts configured- one for your eth0 ip and one for your eth1 ip; to have whichever of those two networks is your MaaS network's hostname in /etc/hostname; and to ensure that you've installed maas, maas-dhcp and ensured that either during the install or with dpkg-reconfigure you've specified the IP data of the MaaS network.
<cheez0r> I find that building the initial MaaS install by specifying the public or 'outside' IP network and DNS servers that are reachable, and then adding the MaaS network to the machine, setting the hostnames up, then dpkg-reconfigure maas, and apt-get install maas-dhcp to be the easiest procedure.
<roaksoax> rvba: yui is shipping some non free files right?
<rvba> roaksoax: off-hand, I've got no idea.
<roaksoax> rvba: what about these two? rm maas-$(VER).orig/src/maasserver/static/jslibs/yui/3.4.1/build/uploader/assets/uploader.swf rm maas-$(VER).orig/src/maasserver/static/jslibs/yui/3.4.1/build/io-xdr/io.swf
<rvba> roaksoax: note that in the version we currently use (3.5.1), uploader.swf is in ./src/maasserver/static/jslibs/yui/uploader-deprecated/assets
<roaksoax> rvba: http://lists.alioth.debian.org/pipermail/pkg-javascript-devel/2012-June/003761.html
<roaksoax> Daviey: ^^
<rvba> roaksoax: io.swf is a tiny flash component required (I think) to circumvent some of the limitations of the browsers.
<burnbrighter> cheez0r: any chance I could see what your host file looks like?  And did you have to do any dnsmasq tweaking?  Did you have external dns resolution issues there?
<cheez0r> burnbrighter: http://pastebin.ubuntu.com/1052955
<cheez0r> THe /etc/dnsmasq.conf file is populated from a cobbler template at /etc/cobbler/dnsmasq.template so that's where those dhcp-host lines are located, and then when you maas-import-isos it moves that file into place at /etc/dnsmasq.conf and does service dnsmasq restart for you.
<burnbrighter> thanks - what did you do with the 127.0.1.1 address maas created?  did you get right of that?
<cheez0r> It never created such an address for me.
<cheez0r> when you do the initial install you configure the public side network (outside)
<cheez0r> then you add the internal network and reconfigure maas, then install the maas-dhcp package
<cheez0r> 127.0.0.1 is a loopback address and should remain; I'm unclear where the 127.0.1.1 address is originating from.
<burnbrighter> that was confusing for me because the hostname actually points to that address
<cheez0r> shouldn't.
<burnbrighter> for example, my host name is maas01, so that was pointed to 127.0.1.1
<cheez0r> weird. 127.0.0.1 should point to localhost
<cheez0r> your hostname should point to the maas network IP address
<cheez0r> and a second host should be configured for the public IP side
<burnbrighter> and everything just worked for you?
<cheez0r> yeah
<burnbrighter> how about external dns resolution problems?
<burnbrighter> sorry, I haven't read everything you wrote above - too many meetings :)
<cheez0r> none- if you do the initial install with the public IP configured, you can reference your real DNS servers
<cheez0r> that should eliminate any issues once you install maas-dhcp and that installs dnsmasq
<burnbrighter> hmmm.  I'm seeing dnsmasq or resolvconf overwriting dns
<burnbrighter> I mean resolv.conf
<burnbrighter> it always gets written to 127.0.0.1
<cheez0r> yeah you have to configure dns-nameserver to /etc/network/interfaces
<cheez0r> so on your public interface you configure dns-nameservers and then service network restart and it should populate /etc/resolv.conf appropriately.
<burnbrighter> and you set each one separately there?
<cheez0r> one line, multiple servers separated with spaces
<cheez0r> gotta run- good luck, I'll respond in ~1hr when I get back
<burnbrighter> ahhh...resolv.conf is populated based on the interfaces file???
<burnbrighter> ok, thanks for the info
<cheez0r> burnbrighter: with resolvconf they are, and resolvconf is on the MaaS node by default.
<burnbrighter> I don't think I see any entries in resolvconf from maas
<cheez0r> MaaS has nothing to do with resolvconf.
<burnbrighter> exactly
<cheez0r> when you configure the node during the install of MaaS, you supply DNS server IP addresses.
<cheez0r> It inserts them into the /etc/network/interfaces file, from which the resolv.conf is generated.
<burnbrighter> ? I don't it does
<burnbrighter> s/don't/don't think/
<cheez0r> During the install of the MaaS node it prompts you for DNS servers.
<cheez0r> You must've typed in 127.0.1.1.
<burnbrighter> k, I don't remember that part
<burnbrighter> no, I definitely do not
<burnbrighter> I never put in 127.0.0.1
<burnbrighter> that's populated by maas
<cheez0r> 127.0.0.1 is a loopback address. It is present on most unix/linux systems.
<burnbrighter> which version are you using.  and yes I know that
<cheez0r> I'm installing from the Ubuntu 12.04 CD.
<burnbrighter> are you using the regular apt-get version?
<burnbrighter> yeah - I don't recall it asking about DNS, and I know for certain I've never put in the loop back address
<cheez0r> No, you install the MaaS head node by booting it from the Ubuntu ISO and choosing to install MaaS, and then having it install Ubuntu for you.
<burnbrighter> correct
<cheez0r> If you're doing it a different way I've got no experience with it and therefore can't help you.
<burnbrighter> the only two ways I know how to do it are the one you mention, the other is installing base 12.04, then adding maas after the fact
<burnbrighter> in any case, back to our conversation above - are you saying what's in /etc/network/interfaces gets written to /etc/resolv.conf by the OS?
<cheez0r> No, by the resolvconf package.
<burnbrighter> ok, and you can set it separately for each interface?
<cheez0r> yes, but generally you dont.
<cheez0r> You only set it for the interface that can see that DNS server, and then it resolves there for the entire box.
<burnbrighter> my problem is I either get internally resolvable dns OR external dns, but never both together
<cheez0r> what does that statement mean?
<cheez0r> What is an 'internally resolvable dns'?
<burnbrighter> the internally resolvable addresses for dns created by maas-dhcp
<burnbrighter> or cobbler
<burnbrighter> not sure which
<cheez0r> You are creating a MaaS network. You are installing the MaaS node. It should have 'external' DNS servers that the MaaS node resolves against. The MaaS node should resolve hosts for the rest of the MaaS cluster.
<burnbrighter> let me explain a little...
<burnbrighter> if I do a ping against node-0050c06467.maas.local and my resolv.conf has nameserver 127.0.0.1, it works
<burnbrighter> if I do a ping against www.cnn.com with nameserver set to same, it doesn't
<cheez0r> I don't know enough about dnsmasq to help with that issue.
<cheez0r> If you follow the steps I gave, it will work. Beyond that I don't know.
<burnbrighter> if I change resolv.conf to external dns, or even my router which passes dns ie. 192.168.1.1, then pinging www.cnn.com works, but node-0050c06467.maas.local doesn't
<burnbrighter> bigjools was suggesting tweaking the dnsmasq config, but also informed me they are dropping support in the next release anyways
<cheez0r> My solution to that was to statically map my DNS entries in /etc/hosts and to statically assign DHCP entries in /etc/dnsmasq.conf
<cheez0r> you saw that configuration.
<burnbrighter> yeah, I'm going to try it.  I've been trying to get this right for two weeks and I'm becoming a bit frazzled
<cheez0r> I hear ya, I've been banging on mine for about a month now
<burnbrighter> I'm trying to do this in a VM too
<burnbrighter> with fully virtualized network
<burnbrighter> well, not fully
<burnbrighter> bridging the set up adds its own set of complications
<burnbrighter> eventually I will need to add a third network in for a third party piece of software the works with openstack
<burnbrighter> that needs to communicate over a bridged dmz interface.  been having problems with bleed over in bridging
#maas 2012-06-22
<jtv> rvba: I posted some notes on your MP.  Could you have a look and see if I'm making any sense at all?
<rvba> jtv: sure.
<roaksoax> negronjl: how was your presentation btw?
<negronjl> roaksoax: ping
<negronjl> or Daviey: ping
<roaksoax> negronjl: here
<negronjl> roaksoax: I'm having an issue with the mirror and was wondering if you can help me
<roaksoax> negronjl: shoot
<negronjl> roaksoax: I re-created the mirror for maas and i am now unable to boot machines ( incorrect architecture )
<negronjl> roaksoax: I re-created the mirrror with the ( already there ) create-mirror.sh
<negronjl> roaksoax: I have downloaded the mirrors for both i386 and amd64 ... any thoughts ?
<negronjl> roaksoax: just to be on the safe side ... I am now trying to boot without the mirror ( downloading over the wire )
<negronjl> roaksoax: that seems to be working ... so there is something broken with my mirror
<roaksoax> negronjl: uhmmm no idea
<roaksoax> negronjl: smoser will be able to help you he was the one who created it
<negronjl> roaksoax: no worries .. I'll debug further ( or as smoser as well ) ... just thought that maybe you had encounter this in the past
<smoser> oh yeah, i can always help.
<smoser> what is the issue?
<negronjl> hi smoser ... I re-created the mirror for the maas deployment and it is now complaining about not having the right architecture
<negronjl> smoser:  i am using apt-mirror ( actually the create-mirror.sh script in the maas server )
<smoser> what is complaining about not having the right arch?
<smoser> negronjl, that was probably a bad idea
<roaksoax> negronjl: we did hit it i think but smoser fixed it
<negronjl> smoser: which part was a bad idea ?
<smoser> i think recreating the mirror.
<negronjl> smoser: i had to
<smoser> heres what sucks
<smoser>  * we had a tiny link
<smoser>  * we needed a local mirror
<smoser>  * we tried to be smart and only get some portions of the full mirror
<smoser>  * that sucked.
<smoser> negronjl, basically, the issue is that you need i386, and that create-mirror wasn't getting it.
<negronjl> smoser:  i mirrored both i386 and amd64
<smoser> the best thing to do, if you have a nice big fat link, is to make that create-mirror script get a lot more stuff.
<smoser> well, then what are you missing?
<negronjl> smoser:  I am still using the create-mirror script .. I just modified the line for the installer ( it had deb-i386 and i added the amd64 one )
<negronjl> smoser:  if you have an apt-mirror config file that would get it all ( both archs to be sure ), it would help
<negronjl> smoser: i have 35M downlink so I can handle the download
<negronjl> smoser: once I have the entire mirror, updating it would be easier and we would have everything that we need.
<smoser> so what are you missing?
<smoser> what was complaining?
<negronjl> smoser:  it appears to be downloading/using the i386 deb installer and, apparently that's not to the liking of the HP microserver
<negronjl> smoser: so I downloaded amd64
<negronjl> smoser:  that doesn't work either ...
<negronjl> smoser:  I haven't tried deleting the i386 stuff and just downloading amd64 ... didn't want to do that if I didn't really have to
<negronjl> smoser:  thoughts ?
<smoser> negronjl, i'm a bit confused.
<smoser> what is using the i386 deb installer?
<smoser> sorry for not being up to date on what you're doing or the problems your having
<negronjl> smoser:  no worries ... let me grab a chunk of syslog ( where the complains are ) and I'll pastebin them so we have a better idea ... give me a minute.
<smoser> k
<negronjl> smoser: can't pastebinit 'cause the machine is not booting :) ... the error: unsupported archive
<negronjl> smoser:  it is trying to load the i386 deb installer
<negronjl> smoser: and rejecting it ( wrong arch )
<negronjl> smoser: This all works if I download it from the wire so, I may just have to let it install and see what it's using after when I can log on to the box
<smoser> negronjl, but hwat is "wrong arch" ?
<smoser> i need more data there.
<negronjl> smoser:  i know ... me too :P  I'm also trying to find out ...  kind of a dumb error though ... wrong arch ...but it doesn't really tell me what it is expecting or what the _right_ arch is :/
<smoser> what are you trying to do right now?
<smoser> enlistement?
<smoser> and how much of our hacked setup are you still using?
<smoser> because there is probably data there i dont remember.
<smoser> s/data/hacks/
<negronjl> smoser:  I am trying to enlist ( I guess it is what is called when you first boot the machine so it shows as "Declared" )
<smoser> right.
<negronjl> smoser:  I am using all of the setup ( 9 HP micro servers and one buster laptop )
<negronjl> s/buster/busted/
<smoser> so that should probably work. i dont knwo why it would show wrong arch.
<smoser> by default enlistment *is* with i386
<smoser> so that is expected
<smoser> (the reason is that i386 boots on amd64, and you dont know what the system is, so you take least common denominator)
<smoser> once enlisted it should be recognized as amd64
<negronjl> smoser:  ok .. I'll try to get more information about it.  At least I know that it "should" be the i386 one ...
<negronjl> smoser: i got past the wrong arch issue but, now the iscsi is trying the wrong address ( an address that I had previously used ).  Can you tell me how to have that changed to the correct address ?  Or where the that setting would be ?
<smoser> you might want to dpkg-reconfdigure maas
<smoser> i think that might do it.
<smoser> the IP addresses are rendered into /var/lib/tftp (i think) ... grep down that path.
<smoser> but they start with the setting of the 'server' that you should be able to set with dpkg-reconfigure
<smoser> you might have to 'maas-import-isos --update'
<smoser> but i dont think so
<negronjl> smoser: sorry ... missed your reply ... thx.  I'll try that
#maas 2012-06-23
<burnbrighter> Nodes stuck in commissioning.  Can someone tell me if this indicates the time difference problem:
<burnbrighter> [Fri Jun 22 16:02:43 2012] [error] Expired timestamp: given 1340380970 and now 1340406163 has a greater difference than threshold 300
<burnbrighter> [Fri Jun 22 16:02:49 2012] [error] Content-Type: text/html; charset=utf-8
<burnbrighter> [Fri Jun 22 16:02:49 2012] [error] WWW-Authenticate: OAuth realm="OAuth"
<burnbrighter> weird my clocks both seem to be in UTC and in sync.  So why am I seeing this?
<burnbrighter> ok, clock off by a whole 24 hours.  didn't see that part
#maas 2013-06-17
<AskUbuntu> how to use maas with no network | http://askubuntu.com/q/309132
<rvba> roaksoax: Hi Andres.  The most recent maas package failed to pass our integration tests: the nodes are not enlisting at all.  Did you test your preseed-related changes on a real package or could this change be responsible for the breakage?
<roaksoax> rvba: hi Raphael! yeah i tested them on hw and it worked just fine. let me investigate. is this latest in saucy?
<rvba> roaksoax: the latest package in the daily ppa
<roaksoax> rvba: ok. ill test agaisbt whats in saucy, which include my preseed changes
<roaksoax> rvba: ill let you know how it goes
<rvba> roaksoax: ta.  We've changed a few other thingsâ¦ but nothing serious, mostly saucy-specific test fixes.
<roaksoax> ok
<roaksoax> yesh cause i have a running cluster here at home and everything was working as expected
<roaksoax> rvba: ok, so preseeds render just fine
<roaksoax> rvba: so it boots into the instance and gets stuck there it seems
<rvba> roaksoax: yeah, that's what I saw in the lab, no obvious error but the node seems stuck.
<roaksoax> rvba: so it is either the instances,or maybe even cloud init
<roaksoax> idk
<roaksoax> i'll need to ask scott
<roaksoax> smoser: ping
<roaksoax> rvba: it seems to access the metadata just fine: http://pastebin.ubuntu.com/5774038/
<rvba> Yep, I saw that.
<roaksoax> rvba: cloud-init seems to be failing
<roaksoax> rvba: http://paste.ubuntu.com/5774042/
<rvba> roaksoax: good catch
<roaksoax> rvba: ok I think I know why that is, but I need smoser's input
<roaksoax> rvba: quick thing. Is there a way to tell python to "insert a space in each new line of a string?"
<roaksoax> rvba: got it
<roaksoax> rvba: this would fix it: http://paste.ubuntu.com/5774113/
<roaksoax> no it won't
<roaksoax> rvba: this would: http://paste.ubuntu.com/5774117/
<roaksoax> thoughts?
<rvba> roaksoax: this would need a serious explanation :) because it's a bit ugly.
<roaksoax> rvba: i know :(. Ok so basically, the enlistment preseed requires 3 spaces before any line of code, while the user_data template doesn't require
<roaksoax> rvba: so when it renders in user_data it is just fine
<roaksoax> rvba: i'll show you in a bit
<roaksoax> rvba: this is the reason: http://pastebin.ubuntu.com/5774132/
<rvba> roaksoax: the logic that adds indentation would be better placed in the template itselfâ¦ maybe allenap will have an opinion about this as he is our tempita grand-master :)
<roaksoax> allenap: ^^
<roaksoax> :)
<allenap> roaksoax: Hi, my name's Clippy. I see you're trying to write a structured configuration file with a free-form template language. Let me help you with that ;)
<roaksoax> allenap: hi clippy :)
<roaksoax> allenap: so the issue is this for enlistment: http://pastebin.ubuntu.com/5774132/
<allenap> roaksoax: Yeah. The file is YAML, right?
<roaksoax> yeah
<roaksoax> allenap: so cloud-init expects that it has spaces before the start of the line. The ugly fix is like: http://paste.ubuntu.com/5774151/
<allenap> roaksoax: Yeah, that's exceedingly ugly :) I'll try to think of something better.
<roaksoax> allenap: awesome! thanks for looking at it :)
<rvba> roaksoax: allenap: A less ugly way would be to provide a tempita tag to do the indendation: something like {{maas_ipmi_autodetect_py|index:4}}
<rvba> s/index/indent/
<roaksoax> rvba: yeah that would be awesome
<allenap> Sorry, otp.
<allenap> rvba: Yeah, agreed.
<allenap> It's still ugly, buy much less so :)
<allenap> s/buy/but
<smoser> roaksoax, here now
<smoser> "cloud-init expects that it has spaces before the start of the line"
<smoser> is really "cloud-init reads yaml", yaml format uses indentation.
<mgz> you can always just spell it as json if you don't like significant whitespace
<smoser> this is true.
<smoser> but i don think json soles the problem here.
<allenap> mgz: The problem in this example is that it's trying to interpolate one string into the middle of another string. I think a proper solution would be to manipulate the string outside of a template environment that doesn't grok YAML.
<smoser> the core issue is that therre are 2 things cooperating on the production of an output (that happens to be yaml formatted). and the first thing is unknowledgable about this.
<smoser> (and best for it to be left unknowledgable actually)
<smoser> i think i'm saying the same thing allenap said...
<allenap> Yeah, I think we're in agreement here :)
<allenap> For _now_, I think adding a tempita tag (really a function) to indent is the best way to approach this.
<allenap> roaksoax: I think http://pastebin.ubuntu.com/5774286/ will work.
<roaksoax> allenap: ok cool thanks
<roaksoax> smoser: sorry cleaning lady was clceaning my office
<smoser> allenap, yeah, i think i think the same thing.
<roaksoax> rvba: https://code.launchpad.net/~andreserl/maas/enlist_userdata_ipmi/+merge/169836
<roaksoax> smoser: how does cloud-init loads yaml?
<roaksoax> yaml.load?
<roaksoax> rvba: what would you assert against?
<smoser> yaml.safe_load()
<roaksoax> smoser: so I would simply assert that it loads successfully for a test right?
<smoser> yeah, i' mguesing it was failing load before (as invalid yaml)
<roaksoax> smoser: yeah, so I'm gonna add a unittest that checks that the rendered preseed can be successfully loaded as yaml
<roaksoax> so i'm wondering what to compare against
<roaksoax> rvba: this seems to work: http://paste.ubuntu.com/5774562/
<roaksoax> allenap: ^^
#maas 2013-06-18
<rvba> roaksoax: Hi!  Just a heads up: the fix you landed yesterday in MAAS solved the enlistment problem in the lab.  Thanks!
<roaksoax> rvba: cool thanks!!
<b4sher> Is there a way in MAAS to modify boot options like 'noapic' and 'nomodeset'?
<salma__> I have problem adding nodes to the server. commission fails.
<salma__> hi
<salma__> Node commissioning failes, can anyone help!
#maas 2013-06-19
<ross_cav> hello to the MaaS crew.
<ross_cav> I'm just trying to find out the best place to ask some MaaS related questions. I've got a VM environment to test it out, but behind a proxy. So, seems to be having some issues setting up the nodes when they try to connect to archive.ubuntu.com. Not sure where exactly toe ask.
<bigjools> hey rvba, can you help --^ :)
<rvba> ross_cav: here is a good place to ask questions (you can also use http://askubuntu.com/questions).
<ross_cav> rvba: Cheers, I just setup an account there, but thought I'd double check. I'll go there to see what I can find out.
<AskUbuntu> Ubuntu 12.04 MaaS and JUJU's proxy issues | http://askubuntu.com/q/310153
#maas 2013-06-22
<AskUbuntu> MaaS minimum requirements with juju-jitsu? | http://askubuntu.com/q/311410
<AskUbuntu> Juju + openstack. Bootstrap is successful, But cant create any other services | http://askubuntu.com/q/311583
#maas 2014-06-16
<AskUbuntu> Ubuntu 14.04 LTS server + MAAS WEB gui wont import boot image | http://askubuntu.com/q/483883
<rvba> bigjools: we've used a FakeRequest in a couple of placesâ¦
<rvba> :)
<rvba> bigjools: right, you need to shove whatever field you need in the FakeRequest.  See for instance how we've set the 'user' field in the FakeRequest in src/maasserver/tests/test_api_support.py.
<bigjools> rvba: eurgh
<bigjools> rvba: it's easier to create a real request surely
<rvba> bigjools: we've used the RequestFactory in a couple of places as well (it's coming back to me now)
<rvba> bigjools: ugly I know, but please have a look at src/maasserver/tests/test_middleware.py:fake_request
<bigjools> rvba: je vais regarder
<bigjools> rvba: maybe I should put that util in the testing dir
<rvba> bigjools: yeah, looks like we've added many different ways of creating a test request.  All over the place.
<bigjools> rvba: I had noticed :(
<bigjools> rvba: I'll put it in the factory
<rvba> Sounds good.
<bigjools> FWIW those getCamelCaseThings really jar with the make_things!
<bigjools> rvba: that's not right that Messages uses a class property as storage in that thing, is it?
<rvba> bigjools: why?
<rvba> bigjools: request._message it is an instance of Messageâ¦ what's wrong with that?
<bigjools> rvba: the fake Message has self.messages as a class property
<bigjools> seems a bit weird
<bigjools> works, anyway
<bigjools> rvba: https://code.launchpad.net/~julian-edwards/maas/staticip-api-exceptions/+merge/223192 :)
<rvba> bigjools: I'll review it (still otp) soon
<bigjools> rvba: thank you.  can you or jtv please review https://code.launchpad.net/~julian-edwards/maas/allocate-ip-on-start-2/+merge/222747 as well please, I am totally blocked on that now
<rvba> bigjools: sure.  One of us (me, allenap or jtv) will take care of it today.
<bigjools> rvba: a thousand thank yous
<jtv> Or "toosind tak" as they say in Danish.
<jtv> (Which happens to be their standard way of saying "thanks a lot")
<bigjools> rvba: "Why using the hostname here and the system_id in the API error?"
<bigjools> Why do you think? :)
<rvba> bigjools: sorry, I meant "Why are you using the system_id in the API and the hostname in the UI?"
<bigjools> rvba: same response :
<bigjools> :)
<rvba> heh
<bigjools> IOW, which would you, as a consumer of each, prefer to see?
<rvba> bigjools: I tend to favor using the system id, because it's guaranteed to be constant.
<rvba> But I can see why the hostname makes sense in the UI.
<bigjools> so you see why I used the system id in the api and the hostname in the web ui? :)
<bigjools> the system id appears nowhere in the web ui
<rvba> Well, it's in the URI of course.
<rvba> I just wish we had a pattern for these kind of things.
<bigjools> the url to the page is irrelevant in the context of a good web ui
<rvba> The system id should at least be displayed on the node's page.
<rvba> But anyway, that was a detail.
<rvba> allenap should be in the process of reviewing your other branch.
<allenap> rvba: I am in the process of doing that.
<rvba> allenap: cool ;)
<bigjools> allenap: much grassy ass.  I shall be heading to bed now.
<allenap> bigjools: nn
<allenap> Nearly done.
<bigjools> be gentle with me
<galebba> Happy Monday, newbie here.Sorry to repeat the question i asked here yesterday.Where should i look for logs that shows ipmi powering on/off nodes ? I have setup maas and trying to add the first node. Booted the node from the dvd and pointed to maas server. Node get powered off as expected but doesnt show up under the nodes in the gui.
<bigjools> allenap: just caught the review before actually going to bed.  Thanks :)
<bigjools> galebba: /var/log/maas/celery.log
<galebba> Thanks. but it does not show any indication of an issue. This is all i have  INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[fe6beb91-8dfa-4eee-9f7b-bfa9c9ed01bf] succeeded in 0.0749380588531s: Non
<galebba> and nothing in maas.log and pserv.log
<bigjools> make sure there's power parameters defined in the edit node page
 * bigjools off now, bye
<rbasak> galebba: the power control pieces are templated scripts. I forget where they are exactly, but one option to debug is for you to wrap them or amend them for debugging.
<galebba> so i got the nodes to power on.off using the ipmitool with the follwoing,  ipmitool -H 172.18.112.136 -U admin -P password -I lanplus  -y 0000000000000000000000000000000000000000 chassis power status
<galebba> how do i insert a crypto key in to Maas ipmi templates as it seems the use of the keys is not in maas
<galebba> btw this is maas 1.4 on 12.04 LTS and Cisco UCS C series server
<blake_r> allenap: ping
<alexpilotti> blake_r: ping
<blake_r> allenap: hey
<blake_r> alexpilotti: hey
<alexpilotti> blake_r: do we have a meeting today? :-)
<blake_r> alexpilotti: yes
<alexpilotti> blake_r: trying to join on the hangout but no luck
<blake_r> alexpilotti: sorry running late
<allenap> Hey blake_r, whatâs up?
<blake_r> allenap: how does rpc handle long running operatings
<blake_r> allenap: will it block, or is it async
<allenap> blake_r: It wonât block any other RPC calls. If you use it from Twisted-land then it wonât block, but the helpers that are in (canât remember offhand) will block the running thread if you need blocking.
<blake_r> allenap: from maasserver side will it block
<allenap> blake_r: Weâre also planning on having âthreadsâ (not OS threads) that can be initiated and queried, so things like boot and install and can started by a Django thread and revisited later on.
<blake_r> allenap: i got a branch for diskless booting
<allenap> blake_r: From Django, yes, theyâll block, but all calls go via Twistedâs event-loop, and thereâs one of those running in every Django process on the region.
<blake_r> allenap: i want to use rcp to create the disk on the cluster
<allenap> blake_r: Cool.
<blake_r> allenap: so it will take a while
<blake_r> allenap: so if i make the call, it will wait for the response?
<blake_r> allenap: i don't want to block the webrequest, but i want to wait to execute the poweron
<allenap> blake_r: You could have a response which says âIâve started, hereâs my uuid if you want to check on my laterâ.
<allenap> s/my/me/
<blake_r> allenap: rpc returns a uuid?
<allenap> Thatâs our proposal for the robustness work.
<allenap> blake_r: Nope, but it could do. Do you want a call about this later?
<blake_r> allenap: sure
<allenap> blake_r: In 3h okay?
<blake_r> allenap: 3h?
<allenap> I have another call then dinner and kids into bed stuff to do between now and then.
<allenap> blake_r: 3 hours.
<blake_r> allenap: oh okay, thats fine
<blake_r> allenap: i send you an invite
<allenap> blake_r: Ta.
<skinder_> where's a good place to inquire as to problems with maas? i've setup a cluster controller that, when hosts in the subnet pxe boot and attempt to start the provisioning process, i get an error about / not being found
<skinder_> i'm a bit stumped
<MilesDenver> Yea, I think this is a good place skinder
<MilesDenver> At the moment, I'm working on understanding the actual steps to build a system.  The language seems off and my system never seems to stay powered on.
<MilesDenver> Documentation doesn't have steps.
<MilesDenver> skinder_: have you modified your preseed files?
<skinder_> nope, a very basic unmolested install. the region controller is able to do its thing, and provision hosts in the subnet it's in, but when i added the second cluster controller, hosts in its subnet don't complete the pxe boot process.
<MilesDenver> both the first and the second are in the same subnet?
<skinder_> no, is that necessary?
<blake_r> skinder_: can the second cluster controller, reach the region controller
<skinder_> yes
<MilesDenver> no... it won't work actually.  MaaS is pretty good at making sure it's the only DHCP in the logical network
<blake_r> skinder_: it needs to be able to reach the region, and so do the nodes!
<blake_r> skinder_: do the nodes, have a route to the region as well?
<skinder_> yes
<skinder_> L3 connectivity is good
<blake_r> skinder_: what is the error you are getting on the nodes?
<skinder_> blake_r: one moment, i'll get you the exact error
<skinder_> MilesDenver: well the two cluster controllers are the only dhcp servers in their respective broadcast domains, so that shouldn't be a problem i don't think
<skinder_> and they're in different subnets, no dhcp forwarding is happening either
<MilesDenver> Keep talking through the issue.  I'm not the expert Blake_r is, but i'll help where I can
<skinder_> blake_r: "The disk drive for / is not ready yet or not present. Continue to wait, or Press S to skip or M for manual recovery."
<skinder_> that is displayed directly after the "Net device info" and "Route info" tables are displayed
<blake_r> skinder_: has it already installed? or should it be installing?
<blake_r> skinder_: on the cluster, do "sudo service tgt restart"
<blake_r> skinder_: also you do have the boot images imported for that cluster?
<skinder_> blake_r: yes, boot images are installed
<skinder_> blake_r: and no, it hasn't already installed
<blake_r> skinder_: did it say the anything about ISCSI? trying to connect?
<blake_r> skinder_: are you using fast-path installer?
<skinder_> blake_r: yes, i have seen iscsi messages zip by
<skinder_> tgt is restarted, will try again
<blake_r> skinder_: so thats fast-path
<blake_r> skinder_: yeah let me know if that works, if not
<blake_r> skinder_: got something else for you to try that will help you debug
<skinder_> blake_r: sweet, thanks. i will update momentarily.
<skinder_> blake_r: still gets stuck at the same spot, same error
<MilesDenver> I have a sad question - Why doesn't my new node stay running?  It displays it's hostname, but shuts down in less than a minute.
<MilesDenver> With a simple web interface, you can add, commission, update and recycle your servers at will
<MilesDenver> Does Commission mean set the Operating System on it and leave it running?
<skinder_> i thought it was commission -> install
<skinder_> oh wait derp that's probably not what you were asking. i've seen commissioned nodes stay powered on.
<MilesDenver> skinder_ thanks
<MilesDenver> I'm testing with a system that boots glacially slow.
<skinder_> fun fun
<MilesDenver> seems like "Start" is the same as "Accepting" a node
<MilesDenver> START a) Finds DHCP server b) Receives PXE details c) Boots temp image d) Reports to MaaS server e) Node shuts down
<MilesDenver> COMMISSION must therefore load an OS and stay running.
<skinder_> like you MilesDenver, i wish there were more details about the entire process. what happens during the commissioning, install, etc.
<skinder_> makes it hard to troubleshoot without that
<skinder_> especially when using a java remote kvm console in a linux desktop :P
<MilesDenver> skinder_: so, we hang out here and help each other.  That works.
<MilesDenver> hahaha
<MilesDenver> Commissioning booted with the proper OS and name... ran for about a minute and powered off.
<jhobbs> MilesDenver: "start" is load an os and stay running
<jhobbs> MilesDenver: "commission" is more or less "get a node ready to install to"
<MilesDenver> jhobbs: cool.  I'll try that.  Just deleted my node thinking I had it in some strange state
<MilesDenver> Not really sure why the published doc is missing that detail.
<blake_r> skinder_: same issue?
<skinder_> blake_r: yes
<blake_r> skinder_: is the node, that is trying to boot, already in MAAS, liek shows up in the WebUI
<skinder_> blake_r: no
<blake_r> skinder_: okay, is it amd64?
<blake_r> skinder_: actually that doesn't matter
<blake_r> skinder_: do this on the cluster
<blake_r> skinder_: "tftp 127.0.0.1"
<blake_r> skinder_: "get /pxelinux.cfg/default"
<blake_r> skinder_: please pastebin the output
<blake_r> skinder_: "cat default"
<skinder_> blake_r: it'll be posted shortly
<skinder_> blake_r: http://pastebin.com/rHcdHNaB
<blake_r> skinder_: what version of MAAS are you runing?
<skinder_> blake_r: 1.5.4
<skinder_> blake_r: ubuntu 12.04.5
<blake_r> skinder_: newest version uses trusty, for enlistment and commisioning, this is showing precise
<blake_r> skinder_: even on precise should boot trusty for enlistment
<skinder_> blake_r: kk will give that a go
<blake_r> skinder_: you are running an older version
<blake_r> skinder_: even the kernel paths are old
<skinder_> blake_r: apparently one of my coworkers upgraded from 1.3 to 1.5
<blake_r> skinder_: then you need to run, maas-import-pxe-files again!
<skinder_> blake_r: did, but i'll do so again
<skinder_> blake_r: so apparently trusty wasn't listed in the /etc/maas/import_pxe_files sources, i added that, going to re-run the pxe file import
<blake_r> skinder_: okay, let me know
<skinder_> blake_r: same problem. even after the pxe image update, it's still pointing to the same precise images in the pxe cfg, not trusty
<blake_r> skinder_: "sudo service maas-pserv restart"
<blake_r> skinder_: try again
<skinder_> blake_r: already did that
<skinder_> i'm honestly just considering nuking these servers, installing a fresh copy of 14.04 and maas, and going from there
<MilesDenver> That's what I did
<blake_r> skinder_: your going to have better luck with that
<blake_r> skinder_: but it should work
<blake_r> skinder_: apt-cache policy maas-cluster-controller
<skinder_> http://pastebin.com/7kUBHVjW
<blake_r> skinder_: your using 1.4
<blake_r> skinder_: not 1.5.3
<blake_r> skinder_: if your region is not the same version, it will not work
<blake_r> skinder_: that makes since, as 1.4 is the highest on precise
<skinder_> well, maas --version on the region controller says 1.5.4, and the package is named the same as on the cluster controller
<skinder_> curious
<blake_r> skinder_: well that is very strange
<MilesDenver> Still seeing the behavior where after commissioning the node builds. runs for 90 seconds and shuts down.
<MilesDenver> If I disable DHCP, it can boot and stay running.... so something it's getting registered back to the MaaS server to let it know the system build is complete.
<MilesDenver> But I cannot log in to see the Node I just built, because it has not IP address
<MilesDenver> anyone know how to debug a system that stays in a cycle of building.  Meaning after a successful node build, it shuts down and if you start it again is rebuilds... and shuts down
<AskUbuntu> Openstack Neutron - Cannot Access Tenant Router Gateway | http://askubuntu.com/q/484293
#maas 2014-06-17
<jtv> A few welcome confirmations from Ante.
<bigjools> rvba: svp: https://code.launchpad.net/~julian-edwards/maas/bulk_action_error_catch/+merge/223345
<rvba> bigjools: on it
<bigjools> rvba: merci
<rvba> bigjools: potage
<rvba> bigjools: jtv approved it 14 minutes ago.
<jtv> Hence my "too late" in the call...
<bigjools> rvba: so much for the long polling
<rvba> gmb: the thing is, everytime you push a new revision, you get a new *version* of the same MP
<gmb> ISWYM now
<gmb> rvba: Then sure, please do.
<rvba> gmb: if you go to https://code.launchpad.net/~gmb/maas/populate-mac-address-ngi/+merge/223209 and *then* click on the "r2427 to r2422" you'll see all my comments.
<gmb> rvba: Cool, thanks.
<rvba> (all 6 of them)
<bigjools> that's like Shiteveld.
<jtv> allenap, rvba: I put your faces on the cards for the things you've been investigating.
<rvba> k
<jtv> They're "investigate" cards, so this doesn't mean that I'm promising the world that you'll be doing the coding, today.  :)
<rvba> heh
<allenap> jtv: Ta.
<MilesDenver> Facing a problem where my new Nodes don't stay running, instead they stop, and if powered on again will reload the OS?  Any guidance on approaches to troubleshoot?
<blake_r> MilesDenver: That means that installation is failing
<blake_r> MilesDenver: you need to check to output of the install log
<rvba> gmb: I just saw your branch go byâ¦ did you consider adding the validation inside src/maasserver/forms.py:validate_nonoverlapping_networks instead of inside the model code?
<gmb> rvba: Gaaah, I thought about it and then, after doing the *other* bits of validation, forgot to come back to it and just autopiloted the code onto the model. Iâll move it.
<blake_r> jtv: fixed the x509 ssl key branch
<blake_r> jtv: now there is two
<blake_r> jtv: one for the migration and model https://code.launchpad.net/~blake-rouse/maas/x509-ssl-key-migration/+merge/223428
<jtv> OK I'll have another look.
<blake_r> jtv: one for the api wiring https://code.launchpad.net/~blake-rouse/maas/x509-ssl-key/+merge/223429
<blake_r> jtv: thanks
<jtv> So these supersede the old branch?
<jtv> blake_r: one branch done
<blake_r> jtv: thanks
<MilesDenver> blake_r: About those install logs -- I cannot log in because the system shuts down.   If I kill the DHCP server, it boots without an IP, but none of the user's have passwords?
<MilesDenver> blake_r: I'm just rebuilding my MaaS.
<bmorriso> This page isn't accurate https://wiki.ubuntu.com/ServerTeam/MAAS/AvahiBoot yet the link exists in today's error message "You can boot this node using Avahi-enabled boot media or an adequately configured dhcp server. Seehttps://wiki.ubuntu.com/ServerTeam/MAAS/AvahiBootÂ for instructions."
<MilesDenver> Argh, rebuilt MaaS server and the same problem.
<MilesDenver> Is there an easy way to add a root password in the preseed?  Not being able to login because it shuts down is limiting
<MilesDenver> NVM - I created an amd64_generic_trusty.hostname that points to a special preseed_notmaster and made the change in not_master
#maas 2014-06-18
<mwhudson> bigjools: if i say the words "curtin" and "arm64" or "armhf" in the same sentence does that provoke any thoughts?
<bigjools> scary ones
<mwhudson> heh
<bigjools> I don't know if it's ever been tried on those arches
<bigjools> arm IO mean
<bigjools> gah can't type today
<mwhudson> the thing that's lurking in my mind is the need (on current platforms) to run flash-kernel
<bigjools> if you can find an image, give it a go :)
<mwhudson> what sort of images does curtin usually install?  cloud images?
<bigjools> they're big tarballs IIRC
<mwhudson> bigjools: if i say the words "curtin" and "documentation" in the same sentence...
<bigjools> wheeeeeee
<mwhudson> http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/examples/basic.yaml is just installing a clout root.tar.gz
<mwhudson> *cloud
<mwhudson> i didn't think those had kernels in though
<bigjools> the kernels are downloaded separately
<mwhudson> oh!
<mwhudson> bigjools: do you know where it gets them from?
<bigjools> mwhudson: http://maas.ubuntu.com/images
<bigjools> http://maas.ubuntu.com/images/ephemeral-v2/releases/streams/v1/com.ubuntu.maas:v2:download.json
<mwhudson> ah ok
<mwhudson> bigjools: do you know where in curtin (which 'stage') the kernel gets handled?
<bigjools> I don't, sorry
<bigjools> it's a bit of a black hole to me, and as you note there's no docs
<mwhudson> i bet smoser doesn't have anything else to do this cycle
<mwhudson> one way appears to be more or less "chroot /target apt-get install linux-image"
<mwhudson> which ought to work fine x-platform
<bigjools> yeah
 * bigjools brb
<mwhudson> actually i think that's more or less what it always does, not sure it uses the simplestreams data
<mwhudson> oh hahaha
<mwhudson>     machine = platform.machine()
<mwhudson>     if machine.startswith('armv7'):
<mwhudson>         update_initramfs(target)
<mwhudson>     else:
<mwhudson>         setup_grub(cfg, target)
<mwhudson> yeah so about that...
<mwhudson> not sure why the update_initramfs part is necessary, that's done by installing the kernel pacakge
<mwhudson> unless triggers are disabled or some such insanity
<jhobbs> does the kernel package get installed on a curtin install?
<jhobbs> it's not part of the image that gets unpacked?
<mwhudson> jhobbs: that's the way i read it
<mwhudson> http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/curtin/commands/curthooks.py#L139
<jhobbs> i added that line; iirc the initramfs wasn't getting run until that setup_grub() ran
<jhobbs> which didn't work on armhf, so i just did update_initramfs
<mwhudson> huh
<mwhudson> maybe there is some conditionality then; the kernel package is only installed if there is no kernel in the installed image or something
<mwhudson> but i don't see that
<mwhudson> now
<mwhudson> can i remember why i was interested in this question?
<mwhudson> jhobbs: so the way maas uses curtin is that it netboots a cloud image and uses cloud init data to run curtin?
<mwhudson> how does the cloud init data get to the machine?
<jhobbs> mwhudson: wget from maas more or less
<mwhudson> ah so the cloud-init user-data is "get more config from this url"?
<mwhudson> ah yes
<mwhudson> ah i think you can put something like ds=nocloud-net[s=$url] on the kernel command line
<mwhudson> and then cloud-init will hit $url/user-data and $url/meta-data
<jtv> bigjools: We don't really support multiple IPs per NIC for cluster interfaces, but I figured we could either say TFB, or treat them as multiple interfaces.
<jtv> Depending on whether any actual need arises.
<bigjools> jtv: treat them as aliased separate interfaces
<bigjools> I did think about it at the time
<jtv> I guess we could only manage one anyway.
<jtv> Doesn't make much sense to run two dhcpds (for the same IP version) on the same NIC but for different networks.
<jtv> (Not counting VLANs where you've got, from our perspective, different NICs anyway)
<bigjools> no, you'd run a single dhcpd for all interfaces like we do now
<jtv> Same thing applies regardless â dhcp daemon or dhcp service, you can't just serve two unrelated dynamic IP ranges on the same NIC without some way to configure which client should go in which range.
<jtv> We neither have nor, to my knowledge, need such configuration.  :)
<bigjools> cheap karma: https://code.launchpad.net/~julian-edwards/maas/ui-fixes/+merge/223506
<bigjools> jtv: right, the whole thing is somewhat nonsensical
<jtv> Cheap karma?  Coming!
<rvba> bigjools: haha, I see the wrong capitalization came from Django generating the labels based on the name of the fields. :)
<bigjools> rvba: corrrrrect
<bigjools> rvba: I am learning too much about Django lately.
<bigjools> *twitch*
<rvba> bigjools: too late, I'm anointing you "APAC Django expert attachÃ© to the MAAS team" ;)
 * bigjools sets up mail forwarder to rvba for anything with "django" into
<bigjools> s/into/in it/
 * bigjools having a bad typing day
<jtv> Recapitalising words is one of those things frameworks really oughtn't be doing for us.
<jtv> bigjools, did you run into one of those "Ip address" fields?
<bigjools> jtv: I bumped my nose on it
<jtv> Exercise: auto-capitalise MAISON DU THE for display as a title.
<jtv> "Oh, âtheâ is an article, let's lower-case that."
<jtv> (I believe it should become thÃ©)
<jtv> (Sorry, ThÃ©)
<rvba> jtv: shouldn't it by "MAISON DU THÃ" in the first place?
<jtv> In principle, yes â but aren't all-caps names like this often displayed without the accents?
<rvba> Very often indeed.  But it doesn't make it right.
<jtv> But it's easy to think nobody will care, when actually some piece of software may come along and make assumptions about how your capitalisation rules work.
<jtv> And don't get me started on how the Scots are going to fill out fields labeled "Mac"...
<jtv> "Ach, I entered âGregorâ but it doesnae work!â
<rvba> jtv: care to review a tiny branch? https://code.launchpad.net/~rvb/maas/fix-manpage-generation/+merge/223511
<jtv> OK
<rvba> Ta
<rvba> jtv: I also want to get started on doing some actual coding on IPv6.  Could we have a call (it will be short) to coordinate our work on this?
<jtv> rvba: sure â there's some low-hanging fruit.
<bigjools> rvba, jtv: please make sure you leave something obvious for me to do
<bigjools> since I won't be able to pre-imp unless you volunteer to stay up late :)
<jtv> bigjools: there'll be stuff left...  any questions about the cards on the board?
<rvba> bigjools: did you talk with Andres about the additional work he wanted to get done on the DHCP stuff?
<bigjools> jtv: I've not looked yet
<bigjools> rvba: sort of
<jtv> bigjools: they all have detailed descriptions... if you see something unclear about one, it's probably going to be unclear about all of them, so let me know.
<bigjools> jtv: ok ta
<bigjools> jtv: I guess the first thing is to try and get ipv6 working on my test network
<jtv> There are also a bunch of known jobs that you can do without though.
<bigjools> jtv: why a feature flag?
<jtv> bigjools: because there are places where it doesn't look as if we can do both IPv4 and IPv6 â we'll have to choose.
<bigjools> jtv: example?
<jtv> Which do we tell dhcpd to serve?
<bigjools> isn't that automatic depending on what format IP addresses are entered in the cluster interface edit form?
<jtv> Which cluster addresses do we discover?
<bigjools> all of them, like now
<jtv> We don't do that now.
<jtv> We only discover the IPv4 ones.
<bigjools> that's all of them then, barring the ones we are about to add
<jtv> ...
<bigjools> we already discover all ipv4, we just add discovery of ipv6 interfaces.
<bigjools> and if you set them to be managed, we tell dhcpd to serve on there
<jtv> Yes, we can do that one dynamically.
<jtv> We'll need some validation changes to make it work that way of course.
<jtv> The problem is that right now we have a bunch of places that will quietly give wrong answers if we mix IPv4 and IPv6.  A feature flag buys us the freedom to fix those without breaking normal operation.
 * jtv steps out for about 2 minutes
<bigjools> jtv: I say just do it right up front
<bigjools> a feature flag is just procrastinating
<jtv> bigjools: I think we can do it without a feature flag, if we can continue to run celery and such as IPv4, assuming the DNS stuff works out.
<AskUbuntu> MaaS region controller with 2 Cluster | http://askubuntu.com/q/484951
<bigjools> jtv: I think that would be OK from what I read
<jtv> gmb: just spotted one of my pet antipatterns in update_mac_cluster_interfaces.  When you find yourself writing a search loop, extract it.  Don't do what you need to do inside the loop body and then decide that you're only going to do it once, so you can return.
<jtv> "Find item.  If found, process item."
<jtv> Not: "Loop over items.  If item is what I want, process item.  Break out of loop as an optimisation."
<Solution-X> anyone here do much with MAAS? I have a fresh install of 14.04 installed as MAAS controller that is being cranky and refuses to load the images. Installed OS, booted up, apt-upgrade, reboot, create MAAS user, login, click download images. Also tried "sudo maas-import-pxe-files" after as a backup and that completes but does not result in the webpage recognizing the images' existence. Also checked
<Solution-X> celery.log and it doesnt spit any errors/warns.
<jtv> Solution-X: and the boot images do not show up in the UI?
<jtv> It should normally be the one or the other â you get the images, or there's going to be an error in that log...
<jtv> (Be aware that the download is huge...)
<Solution-X> correct
<Solution-X> which is whats confusing me
<jtv> In the celery.log, do you see the import task at least starting and finishing?  Or only starting, or neither?
<Solution-X> give me a moment and ill drop a paste in...there may be something im simply not seeing
<jtv> OK.  I'll have to leave soon, but would like a look.
<Solution-X> just started and saw the start task and iftop is showing traffic
<jtv> Then that sounds as if it's probably still downloading.  Have a look in /var/lib/maas/boot-resources/cache; you should see the files growing.
<Solution-X> that is my impression as well
<jtv> You should see a snapshot directory in /var/lib/maas/boot-resources, too.
<Solution-X> im trying to more or less tail the process to you without shoving all the logs down your throat
<Solution-X> cache  snapshot-20140618-092857  snapshot-20140618-095234  snapshot-20140618-100031  snapshot-20140618-100657  snapshot-20140618-100932  snapshot-20140618-115549
<jtv> Looks like you're running a bunch of downloads at the same time, or maybe started a few that were aborted.
<Solution-X> probably aborted
<jtv> Hope so, because if they're still running they may slow you down.
<Solution-X> iftop traffic has subsided, load is 0 https://privatepaste.com/a98fcd3f02
<Solution-X> bad formatting but that link is celery
<jtv> It goes through phases; it may be unpacking an image file.
<jtv> Yup, your imports are probably still running.
<jtv> If you're running maas 1.5 (the default for 14.04), make sure you only have one instance of the import script running.  If the downloads are too much, review /etc/maas/bootresources.yaml.  (Finicky syntax, so be careful).
<jtv> That file is no longer there in 1.6, but in 1.5 it'll help you restrict what gets downloaded.
<jtv> If the script was going through an unpacking phase just now, watch for changes in the latest snapshot directory.
<jtv> Gotta run now!
 * jtv runs
<Solution-X> looks like one of the previous tasks was causing the subsequent imports to fail...maybe i click import when it was already attempting an import when i first set it up...cleared the snapshots and ran another import, this time it was successful
<Solution-X> thanks again for the help and advice!
<jtv> Clearing the snapshots shouldn't make much of a difference, but they can pile up a bit.  They won't use much space though, because it's all links into the cache dir.
<MilesDenver> Is there a way to turn off "maas maas nodes accept-all"
<wrale_> is maas 1.3.1 the latest packaged for ubuntu 12.04.4?
<wrale_> will i still be able to use juju with this version?
<wrale_> is there a simple way of installing the latest version of maas on precise?
<MilesDenver> Back working on my system that never Commissions.  I can now look at the install logs, but I'm not seeing the problem.
<MilesDenver> The only problems I'm find are possibly not problems
<MilesDenver> Could not open device at /dev/ipmi0, so Unable to start ipmievd during installation.
<MilesDenver> and ci-info: no authorized ssh keys fingerprints found for user ubuntu.
<MilesDenver> Anyone have troubleshooting suggestions?
<MilesDenver> My cloud-init.log does show an awful lot of "Failed at attempted import of 'modulename'" -- http://pastebin.com/1daAfrTp
<MilesDenver> I suspect I have this problem which is possibly hardware related - https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1321885
<ubot5> Ubuntu bug 1321885 in openipmi (Ubuntu) "IMPI detection and automatic setting fail in ubuntu 14.4 maas" [Undecided,Confirmed]
<MilesDenver> I'm going to manually build and curse the time I've wasted
<jhobbs> MilesDenver: can you test running this script on your server
<jhobbs> http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/view/head:/etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect.py
<jhobbs> if you could run that and post the output to the bug that would be good
<MilesDenver> hmmm. ok
<jhobbs> you'll have to isntall freeipmi-utils first
<jhobbs> and run it as root
<jhobbs> that's the script that does ipmi config/detection
<jhobbs> unfortunately it's hard to get good output from it from the logs
<MilesDenver> Where does it output?
<jhobbs> it just outputs to console
<jhobbs> when you run it yourself
<MilesDenver> hmmm, it only output "maas,WOzMJ8pfeG,172.26.0.186,LAN_2_0"
<jhobbs> well, that's a good sign
<jhobbs> can you use those credentials to access the system?
<MilesDenver> oh... with some kind of IPMI tool.
<jhobbs> ah yeah
<jhobbs> from a remote system, you could use ipmitool - ipmitool -H 172.26.0.186 -U maas -P WOzMJ8pfeG power status
<jhobbs> can you also post the output of ipmi-locate ?
<MilesDenver> thanks - ipmi-locate http://pastebin.com/Zjtt8s80
<MilesDenver> and the ipmitool responds "Error: Unable to establish LAN session Unable to get Chassis Power Status" which likely means a firewall problem
<jhobbs> MilesDenver: yeah it looks like detection and configuration is working
<jhobbs> MilesDenver: can you ping that host from the maas server?
<MilesDenver> I'm supposed to be building an OpenStack Compute Node on this system.  Tomorrow I need to build it manually
<jhobbs> ah
<jhobbs> out of time for debug then?
<MilesDenver> jhobbs well, I can probably debug a bit longer
<jhobbs> err, can you try this command ipmitool -H  172.26.0.186 -U maas -P WOzMJ8pfeG power status
<jhobbs> oops
<jhobbs> ipmitool -Ilanplus -H  172.26.0.186 -U maas -P WOzMJ8pfeG power status
<MilesDenver> jhobbs: sure.... long wait (probably my firewall) then
<MilesDenver> "Error: Unable to establish IPMI v2 / RMCP+ session Unable to get Chassis Power Status"
<jhobbs> k
<jhobbs> yeah i agree - sounds like a network issue
<lifeless> or a wedged BMC
<MilesDenver> lifeless: it's network.  I've never asked for the ports to be opened.
<lifeless> MilesDenver: oops;)
<MilesDenver> Seems like there is beer downstairs.  I'll come back to this tomorrow, but we have an all hands meeting in 10 minutes.
<MilesDenver> Actually, I'll probalby end up on the patio with my laptop in about an hour
<jhobbs> sounds rough
<MilesDenver> wish you were here
#maas 2014-06-19
<AskUbuntu> Install only maas-region-controller on base Ubuntu 14.04lts base server | http://askubuntu.com/q/485398
<jtv> allenap, gmb, rvba: if we don't actually need to provision through IPv6, we can simplify the immediate task a bit more.  (I've been removing tasks.)
<gmb> Yay.
<jtv> It does mean figuring out how we might deal with dual-stack DHCP.  This may help, although much of the work will be on our side rather than in dhcpd: http://tools.ietf.org/rfc/rfc4477.txt
<rvba> jtv: not sure we should remove the tasks, maybe just re-prioritize.  The exact scope of the IPv6 work is still a bit unclear.
<jtv> Yes.  But I've removed the tasks for things like "investigate Rabbit on IPv6" because it's OK to keep them on IPv4.
<jtv> bigjools: we can just map a node's hostname along with its static IP.
<rvba> bigjools: I still don't understand how you got a static IP in the DNS config, unless you got lucky and a completely unrelated action triggered the rewrite of the zone file.  I would expect a test like http://paste.ubuntu.com/7668820/ to pass.
<rvba> bigjools: I'll see if I can understand why it worksâ¦ (upgrading my test server to utopic nowâ¦)
<blake_r> allenap: why no database access in twisted?
<allenap> blake_r: Blocking IO. It just needs to be deferred to a thread, so thereâs no problem, it just shouldnât happen in the reactor thread.
<blake_r> allenap: so if I need to modify the database while in a twisted deferred? what would I do
<allenap> blake_r: return deferToThread(my_database_function)
<blake_r> allenap: okay
<allenap> blake_r: In the diff on the mp, check_query_eventloops is the database function.
<blake_r> allenap: okay i see
<breeze441> Hey Guys, i am setting up MAAS for the first time, how can you pass the DNS name server info to the maas managed nodes in the case not using maas as a dns but just dhcp ?
<MilesDenver> Breeze441: I'm guessing you could use the preseed to do this.
<MilesDenver> I'd like to change the setting "maas maas nodes accept-all" back to NOT accepting all?
<stepheno_> hey all, quick question about the auto-discovery/enlistment process.
<stepheno_> I'm able to pxe boot machines which have not been manually defined in the nodes interface, but they do not receive an ip address after they receive their pxe boot image(during pxe boot)
<stepheno_> looking at the dhcpd.conf, it seems like only dhcp requests with the vendor identifer 'PXE' will be answered, so how does the freshly pxe-booted but not yet registered machine receive an ip?
<Ratabuntu> I just got it working this morning, one node up one the region/cluster, PXE boots
<Ratabuntu> MAC address?
#maas 2014-06-20
<rvba> bigjools: with your fix in https://code.launchpad.net/~julian-edwards/maas/utopic-syslinux-location/+merge/223851 we should finally have a working Utopic package.  (The other blocker got resolved yesterday.)
<rvba> bigjools: can you confirm that you saw a *static IP* in the forward zone file?
<rvba> bigjools: I'm testing the new static DHCP stuff and I'm confused: my commissioning node gets a static IP.
<bigjools> rvba: thanks for landing the fix. Yes I saw a static IP for a CNAME.
<bigjools> rvba: static IP for commisioning is either a bug or a feature depending on your PoV
<bigjools> :)
<rvba> bigjools: well, looks like the lander is down (I just redeployed the lander now) so it's not landed yet.
<rvba> bigjools: Yeah, I figured out the CNAME thing eventually: even static mappings end up in the leases fileâ¦ so they are parsed and stored in the DBâ¦ eventually.
<rvba> bigjools: oh, it just got landed
<bigjools> yeah static leases in there is a feature too
<bigjools> not that worried about those
<bigjools> in fact it drives the DNS at the moment :)
<rvba> bigjools: well, have a look at my email, I found myself in a weird situation were I had conflicting IP addresses (an allocated node had a *dynamic* IP).
<bigjools> rvba: I skimmed it, it's too late and I am too tired to take it all in
<rvba> bigjools: sure, no worries.  It's just good to be able to QA thisâ¦
<bigjools> rvba: yes, thank you for doing some QA
<rvba> welcome
<rvba> Starting a CI run nowâ¦
<allenap> jtv: Youâve got some conflicts in https://code.launchpad.net/~jtv/maas/dual-stack-network-helpers/+merge/223869
<allenap> jtv: Iâll review that after lunch if it hasnât been taken by then.
<rvba> Reviewer needed! https://code.launchpad.net/~rvb/maas/fix-bootloader-path/+merge/223920
<rvba> allenap: gmb: Anyone up for a tiny review? ^
<gmb> rvba: Sure
<rvba> Ta
<wrale_> on 1.5.1+bzr2269-0ubuntu0.1: i'm running supermicro nodes.  i manually typed in the IPMI information which I know to be correct via ipmiview (SM's tool).  node says ready and has IPMI v2.0 config, yet start but is greyed out.. is this normal?
<jtv> wrale_: if Start is greyed out in the UI, there should be a tooltip to tell you the reason.
<jtv> For me it's usually "need to register an SSH key or you won't be able to access the node."
<jtv> That reminds me â blake_r, jhobbs: we inhibit the Start node action in the UI if the user has no ssh keys registered... Does that need to be smarter and look at SSL and/or SSH keys depending on the chosen OS?
<blake_r> jtv: The node should fail to start if missing sslkeys and windows
<blake_r> jtv: meaning not allowed*
<blake_r> jtv: quick question
<blake_r> jtv: would it be okay to do this
<blake_r> jtv: http://paste.ubuntu.com/7674825/
<jtv> Looking...
<jtv> But note that "fail to start" is different from "inhibit."  If you don't register an SSH key, and then you browse to a node's page, the Start button will be greyed out and a tooltip will say that you need to register an SSH key.
<jtv> blake_r: it's ugly but in the context I think it makes sense.
<jtv> (Your paste, that is)
<blake_r> jtv: i understand, having a tooltip that says register sslkey would be good
<blake_r> jtv: tahnks
<jtv> ISTM if you want to start a Windows node, you can use either type of key; if you want a regular node, you need an SSH one.
<roaksoax> hey guys, if we were to add a way for users to add custom power methods, is that soemthing we would want to do?
<roaksoax> jtv: ^^
<jtv> AIUI, no.
<roaksoax> rvba: ^^
<blake_r> roaksoax: yes
<roaksoax> like "Add power method"
<jtv> We want them all in the codebase, so we can change the API, right?
<blake_r> roaksoax: look at my plugin architecture
<roaksoax> blake_r: branch?
<blake_r> roaksoax: doc
<roaksoax> blake_r: doc?
<roaksoax> :)
<roaksoax> jtv: right, so if we were to do a plugin architecture such as the one blake_r designed in the dcoument. Why would we not want to do that?
<roaksoax> jtv: what are the benefits of having it so
<jtv> Orders.  :)
<roaksoax> jtv: what are the disadvantages?
<roaksoax> jtv: just exploring cause someone on canonical was requesting whether we can do that
<jtv> Right.  My recollection is that we want to keep all power methods in a single codebase, to keep development unified.
<gmb> rvba, jtv, allenap: Can haz review: https://code.launchpad.net/~gmb/maas/clean-ip-ranges-ipv6/+merge/223936
<jtv> I'll take it.
<gmb> Thankee squire
<rvba> roaksoax: Like jtv said, I think we want to keep all the code in MAAS itself if possible.  But the plugin architecture allows external packages to contribute power method so it's technically possible.
<wrale_> jtv: brilliant.. thanks!
<jtv> wrale_: was it the SSH key?  :-)
<allenap> roaksoax: Mark wants it all in MAAS, so that we manage QA, via CI and the like. Having a hook point for adding new power methods is, IMO, about helping the user to help herself; if we make it easy for someone to make a new power method, thatâs a happy user. However, they may be unhappy when we change the API and their code breaks...
<wrale_> indeed!
<jtv> :)
<wrale_> strange UI feature
<allenap> roaksoax: I think we should have a hook, but not something that can be added via the UI. As in, only a developer could make it. That way thereâs a greater chance that it can be upstreamed to us.
<allenap> roaksoax: And, iirc, thatâs what blake_râs doc suggests.
<wrale_> i'd rather fail fast and find i can't login than greying out this box
<wrale_> hmm maybe that's not fail fast.. either way :)
<jtv> Well theoretically, being able to see the notice _before_ you click the box should be faster.
<wrale_> ^ yup
<jtv> The question is whether the tooltip is obvious enough.
<jtv> Maybe we should just show that message under the button.
<jpds> jtv: You around?
<jtv> Hi jpds
<jpds> jtv: I'm having issues with a fresh install of MAAS and gmb said that you might be able to help.
<jtv> ?
 * gmb also named rvba, just to spread the misery
<rvba> heh
<jpds> Oh, sorry.
<gmb> Also allenap, first mate of HMS MAAS
<gmb> Basically anyone whoâs not me, thenâ¦
<jpds> I'm getting this error: http://paste.ubuntu.com/7674927/
<jtv> Good to know who our friends are
<jtv> jpds: I think that means there's a problem with the credentials string.
<jtv> The oauth module probably just happens to validate the "consumer" part of the string first, finds a fatal problem there, and says "invalid consumer" instead of "this is a recipe for mussels Ã  la mode, not a valid credentials string."
<gmb> jtv: This is on the first PXE boot of a node, which is whatâs confusing me, and why I refered tit to you guys. :)
<jtv> Ooo
<allenap> jtv: I believe we added the is_recipe_for_mussels_a_la_mode() validator a while back.
<jtv> Then might it be the old clock-skew problem?
<gmb> D'oh!
<gmb> Didnât think of that.
<jtv> jpds: I thought we'd fixed it, but it might be something like "node's hardware clock is wildly out of sync with the server's clock."  It upsets oauth.
<jtv> Maybe you're not getting NTP?
 * jpds returns.
<jpds> jtv: Well, ntp's on the box. Offset: -0.030.
<jpds> Oh, I got a machine in MAAS.
<jpds> jtv: It's still happening.
<gmb> allenap: When you say âIt might be worth allocating from the fc00::/7 range too; http://www.simpledns.com/private-ipv6.aspx has an explanation why.â do you mean âinstead of just from entirely random address spaceâ?
<allenap> gmb: Yeah.
<gmb> Ok
<allenap> gmb: Iâve got an extremely little bit of bad news for you. I really like the way youâve fixed getRandomIPv6Address(), but thereâs a bug: randint() can return either endpoint, one of which is network.size, and network[network.size] results in IndexError. On average itâll only happen once every 2658455991569831745807614120560689152 calls though, so it
<allenap> really is a *very* small bug.
<wrale_> i wish #juju were as responsive as #maas
<wrale_> how can i disable swap partition creation during the node install process (during maas provision)?  do i need to edit the preseed files?
<wrale_> swap is killing my installs to a 240GB ssd on a server with 256GB ram
<blake_r> wrale_: if you use fast-path installer
<blake_r> wrale_: no swap partition is created
<blake_r> wrale_: there is a button on the node view to enable it
<blake_r> wrale_: its normally enable by default
<wrale_> thanks blake_r .. i'll try this
<wrale_> blake_r: that worked.. thanks again.  any idea how i might get the juju status command to get its dns lookups from the maas running on the same machine?
<blake_r> wrale_: set your first dns as the maas ip
<wrale_> blake_r: cool.. will do.. thank you
<blake_r> wrale_: dont forget to set your upstream dns address on the maas settings page
<wrale_> hmm.. can't seem to find that.. will look for docs
<wrale_> oh.. nvm. .found it
<wrale_> is it normal for my secondary interfaces (enabled with dhcp and dns in maas) to remain unconfigured after booting a node (juju bootstrap) with the fast installer?
<AskUbuntu> Can Ping but Cannot SSH to Openstack VM Instace | http://askubuntu.com/q/486151
#maas 2014-06-21
<Ratabuntu> Is it wrong that I've set my upstream dns to bind9 on another cluster?
<Ratabuntu> wc
<Ratabuntu> wtf is on with the default irc client
<Ratabuntu> obviosly not wrong chan
<jtv> Ratabuntu: the cluster controller doesn't run bind...  but maas itself doesn't really care what dns you use.
<jtv> (The region controller runs bind)
<Ratabuntu> yeah, I figured I'd add maas-dns package, and it might have added the region-min as well. Anyway haven't had any problems with it yet. But I've ony provisioned a couple of nodes. So I am running a region controller with dhcp/dns and cluster controller with DNS, problably going to add a third I want to add two separate vlans to the config
#maas 2014-06-22
<AskUbuntu> What does "LDS" stand for in the openstack reference implementation? | http://askubuntu.com/q/486609
<AskUbuntu> what is maas exactly? | http://askubuntu.com/q/486701
<loki27_> Hi
<loki27_> I have installed 1 master MAAS node and deployed 4 nodes that are in "Ready state"
<loki27_> now what ?
<Ratabunti> Yeah, I think you'l probably want to depploy services using juju
<loki27_> That's what i fured out..
<loki27_> trying to do that
<Ratabunti> me too
<loki27_> are you deploying juju on the master maas ?
<loki27_> or standalone
<Ratabunti> well , I've deployed it on the 2 clusters controllers and a node
<Ratabunti> not sure if that's right, documentation goes a bit diverse at this point
<loki27_> heh it's not realy straight forward at a point
<loki27_> (same thing with mirantis anyway..)
<Ratabunti> I won't bother trying that then, I donwloaded it in case I spent too much time with maass/juju/openstack
<loki27_> Ha see... im moving on ubuntu because i've spent lots of time on mirantis ;)
<loki27_> It does work.. but is still complex
<Ratabunti> and landscape - not sure what that's all about, trying out the 30 day trial, but it seems to not sync up with the cc's or region controller
<loki27_> Have not tried this one
<loki27_> I liked mirantis because they should be vendor neutral
<loki27_> Ubuntu/MAAS should be also, but not sure about canonical
<Ratabunti> just adds them so you can monitor/update/reboot everything I can do from maas webfront and rrdtool
<Ratabunti> just thought I'd give it a go, but seems to be adding an extra layer of confusion
<loki27_> I have not found a simple deploy solution yet that can be considered for production use
<loki27_> And that may be a good thing.. one has to understand what's going on to deploy openstack, no matter what flavor is used
<Ratabunti> I tried that first, took a while, but was getting somewhere
<loki27_> It depends on what you are trying to deploy..
<Ratabunti> Well, I'm forcing everyone at home to stop using windows. I want virtual servers for myself. and would like to get that maximum from the hardware i have a my disposal
<Ratabunti> home cloud/nas/mining/remote control/snapshots
<loki27_> Well i think you bring LOTS of overhead if you need virtualisation..  :P
<loki27_> Unless you do it to learn openstack hehe
<loki27_> You could probably go with a solution like Proxmox.. not expensive, and you have the ability to build CEPH storage now
<loki27_> and deploy on all your nodes, with a low overhead
<Ratabunti> Well, aparaently not if you can forward your GPU's to the guest OS, and don't talk to me about proxmox ol
<Ratabunti> lol
<loki27_> really ?
<Ratabunti> Thtat was the forst attempt
<loki27_> i tested it,was great
<loki27_> would consider for home usage .. but did not want to go with it for commercial needs
<Ratabunti> Just could not get it to VT-d my pci-e cards
<loki27_> it does not have the multi-tenancy and scalability that comes with ostack
<loki27_> You will probably get the same issues with openstack ..
<loki27_> if you use KVM, it's more of a KVM issue then Proxmox or Openstack really .
<Ratabunti> probably but openstack is far more versitle
<Ratabunti> and scalable
<loki27_> true, you really need scalability for home ? :P
<Ratabunti> and long term, it's probably a better skill to have
<Ratabunti> who says, it'll remain at home, one of the ccs is moving soon
<loki27_> http://www.hostingpics.net/viewer.php?id=205476201405281707192.jpg
<Ratabunti> nice
<loki27_> HA virtualisation lab :P
<Ratabunti> I have two mining rigs with 12 GPU's altogether so far, ad
<Ratabunti> will be expanding soon.
<Ratabunti> dinner got to ge
<Ratabunti> nice talkng
<loki27_> ok bye
<DreamDemon> Anyone dealt with MAAS and hardware that is not the same?
<DreamDemon> ie: HPDL385 & Supermicro SuperServer?
<DreamDemon> anyone alive?
#maas 2015-06-15
<mup> Bug #1464866 changed: Unable to commission node which needs hwe kernel on trusty <maas-images:New> <https://launchpad.net/bugs/1464866>
<mup> Bug #1464984 changed: Crash in TFTP server booting NUC under UEFI <MAAS:New> <https://launchpad.net/bugs/1464984>
<mup> Bug #1464894 changed: Crash in TFTP server code serving UEFI boot loader <MAAS:Triaged> <https://launchpad.net/bugs/1464894>
 * Mmike grabs food
<mup> Bug #1465305 opened: Spurious test failure: UnknownRemoteError: Code<UNKNOWN>: exceptions.TypeError: query() argument after ** must be a mapping, not unicode <MAAS:Triaged> <https://launchpad.net/bugs/1465305>
<ctlaugh_> I am trying to setup MAAS 1.8.0 rc3 on an HP Moonshot.  When I click "Chassis" under the "Add Hardware" dropdown, choose "Moonshot Chassis Manager" for "Power type", and enter my host/username/password info, I get the following: http://paste.ubuntu.com/11719979/.  Any suggestions?
<kiko> ctlaugh_, you get that in the web UI?
<kiko> newell, ping? ^
<kiko> ctlaugh_, just for comparison, have you tested 1.7.5 and does it work?
<ctlaugh_> kiko, newell: yes -- it shows up in the UI
<kiko> roaksoax, potential 1.8 bug above
<kiko> let me get our moonshot qa team here
<newell> ctlaugh_: so just to make sure I understand what you are doing, you tried probe-and-enlist with 1.7.5 and 1.8.0 rc3 and the first one works and the later doesn't?
<kiko> newell, he's doing add hardware which AIUI is new in 1.8
<newell> kiko: okay never tried adding a chassis with that new capability yet, could be a new bug
<newell> If a bug is filed I don't mind taking a look at it
<kiko> newell, could you test meanwhile? ctlaugh_ can add a bug, but his experience is pretty simple
<kiko> ctlaugh_, which cartridge type?
<ctlaugh_> kiko: I haven't tried 1.7.5.  I just added the ppa:maas-maintainers/testing and grabbed the lastest out there.
<kiko> ctlaugh_, if you could try with 1.7.5 that would give us subsidy to stop the presses with 1.8 and get the bug fixed
<kiko> i.e. if it is a clear regression in 1.8
<ctlaugh_> newell: I haven't tried 1.7.5.  The latest I was running with before was some version of 1.7 that did NOT have the new Add Chassis UI.
<newell> kiko: yeah I will test
<ctlaugh_> kiko: I'm using a mix of m300 and m400... BUT, I haven't even gotten to the point of enlisting anything.  I was just trying to add the chassis.
<kiko> ctlaugh_, without the UI, you can still do probe and enlist through the CLI -- did you know that?
<ctlaugh_> kiko: I did know that, but I do it so infrequently that I can't remember how :)
<kiko> ctlaugh_, sure, but I guess my question you've answered -- with 1.7 you did it via the CLI and I presume it worked.
<ctlaugh_> kiko, newell: are there any additional logs that I could grab that would help?
<ctlaugh_> kiko: I did via the web UI and it worked.
<ctlaugh_> kiko: (in 1.7)
<newell> ctlaugh_: if you look at /var/log/maas/*.log files on the region and/or cluster controller machines that would help
<ctlaugh_> kiko: but that was before the "Add Chassis" functionality.  I just manually added the MACs of the nodes I wanted, and manually entered the MSCM IP and auth info and everything worked fine.
<kiko> ah
<kiko> ctlaugh_, so you never used the probe and enlist action
<ctlaugh_> kiko: narinder gupta did help me with a script to probe and enlist the moonshot chassis and nodes from CLI  and that worked with 1.7 too.
<kiko> newell, is it undocumented, maas cli probe and enlist? if not, could you get us a URL?
<kiko> ctlaugh_, ah, that script likely called probe and enlist
<newell> ctlaugh_: kiko: http://maas.ubuntu.com/docs/api.html#nodegroup look for probe_and_enlist_hardware in this section
<kiko> newell, that's not trivial to convert into a cli example though
<newell> kiko: you didn't ask for an example :P, one sec
<kiko> newell, "I hate our docs"
<ctlaugh_> newell, kiko: looks like, from the error JSON, it's trying to do the probe_and_enlist and that's what's failing.  The only thing I see in the logs related to this is a 400 entry in the apache2/access.log.
<ctlaugh_> newell, kiko: http://paste.ubuntu.com/11720179/
<newell> ctlaugh_: if you do this you should be able to probe and enlist from the cli
<newell> $ uuid=$(maas admin node-groups list | grep uuid | cut -d'"' -f4)
<newell> $ maas admin node-group probe-and-enlist-hardware $uuid model=mscm host=10.228.45.67 username=chassis_username password=chassis_password
<newell> where "admin" is the maas profile and the host ip address is supposed to be ip for the Chassis
<ctlaugh_> newell: I just ran the command.... returned with no visible output.
<newell> ctlaugh_: nothing in the logs? nothing showing up on the UI as being enlisted?
<ctlaugh_> newell: No change in the UI yet, no apparent log messages either
<newell> ctlaugh_: can you -->  tail -f /var/log/maas/*.log so that you can see what is happening when you issue that command?
<newell> ctlaugh_: you should definitely see something after issuing the probe and enlist command
<ctlaugh_> newell: regiond.log had a single line show up showing the HTTP POST coming in (with 400 HTTP response code)
<newell> ctlaugh_: can you paste the command you used?
<ctlaugh_> newell: http://paste.ubuntu.com/11720318/
<newell> ctlaugh_: don't know if you responded to me or not, I lost connection
<ctlaugh_> newell: Here's the command and regiond.log message: http://paste.ubuntu.com/11720318/
<newell> ctlaugh_: docs say 400 error is form parameters not being passed properly.  Could you try to use the deprecated cli command probe-and-enlist-mscm and then just delete the model=mscm portion?
<ctlaugh_> newell: That got a "Success" message from the CLI and an HTTP 200 in regiond.log.
<ctlaugh_> newell: Nodes are showing up in the web UI too
<newell> ctlaugh_: would it be possible for you to report a bug on this?  That would help.
<newell> https://bugs.launchpad.net/maas
<ctlaugh_> newell: Yes, no problem
<newell> ctlaugh_: thanks :)
<newell> ctlaugh_: go ahead and post the bug url here when you are done and I will assign myself to it
<kiko> newell, okay, so it's broken for real in 1.8?
<newell> kiko: I haven't tested it myself but when he uses the deprecated probe-and-enlist-mscm it works
<newell> while when he uses the probe-and-enlist-hardware it doesn't, so something is not working right
<kiko> I'm trying to get sfeole to join as I am curious how testing hasn't picked this up
<newell> ctlaugh_: could you do another test for me?  Could you try the original probe-and-enlist-hardware command that was failing but also pass the accept_all=true parameter at the end (in addition to what you had before)?
<ctlaugh_> newell: https://bugs.launchpad.net/maas/+bug/1465353
<ctlaugh_> newell: still failed (with accept_all=true)
<newell> ctlaugh_: thanks
<ctlaugh_> newell: no problem -- glad to help
<mup> Bug #1465353 opened: Unable to probe and enlist a Moonshot chassis in MAAS 1.8.0 (rc3+bzr4000) <MAAS:New for newell-jensen> <https://launchpad.net/bugs/1465353>
<kiko> ctlaugh_, thanks for testing! we'll get this nailed. thank narinder for the assistance if you run into him
<kiko> ctlaugh_, what are you using maas for?
<ctlaugh_> kiko: No specific purpose... just using to quickly provision clean nodes when needed for testing.  In the past, was using w/ Juju to deploy Openstack on the nodes.
<kiko> cool
<newell> ctlaugh_: you still around?
<ctlaugh_> newell: yes
<newell> ctlaugh_: could you test the probe-and-enlist-hardware again with one slight modification?
<ctlaugh_> yes
<newell> do same command but instead do model=mcsm
<ctlaugh_> newell: 'c' and 's' switched positions?
<newell> yeap
<newell> typo ;)
<ctlaugh_> newell: success....
<newell> cool
<newell> ctlaugh_: can you now test with the add hardware on the UI?
<ctlaugh_> newell: still an error (as expected?)... data: host=10.36.0.21&username=administrator&password=password&model=mscm
<newell> oh okay yeah...now we need to change the file
<newell> ctlaugh_: can you do this...
<newell> open with admin rights /usr/lib/python2.7/dist-packages/maasserver/api/nodegroups.py
<newell> do a search for mcsm and switch it to mscm
<newell> save the file
<newell> then on the command line restart the region by doing....
<newell> $ sudo restart maas-regiond
<newell> then retry using the webui
<ctlaugh_> newell: That worked -- no visible "success" message, but the "panel" closed and a few seconds later, a new node that I had previously deleted reappeared.
<newell> ctlaugh_: cool, thanks for testing that out for me
<ctlaugh_> newell: You're welcome.  Glad that's all it is :)
<newell> ctlaugh_: I typo'd it in the source and unit tests so didn't catch until you reported this, thanks
<newell> kiko: https://code.launchpad.net/~newell-jensen/maas/fix-1465353/+merge/261997
<newell> I know this needs to be backported to 1.8 branch as well
<newell> kiko: gotta run for doctors appointment but will do an MP for 1.8 branch when I get back
<mup> Bug #1465367 opened: Enhancement: Display node architecture in table <MAAS:New> <https://launchpad.net/bugs/1465367>
<negronjl> Can any of you guys point me to an example of a commissioning script ?
<negronjl> I can't seem to get a simple script working and I think an example would point me in the right direction
<negronjl> kiko, roaksoax ^^ ?
<kiko> negronjl, roaksoax is on vacation, blake_r is on paternity leave and I am clueless, but..
<kiko> let me try and help
<negronjl> haha
<negronjl> hey kiko ... thx for trying
<negronjl> kiko: Im trying this on MAAS 1.8 if it makes any difference
<kiko> negronjl, I don't think it does, and to confirm, this is not about a preseed, but rather custom commissioning, right?
<negronjl> kiko, correct
<kiko> negronjl, what are you wanting to do during commissioning incidentally?
<kiko> (brb)
<mup> Bug #1465367 changed: Enhancement: Display node architecture in table <MAAS:New> <https://launchpad.net/bugs/1465367>
<negronjl> kiko: Trying to modify /etc/environment by adding some http_proxy .... and https_proxy stuff to it
<negronjl> kiko: Any example that creates a file should give me a good idea
<negronjl> kiko: creates and populates a file I should say
<kiko> negronjl, hmm, but how could that be relevant for commissioning? that feels like something you want to do during install no?
<negronjl> kiko: That's the thing ... I can't find any documentation on when the commissioning scripts run or if they persist ... so I'm trying there ...
<negronjl> kiko: Is this something that I should be doing via /etc/maas/preseed/preseed_master ?
<kiko> ah!
<kiko> negronjl, what you want is a custom preseed
<kiko> negronjl, I happen to have an example of that
<negronjl> kiko: cool
<kiko> http://paste.ubuntu.com/11549501/
<kiko> negronjl, it's a bit cryptic so bear with me
<kiko> negronjl, I think this can only be done via the CLI at the moment
<kiko> (and smoser can correct me if I'm wrong)
<kiko> but
<kiko> $ maas home-smoser node start node-79b67e82-d25c-11e4-a333-00163eca91de user_data=XXX= distro_series=trusty
<kiko> negronjl, the user_data piece is the secret there; it's a base64-encoded string
<kiko> in this case, this is the script that was encoded
<kiko> http://paste.ubuntu.com/11549482/
<negronjl> kiko: The encoded script is what you used in user_data=..... right ?
<smoser> negronjl, you pass in base64 encoded user_data. cloud-init reads that user_data during first boot of the installed system.
<negronjl> kiko: This works for me ... it does what I need
<negronjl> kiko: Any chance of setting this globally ?
<smoser> no.
<negronjl> smoser, ack ... thx
<smoser> you need vendor_data for that (which maas *does* need)
<smoser> well, so not 'no'.
<smoser> you oculd do it in the preseed_master as you showed or in a late_command to /etc/maas/preseeds/curtin_userdata
<negronjl> yup ... that works for me
<smoser> negronjl, just a warning, though, /etc/environment will not be paid attention to by upstrat.
<smoser> upstart
<mup> Bug #1465367 opened: Enhancement: Display node architecture in table <MAAS:New> <https://launchpad.net/bugs/1465367>
<negronjl> damn proxies ... they are everywhere here ... so I'll have to figure something out for that
<smoser> or services that it invokes.
<negronjl> kiko, smoser: thx for the help ... this gives me the information i need
<smoser> negronjl, for such an environment, /etc/environment is the best i've come up with though. :-(
<negronjl> smoser, yup ... that and a hijacking iptables rules to redirect to proxy :/
<negronjl> not elegant but, it works ( mostly )
<smoser> you can make it more dynamic by putting a forwarding proxy like tinyproxy on the system.
<smoser> and pointing everything at that.
<smoser> oh, i never tried hijacking iptables rules.
<smoser> http://bazaar.launchpad.net/~smoser/+junk/sstack-proxy/view/head:/sstack-proxy
<smoser> that is what I do  some places.
<smoser> it is a pain though, in that it means all connections require the intermediate proxy.
<negronjl> smoser: and some things like embedded pip commands ignore proxies altogether
<smoser> yeah, its AWESOME.
<negronjl> The iptables rule that I use ( normally on the MAAS server ): sudo iptables -t nat -A PREROUTING -i <internal_interface> -p tcp -m multiport --dports 80 -j REDIRECT --to-ports 3128
<negronjl> and http_port 3128 transparent somewhere in squid.conf
<kiko> smoser, ah, late_command in curtin_userdata is the piece I was missing
<kiko> smoser, is there a bug on vendor_data for maas?
<smoser> i guess not, kiko.
<smoser> i answered 'vendor_data' to you a while ago when you asked about landscape integration
<kiko> I am clueless about these magic feature strings though :)
<kiko> negronjl, would it be annoying to ask you to post an ask.ubuntu.com question that I could respond with this?
<kiko> negronjl, it could serve as an initial stab for proper documentation
<negronjl> kiko: Sure .. give me a few to test this out
<negronjl> kiko: https://askubuntu.com/questions/636837/are-there-examples-of-commissioning-scripts
<kiko> negronjl, you rock^2
<kiko> negronjl, smoser: http://askubuntu.com/questions/636837/are-there-examples-of-commissioning-scripts/636867#636867
<kiko> your comments welcome
<kiko> smoser, I would like some detail on vendor_data and your reference to preseed_master
<kiko> ah, I see, it could be added to preseed_master as opposed to the late_commands stanza in curtin_userdata
<kiko> the naming of these files is FUCKING CONFUSING!!
#maas 2015-06-16
<dimitern> blake_r, rvba, meeting?
<rvba> dimitern: sorry, I have a conflicting meeting.  Can we postpone?
<rvba> dimitern: unless mpontillo can cover for me.
<mpontillo> rvba: I'm here... but I think the only thing we need to discuss is "the fan"... and I'd want you to be there for that, rvba
<rvba> mpontillo: okay, so let's postpone if that's okay with you dimitern.
<dimitern> rvba, fine with me
<rvba> dimitern: can we talk in, like 1h?
<dimitern> rvba, I won't manage
<rvba> dimitern: okay, let's pick another time then.
<dimitern> rvba, but mpontillo will brief you up :)
<kiko> negronjl, smoser, rvba: for your review, http://askubuntu.com/questions/636837/are-there-examples-of-custom-installation-scripts/636867#636867
<smoser> hmm...
<smoser> "As background, MAAS uses cloud-init as part of its installation process; cloud-init has the concept of preseeds, which in MAAS describe actions to be run at different points in the node lifecycle. "
<smoser> that is odd
<kiko> smoser, it's because I don't know what I am talking about. could you help clarify?
<kiko> smoser, is it odd but factually correct?
<smoser> kiko, no.
<smoser> let me try to write something
<kiko> smoser, thanks
<smoser> kiko, http://paste.ubuntu.com/11725176/
<smoser> so i started writing some information for you... so that i could then clarify the doc.
<smoser> but that got long winded... so you might be able to use some of that, it might be useful for your general undersanding...
<kiko> thanks!
<smoser> http://paste.ubuntu.com/11725201/ <-- just without long lines
<smoser> http://paste.ubuntu.com/11725206/ <-- without *any* long lines
<negronjl> smoser, kiko: This works
<kiko> smoser, I still think that first paragraph is correct; I mean, is there something factually wrong?
<kiko> first MAAS does use cloud-init, correct?
<kiko> second, cloud-init has preseeds, yes?
<kiko> I guess the third piece is not very correct in that the actions in the preseeds are run when booting the ephemeral environment and in the first system boot
<smoser> "preseeds" is a bad word.
<smoser> "preseed" likely implies d-i.
<kiko> smoser, what are they called in curtin?
<smoser> cloud-init doesn't really use 'preseeds'. it uses user-data.
<kiko> I guess I'm confused because they go into a directory called preseeds?
<smoser> curtin reads curtin config.
<kiko> hmm
<kiko> okay, so the yaml-ish files are curtin config?
<smoser> curtin reads curtin config
<smoser> d-i reads d-i config
<smoser> d-i config (and debian package install time configuration) is known as "preseed"
<smoser> i'd avoid the use of preseed when not talking about d-i or debian package preseeds.
<smoser> as that would most certainly confuse me.
<smoser> ie, google 'preseed linux' : https://en.wikipedia.org/wiki/Preseed
<mup> Bug #1465722 opened: 1.8.0 Machine details styling <ui> <MAAS:In Progress by ricgard> <https://launchpad.net/bugs/1465722>
<mup> Bug #1465726 opened: 1.8.0 machine details button sizes not styled dynamically <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1465726>
<mup> Bug # opened: 1465732, 1465735, 1465736, 1465737, 1465739, 1465740
<mup> Bug # changed: 1465732, 1465735, 1465736, 1465737, 1465739, 1465740
<mup> Bug # opened: 1465732, 1465735, 1465736, 1465737, 1465739, 1465740, 1465742, 1465743
<mup> Bug #1465742 changed: 1.8.0 Table design styles <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1465742>
<mup> Bug #1465743 changed: 1.8.0 Tags spacing <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1465743>
<mup> Bug #1465742 opened: 1.8.0 Table design styles <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1465742>
<mup> Bug #1465743 opened: 1.8.0 Tags spacing <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1465743>
<kiko> smoser, how does cloud-init fit in with curtin and d-i? Is it that both curtin and d-i provide stuff for cloud-init to run, and the way you provide that stuff is different for curtin and d-i?
<smoser> cloud-init has almost nothing to do with d-i
<smoser> the only overlap between them is that maas installs the cloud-init package during a d-i install and "preseed"s the configuration of the maas datasource . so that on first boot, cloud-init knows where to find maas.
<cholcombe> maas: for block devices who does the formatting?  maas or the juju charm?
<kiko> cholcombe, the system installation block device or additional block devices?
<kiko> cholcombe, for the former, the OS install done by MAAS (with di-or-curtin/cloud-init) does the formatting/resizing
<kiko> for the latter the charm does it
<cholcombe> i see yeah that's exactly what i needed to know
<cholcombe> i'll take that into account
<cholcombe> i was referring to additional block devices
<kiko> cholcombe, check out the ceph charm, I believe it knows how to handle additional block devices on the OST
<cholcombe> good point.  there's some code in there to test if a block device is legit.  that's prob why that exists
<kiko> oh, to avoid clobbering the system block device for instance?
<cholcombe> possibly or just to make sure it's not getting a bogus drive specified
<kiko> smoser, how come the user_data you supply to node start is always considered to be "late_commands" for curtin?
<kiko> or is that also not exactly right?
<kiko> ah, I think I understand now
<smoser> kiko, its 2 separate cloud-inits entirely.
<smoser> and actually 2 differnet cloud-init endpoints (or at least it used to be
<smoser> i tried to clarify that in in http://paste.ubuntu.com/11725206/
<smoser> line 72 there.
<kiko> smoser, here's an updated attempt, hopefully simpler: http://askubuntu.com/questions/636837/are-there-examples-of-custom-installation-scripts/636867#636867
<smoser> kiko, i'd strip "MAAS uses cloud-init as part of its installation process." as you dont really mention that in any other way, and its just confusing. the end user of maas doesn't care that maas uses cloud-init inside the installation process.
<smoser> also, the first paragraph somewhat implies that cloud-init is not available to the user if d-i is used for isntallation
<smoser> i dont want to nit-pick, you're welcome to ignore that if you think its bike-shedding
<kiko> smoser, yeah, I think I see your point, but it leads to more questions :)
<kiko> smoser, for instance, does user_data do something when you are deploying with d-i? :)
<smoser> kiko, it does something *after* you deploy with d-i
<smoser> whihc is what the end user cares about
<smoser> from their perspective maas just gives them a machine with that has cloud-init inside it.
<smoser> same as ec2.
<kiko> smoser, so you could use the user_data mechanism regardless of whether curtin or d-i is being used?
<smoser> yes
<kiko> oh.
<kiko> could I delete the "Depending on which installer you are using.." paragraph then?
<kiko> ah
<kiko> but of course, the file in /etc/maas/preseeds differss
<smoser> kiko, right. the mechanism for customizing installation differs between installers (they're different installers)
<smoser> but the mechanism for communicating with the installed Ubuntu is consistent.
<plars> what's the best way of importing and booting an image from an iso?
<plars> I've found lp:curtinator and managed to import the results of running that against an daily wily snapshot. And it shows up under "images" in maas in the "Generated Images" section
<plars> but I can't find where to tell it to boot that image. My preference would be to boot from the cli if that's possible
<plars> I'm currently on 1.7.5+bzr3369-0ubuntu1~trusty1 from the stable ppa
<kiko> plars!
<plars> Heya kiko!
<kiko> plars, let me check the API docs, but basically you specify a distro series name when telling a node to start
<plars> kiko: when importing it, I did somthing like: maas testmaas boot-resources create name=ubuntu/deskwily architecture=amd64/generic content@=/tmp/wily-desktop-amd64.iso.tar.gz
<kiko> oh
<kiko> I think you just say
<plars> kiko: and tried booting with something like: maas testmaas node start node-5337016e-3d56-11e4-a8dd-001cc08caad0 distro_series=deskwily
<kiko> maas plars node uuid start distro_series=deskwily no?
<kiko> right
<plars> kiko: right, I would have though so, but it says it's an invalid series
<plars> I purposely gave it a bogus series name so it wouldn't conflict with anything
<plars> surely that's not hardcoded to known releases though, right?
<plars> I wouldn't think so, I've heard of people booting centos, rhel, etc
<kiko> plars, correct
<kiko> plars, does that option show up in the UI at all?
<plars> kiko: well, I see the image under images, but if I acquire a node and look at the options it gives me for the image, I don't see it there
<kiko> that is not good
<kiko> plars, can you put a screenshot of /images somewhere?
<plars> kiko: sure, one sec
<plars> kiko: http://imgur.com/rPVZKP5
#maas 2015-06-17
<awojo> hi
<awojo> If I setup DHCP through MAAS will it cause conflicts with my existing DHCP server
<vibol> Hello,!
<vibol> i use vmware to test maas ... maas requre power type to boot up the node
<vibol> So any recommend for this ? i don't which power type should i choose and i don't know even power address or etc
<dweaver> Trying the MAAS trusty hwe-v kernel using the daily images and the kernel panics on boot.  See bug: https://bugs.launchpad.net/maas/+bug/1465071
<kiko> dweaver, you rock thanks
<kiko> smoser, ^^
<smoser> dweaver, fix is coming :)
<smoser> dweaver, and actually, should be fixed i hope.
<dweaver>  kiko smoser, stellar, thank you so much ;)
<smoser> dweaver, seems still busted even with 20150611, but it shoudlnt be. investigating.
<kiko> plars, rvba thinks you have encountered a bug
<plars> kiko: ah, ok
<kiko> plars, however, we know that some custom images work, i.e. centos
<rvba> kiko: plars: tbh, I'm not that familiar with the custom images code so I can't be sure this is a bug (maybe there is just something I'm missing)â¦ and we don't have documentation for this feature unfortunately.
<kiko> rvba, does anyone other than blake_r know?
<rvba> kiko: roaksoax probably.
<kiko> rvba, and other than him? :)
<rvba> Nobody else really.
<smoser> dweaver, luser error made me think it was still busted.
<smoser> i think i booted too early and got old initramfs.
<smoser> if it is not fixed for you, and you have
<smoser>  20150611 for ubuntu/*/hwe-v/trusty
<smoser> then please let me know.
<dweaver> smoser, that version is in the daily images now?
<smoser> yes.
<smoser> refresh either from web ui, or cli (maas <maasname> boot-resources import)
<smoser> it is only there for ~ 2 hours though.
<dweaver> smoser, OK, I'll sync and have another go, if you don't hear anything else from me then you must be awesome and fixed it, thanks
<dweaver> smoser, OK, syncing now.
<mup> Bug #1465071 changed: Daily hwe-v kernel for trusty results in kernel panic when attempting to mount iscsi partition <MAAS:New> <https://launchpad.net/bugs/1465071>
<mup> Bug #1466151 opened: MAAS unit tests (at a minimum) are broken when bonding driver is loaded <MAAS:In Progress by mpontillo> <https://launchpad.net/bugs/1466151>
<rvba> plars: kiko: Actually you can't use 'ubuntu' as a prefix for your image name because the suffix isn't a valid series name.
<rvba> plars: so just drop the 'ubuntu/' in front of your image.  It should then show up under 'Custom'.
<plars> interesting, I'll give it a try in a bit
<plars> thanks rvba
<rvba> plars: you can check that it's correctly synced by making sure /var/lib/maas/boot-resources/current/custom/amd64/generic/ contains the name of your image.
<rvba> Welcome.
<kiko`> thanks plars
<kiko`> and thanks^2 rvba
<mup> Bug #1466162 opened: commissioning error <MAAS:New> <https://launchpad.net/bugs/1466162>
<CrummyAdmin> Good Afternoon
<CrummyAdmin> I am running an ubuntu server 14.04 instance with MAAS v1.5 as a virtual guest in Virtualbox. I am setting the regional controller, cluster controller, and dhcp server on the same server. eth0 is connected to the NAT network and has public internet connection. eth1 is reserved for the dhcp server. Scope for the NAT network is 10.0.2.0/24 so eth0 is 10.0.2.254 and eth1 is 10.0.2.3.
<CrummyAdmin> eth1 is also a host-only network adapter. DHCP was working well though and assigning numbers in the 10.0.2.0/24 scope and I still had internet connection. After I ran dpkg-reconfigure maas-regional-controller and dpkg-reconfigure maas-cluster-controller and setup accordingly I can no long reach the public internet
<CrummyAdmin> what is the best way to troubleshoot this?
<kiko> CrummyAdmin, you can't reach the internet from the server, or from nodes provisioned from it?
<kiko> also
<kiko> don't use 1.5
<kiko> use the stable PPA above
<CrummyAdmin> I can't reach the internet from the server
<CrummyAdmin> I still needed to download images for maas but am unable to currently
<CrummyAdmin> Should I go ahead and upgrade maas even though I am currently having this issue or wait?
<CrummyAdmin> I can just start from scratch again but would like to be able to troubleshoot for the experience
<kiko> CrummyAdmin, you should upgrade maas regardless
<kiko> it's a bit weird that your uplink went down -- how are your interfaces configured?
<kiko> plars, did that work?
<plars> kiko: not so far... it seems to be syncing?
<plars> I ran:
<plars> maas testmaas boot-resources create name=ci/wily architecture=amd64/generic content@=/tmp/wily-desktop-amd64.iso.tar.gz
<plars> and I see it under images, but it's still under generated, not custom
<kiko> plars, hmm, and are either doing something?
<plars> and there's a circle next to it, when I hover, it says "waiting for clusters to sync"
<kiko> okay
<plars> is there a way to force that?
<kiko> plars, is df changing? :)
<kiko> no, it should happen automatically
<mup> Bug #1466213 opened: commissioning output should show the output of lsblk and  udevadm commands used to gather block device data <MAAS:New> <https://launchpad.net/bugs/1466213>
<plars> kiko: not quickly if it is
<plars> kiko: this was many hours ago that I ran the create
<kiko> plars, and the cluster is healthy, connected, etc?
<plars> kiko: seems to be - it's just a small test system I set up locally
<plars> kiko: just trying to figure out what options maas could provide for testing things based on daily isos
<kiko> plars, I am sure we can do things to make things better
<CrummyAdmin> eth0 static 10.0.2.254 gateway 10.0.2.3 dns-search google.com dns-nameservers 8.8.8.8 8.8.4.4 netmask 255.255.255.0
<CrummyAdmin> eth1 static 10.0.2.3 dns-search google.com dns-nameservers 8.8.8.8 8.8.4.4 netmask 255.255.255.0
<CrummyAdmin> Destination     Gateway     Genmask     Flags  Metric  Ref  Use  Iface
<CrummyAdmin> 0.0.0.0     10.0.2.3     0.0.0.0     UG  0  0  0  eth0
<CrummyAdmin> 10.0.2.0     0.0.0.0     255.255.255.0 U  0  0  0  eth0
<CrummyAdmin> 10.0.2.0     0.0.0.0     255.255.255.0  U  0  0  0  eth1
<CrummyAdmin> that is my route -n output
<CrummyAdmin> arp -a = "? (10.0.2.3) at <incomplete> on eth0
<kiko> CrummyAdmin, you are really a crummy admin :-) you can't have two interfaces on the same network without special handling
<kiko> CrummyAdmin, what is eth1 supposed to be doing?
<kiko> CrummyAdmin, perhaps it's worth taking a step back and explaining your network setup
<CrummyAdmin> I was going to make it a private subnet for the dhcp server but couldn't get internet so someone suggested issuing ip's from it that are on the same subnet so I would still get internet. up until configuring maas it seemed to be working okay lol I am pretty new at this and just doing this to learn
<kiko> sure
<kiko> so the right way to do that is to use the machine to also route from eth1 to eth0
<kiko> if you shut down eth1 for now do you get your internet back?
<kiko> what are eth0 and eth1 plugged into?
<CrummyAdmin> So with virtualbox I have a nat network with dhcp turned off and the Network CIDR as 10.0.2.0/24. this is the connection to the public internet that my maas server has on eth0
<kiko> ah
<kiko> you are running maas inside virtualbox
<kiko> I see
<CrummyAdmin> No I used ifdown eth1 and try connection again and could not ping google.com
<kiko> that's odd
<CrummyAdmin> could a restart be necessary?
<kiko> well
<kiko> ping google.com is not a good test
<kiko> can you ping 8.8.8.8?
<CrummyAdmin> I have 3 physical servers at home I plan to deploy to but learning to configure in virtualbox while on the go seemed to be my best option for now
<kiko> right
<kiko> so most simple setups do the following:
<kiko> - use a server with two interfaces for maas
<CrummyAdmin> from 10.0.2.254 imcp_seq=1 Destination Host Unreachable
<kiko> - maas is configured to manage DHCP/DNS on one of those interfaces
<kiko> - maas is configured to ignore the other interfaces
<CrummyAdmin> okay yeah I told maas to use eth1
<kiko> - you configure routing (and possibly NAT, though not in your case) to allow machines on the managed interface to get outside
<kiko> right
<CrummyAdmin> and I setup dhcp using isc-dhcp-server before hand then provided the config to maas within the web gui
<kiko> oh
<kiko> you shouldn't set up dhcp manually
<kiko> maas does that for you
<kiko> well, unless you really want to
<kiko> but that is not a trivial undertaking
<CrummyAdmin> Oh! I was wondering about that
<kiko> you should also really be on 1.7
<kiko> REALLY
<kiko> I need to split tongiht
<CrummyAdmin> I promise once I get a connection back I will upgrade immediately =)
<kiko> heh
<CrummyAdmin> k thanks for the suggestions I will google routing
<kiko> so I'm not sure what is wrong -- eth1 down should have
<kiko> scratch that
<kiko> I know what is wrong
<kiko> your default route is wrong
<kiko> how are you configuring your networks
<kiko> network manager?
<kiko> or /etc/network/interfaces?
<CrummyAdmin> all through terminal no gui
<CrummyAdmin> yeah /etc/network/interfaces
<kiko> ok
<kiko> so if you look at your setup
<kiko> you have a default route set up to the address of your eth1
<kiko> that can't work
<kiko> your default route needs to be the first hop external to you
<kiko> which I would assume is 10.0.2.1
<kiko> if you shut down eth1 and fix your default route you will have internet back
<kiko> and later
<CrummyAdmin> later thank you again!
<kiko> you will need a second subnet which MAAS will manage, such as 10.0.3.0/24
<CrummyAdmin> okay that's what I originally wanted to do
<kiko> right
<kiko> that should work
<kiko> you don't even need to do anything to route if that's the config
<kiko> as packets coming in from eth1 that are intended for outside will be told to go to your default route, which is reachable by eth0
<CrummyAdmin> so what would 10.0.2.1 be if dhcp is turned off on the nat network and my dhcp server is currently 10.0.2.3
<kiko> well
<kiko> if you are sure eth1 is down
<kiko> can you ping 10.0.2.3?
<CrummyAdmin> should I turn dhcp back on for the nat network. Yes eth1 is down
<CrummyAdmin> nope host unreachable at the moment
<kiko> dhcp is turned off where? outside the virtualbox?
<kiko> I think testing this inside a VM is much harder fwiw
<CrummyAdmin> virtualbox has a virtual network that you can enable dhcp on and virtual box will distribute addresses like a router from the virtual adapter
<CrummyAdmin> that is what eth0 is connected to
<kiko> sure
<kiko> what is the gateway address supposed to be on that network?
<kiko> if it's 10.0.2.3, when you should be reaching it from eth0
<kiko> but perhaps you confused virtualbox when you added a second interface with that same address
<CrummyAdmin> virtual box uses 10.0.2.1 as the gateway I believe
<CrummyAdmin> if it is turned on to handle it
<kiko> can you ping 10.0.2.1?
<CrummyAdmin> yes
<kiko> okay
<kiko> then make that your default route
<kiko> your gateway in /e/n/i speak
<kiko> your internet will come back
<CrummyAdmin> okay
<CrummyAdmin> and it did!
<CrummyAdmin> sweet thank you! Takeaways: let maas configure dhcp, keep both nic on seperate subnets, use routing to configure default route and enable public internet on both nics
<CrummyAdmin> and upgrade my maas install
<kiko> correct
<kiko> uhh
<kiko> no
<kiko> don't enable public internet on both nics
<kiko> well
<kiko> yes
<kiko> I think you can't basically
<CrummyAdmin> Sorry for dropping kiko, thanks again for your assistance. Huge help
<kiko> <kiko> correct
<kiko> <kiko> uhh
<kiko> <kiko> no
<kiko> <kiko> don't enable public internet on both nics
<kiko> <kiko> well
<kiko> <kiko> yes
<kiko> <kiko> I think you can't basically
<kiko> did you get that CrummyAdmin?
<kiko> maas should route
<kiko> not virtualbox
<CrummyAdmin> okay I make up a name of an adapter in the maas web gui correct then put in the details for the subnet I would like it to create on eth1 the private subnet and let it do it's thing
#maas 2015-06-18
<bdx> Hey whats up everyone? ... I run a setup where dns and dhcp run elsewhere than my maas region/cluster controller .... everything seesms to be working fine other than the fact that my maas api no longer contains any entries for  nodes ip
<bdx> If this happens to be a bug and there is no work around for it, I am thinking is that I could create static mappings in my dhcp server, and update the maas api with the nodes ip addresses by hand if needed
<bdx> hopefully someone who's ran into this before might provide some insight....thanks
<bdx> https://www.dropbox.com/s/gx6q5sijldxt04z/Screenshot%202015-06-17%2019.13.59.png?dl=0
<bdx> http://paste.ubuntu.com/11733581/
<bdx> I have ddns configured such that my dhcp server updates my authoritative dns server when it hands out an ip; form which an A record in the zone and a reverse record in the reverse zone are created
<bdx> my node is immediatly resolvable forward and backward by means of my nameservers
<bdx> although, I'm guessing this has nothing to do with why the maas api neglects to add the ip_addresses entry
<bdx> more or less why maas neglects to add the entry to the api
<bdx> anyway...I am headed in for the night...I'll check back later to see if anyone has left any suggestions..Thanks
<bladernr__> Hey, is anyone around?
<bladernr__> more specifically, is anyone around who knows about the way maas imports images from simplestreams?
<bladernr__> We're stuck.  MAAS still shows the import in progress but all the images in /var/lib/maas/boot-resources/cache are no longer growing in size, so it appears the actual download has stopped.
<bladernr__> or completed
<bladernr__> but there is no way I can see to make the cluster sync from the region.
<bladernr__> mattrae_: are you up/
<bladernr__> gah, is there any way to force it to re-scan the images on the region and re-attempt to import to the cluster?
<bladernr__> So looking at maas-cli, all the images appear to be marked complete and synced.  but the maas UI still shows them as "importing" and there's nothing I can do to stop it
<bladernr__> Sigh...
<bladernr__> ok, finally got it kicked and it completed.
<Mmike> Hi, lads. What does it mean when maas web-interfcase says that 'my cluster is disconnected'
<kiko> bladernr_, I know some of it
<kiko> bladernr_, but happy to see you've fixed it
#maas 2015-06-19
<mup> Bug #1466753 opened: maas 1.8rc3 missing dependency on ipmitool <cpec> <MAAS:New> <https://launchpad.net/bugs/1466753>
<mup> Bug #1466753 changed: maas 1.8rc3 missing dependency on ipmitool <cpec> <MAAS:Won't Fix> <https://launchpad.net/bugs/1466753>
<Daniel> Hi everybody
<smoser> hey.
<smoser> node goes into RELEASING_FAILED
<smoser> how can i move it into RELEASED without UI ?
<smoser> never mind. just doing 'maas <name> node release <sysid>' again will do it.
<mup> Bug #1466852 opened: doesnt wait long enough for release power off on power <MAAS:New> <https://launchpad.net/bugs/1466852>
<dweaver> Anyone know if MAAS 1.7.5 will still be available after MAAS 1.8 is released into the stable PPA?
<kiko> dweaver, yes, it will actually be in trusty
<dweaver> kiko, thanks very much, that's great, as 1.5 is pretty useless now
<mup> Bug #1466979 opened: 504 GATEWAY TIMEOUT when trying to start a node via API <oil> <MAAS:New> <https://launchpad.net/bugs/1466979>
<mup> Bug #1466979 changed: 504 GATEWAY TIMEOUT when trying to start a node via API <oil> <MAAS:New> <https://launchpad.net/bugs/1466979>
<mup> Bug #1466979 opened: 504 GATEWAY TIMEOUT when trying to start a node via API <oil> <MAAS:New> <https://launchpad.net/bugs/1466979>
<mup> Bug #1466979 changed: 1.8rc3: 504 GATEWAY TIMEOUT when trying to start a node via API <oil> <MAAS:New> <https://launchpad.net/bugs/1466979>
#maas 2015-06-20
<mup> Bug #1464989 changed: MAAS fails to install on UEFI NUC with curtin bcache error <MAAS:Invalid> <https://launchpad.net/bugs/1464989>
<mup> Bug #1467119 opened: MAAS does not factor in BMC addresses to address availability <MAAS:New> <https://launchpad.net/bugs/1467119>
<mup> Bug #1467120 opened: MAAS should consider device addresses as used <MAAS:New> <https://launchpad.net/bugs/1467120>
#maas 2015-06-21
<mup> Bug #1467213 opened: MAAS 1.8rc3 broken bind's zone files <cpec> <MAAS:New> <https://launchpad.net/bugs/1467213>
#maas 2016-06-20
<mup> Bug #1594207 opened: DHCP6 problems prevent DHCP from starting <MAAS:New> <https://launchpad.net/bugs/1594207>
<mup> Bug #1590689 opened: MAAS 1.9.3 + Juju 1.25.5 - on the Juju controller node eth0 and juju-br0 interfaces have the same IP address at the same time <cpec> <juju> <maas> <sts> <juju-core:Fix Released> <juju-core 1.25:In Progress by dimitern> <MAAS:Confirmed> <https://launchpad.net/bugs/1590689>
<mup> Bug #1594317 opened: Cannot start lxd-bridge.service when MAAS is managing DNS <dns> <MAAS:New> <https://launchpad.net/bugs/1594317>
<mup> Bug #1594325 opened: rndc-confgen: command not found <dns> <MAAS:New> <https://launchpad.net/bugs/1594325>
<D4RKS1D3> Hi, someone know where is save the "pxe interface" in the postgresql database?
<D4RKS1D3> The gui paint a grey dot in the interface.
<D4RKS1D3> I see all the tables but I do not see this information.Thanks.
<dannf> w/ maas 2.0, how do i deploy a node from the cli?
<dannf> ubuntu@cvm12:~$ maas maas machine deploy 4y3h84
<dannf> Can't start machine: it hasn't been allocated
<jhegge> Working on CLI deploy as well, I found that I have to allocate first, something like "maas maas machines allocate system_id=4y3h84" then the deploy can be tried
<dannf> jhegge: brilliant! thx
<mup> Bug #1594588 opened: Hundreds of duplicate entries in /var/lib/maas/dhcp/dhcpd.leases <MAAS:New> <https://launchpad.net/bugs/1594588>
<mup> Bug #1594588 changed: Hundreds of duplicate entries in /var/lib/maas/dhcp/dhcpd.leases <MAAS:Invalid> <https://launchpad.net/bugs/1594588>
#maas 2016-06-21
<jojden> how to parse output of maas node details
<mup> Bug #1594317 changed: Cannot start lxd-bridge.service when MAAS is managing DNS <dns> <MAAS:Invalid> <bind9 (Ubuntu):New> <https://launchpad.net/bugs/1594317>
<mup> Bug #1594317 opened: Cannot start lxd-bridge.service when MAAS is managing DNS <dns> <MAAS:Invalid> <bind9 (Ubuntu):New> <https://launchpad.net/bugs/1594317>
<mup> Bug #1594317 changed: Cannot start lxd-bridge.service when MAAS is managing DNS <dns> <MAAS:Invalid> <bind9 (Ubuntu):New> <https://launchpad.net/bugs/1594317>
<jojden> how to parse node details
<jojden> GET /api/1.0/nodes/{system_id}/ op=details
<jojden> this gives BSON value
<jojden> i need to parse this value
<mup> Bug #1549842 changed: [2.0] Failed to update this region's process and endpoints; unleashed:pid=28940 record's may be out of date <MAAS:New> <https://launchpad.net/bugs/1549842>
<valeech> maas 2.0 beta 3 when I try to update a subnetand move it into a specific space I get the following error: 'NoneType' object has no attribute 'split'
<jhegge> valeech: There are quite a few little MAAS UI bugs like that in beta 3, if you can upgrade to beta 5 or later, most are gone
<valeech> jhegge: thanks. Iâll check it out.
<valeech> now running maas 2.0 beta 7. When I try to load the GUI, I get this error where the interface should be: Error Occured : 'NoneType' object is not iterable
<bdx> blake_r_: will you make a new maas-rack  rev with a current `charm build` ?
<bdx> install hook is failing for maas-rack due to pip version < 8.1.1
<valeech> ok, the NoneType error only shows up on the Nodes and Networks tabs. The other tabs work fine.
<sjl> submitting 3 parallel API calls to op=allocate fails the 2nd and 3rd call, but staggering the 3 calls with random delays succeeds.  Is there a best practice with API calls to ensure safe calls?
<mup> Bug #1594975 opened: simultaneous api calls to op=allocate errors  <MAAS:New> <https://launchpad.net/bugs/1594975>
<valeech> Fresh install of maas 2.0 beta 7. Trying to get juju to bootstrap and having issues. I would like to ssh into the node while juju is bootstrapping so I can watch the status. When I try to ssh to the node I get Permission denied (publickey). If I manually deploy the same node, I am able to SSH in no problem. So when maas deploys its not installing the SSH key. I also noticed that it is not deploying the correct version of ubuntu. I have commi
<valeech> and deploy set to 14.04. When maas auto deploys, it installs 16.04. Again, I can manually deploy and the correct os gets installed and I can SSH.
<mup> Bug #1594991 opened: [2.0RC2] MAAS logs every power query as a NodeEvent <MAAS:Triaged> <https://launchpad.net/bugs/1594991>
#maas 2016-06-22
<tinchux> hi! everyone
<tinchux> I need some /HELP
<tinchux> provisioningserver.drivers.hardware.virsh.VirshError: Unknown state: apagado
<tinchux> I'm receiving this error on my maas controller node
<tinchux> all my machines have this status "	Failed commissioning"
<tinchux> I'm using MAAS Version 2.0.0 (beta7+bzr5112)
<tinchux> with Ubuntu 16.04
<bdx> hey whats up everyone? I am testing out the maas-{region,rack} charms in ha, I seem to be at an impasse where I'm blocked due to "Miossing admin config" as shown here -> http://paste.ubuntu.com/17705033/
<kiko> bdx, oh, nice!
<kiko> sounds like the charm is broken
<kiko> the region charm
<kiko> who owns it?
<bdx> blake_r-_
<kiko> wow
<bdx> https://jujucharms.com/u/blake-rouse/
<bdx> the charms ^ in his namespace error on install though .... I had to pull them down and rebuild to get the wheelhouse deps with the latest pip
<kiko> ah they are new
<bdx> yeah ... I have a feeling theres a step that just might not be documented that I'm missing to pull the rest of it together
<bdx> blake_r_:^
<bdx> figured it out
<bdx> the admin-password set in maas-region configs on deploy isn't picked up, you must also set it again when you are blocked post-deploy
<bdx> ;-)
<LiftedKilt> maas 1.9 worked perfectly detecting and setting ipmi settings and deploying nodes, but I can't seem to get 2.0 (using beta 7 now) to recognize the power type on my nodes
<LiftedKilt> and idea where to poke to try and get it working?
<bugrum> Is there any documentation on how to troubleshoot MAAS when dealing with iDRAC IPMI?
<bugrum> I configured MAAS to act as a DHCP server and it seems to be properly assigning IP addresses if other machines are connected to a switch, however I tried to make the MAAS DHCP server into a multi-homed server so that it can provide IP addresses to an iDRAC network and it doesn't seem to be working
<bugrum> any machine connected to the server via iDRAC IPMI doesn't seem to get an IP address as I thought they would
<arosales> any folks available to help with a MAAS  2.0 beta 7 /  juju 2.0 beta 9 setup?
 * arosales working with valeech
<arosales> current issue: 2.0 beta 7 running in a vm. I have juju 2.0 beta 9
<jhegge> still running juju beta 7, i saw some maas breakage in beta8 so i have stuck with beta7 for the time being
<kiko> arosales, what's the actual problem?
<arosales> current problem definition http://paste.ubuntu.com/17708627/
<arosales> jhegge: hmm so I wonder if juju beta9 and maas beta7 are not compatibile
<mup> Bug #1595279 opened: ipaddresses API doesn't show hostname when reading <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1595279>
<arosales> kiko: ^ problem def above
<kiko> arosales, looks like a bug, but perhaps his network setup is wonky? the paste is really painful to read
<jhegge> arosales: no idea on beta9, wasn't released last i'd looked but beta8 didn't last so i'd hope beta9 was repaired. for what i'm doing beta7 of juju is working.  i agree with kiko, the network setup on the maas server affects the deploy
<kiko> jhegge, did you see the no-default-gw issue that valeech is seeing?
<kiko> jhegge, or perhaps put differently, what breakage did you see in beta8?
<arosales> jhegge so your set up is maas 2.0 beta7 and juju 2.0 beta 7, and this workng?
<jhegge> kiko: haven't seen the default gw issue, i've deployed and had all apt traffic back through the maas-proxy, nicely
<kiko> jhegge, even with beta8 you mean? :)
<jhegge> kiko: can't recall the beta8 issue, something scrolled past in #juju and well, i was busy working on ephemeral image deploys under maas anyway, so i have a full plate, didn't need more juju fun
<jhegge> still on juju beta7
<kiko> heh
<jhegge> 2 betas, working their way to release, it's a bit bumpy
<jhegge> arosales: i have maas beta7 (and a maas beta5) working with juju beta 7
<jhegge> but i'm trying not to the use the UI, just the API.  (and the CLI to work out ideas)
<jhegge> kiko: the network trouble we've seen is having to change the resolv.conf on the maas server to get the nodes to resolve <my maas node>.maas but that's as much an artifice of dev environments as anything
<jhegge> and you do have that doc'd on the end of the page on ... can't recall...
<kiko> I'm surprised changing resolv.conf on the maas server does anything actually
<kiko> jhegge, why would the local resolver there affect anything?
<jhegge> go to a node, ask the node to resolve its own FQDN, fail
<kiko> jhegge, sure, but that would be resolve.conf missing on /the node/, not the maas server?
<jhegge> also, depending on how maas is installed, hosts file doesn't get a good entry for the host itself
<arosales> jhegge: ok perhaps that is the recommendation for valeech
 * arosales not sure if anyone has validated juju2 beta9 and maas2 beta 7 :-/
<jhegge> yeah, the maas server with 2 NICs was part of the issue:  one NAT and one static/private net.  hosts file and resolv.conf may get written wrong on install by ubuntu
<jhegge> but uh, MAAS is made for metal so VMs are a bit trickier
<LiftedKilt> I can't get maas beta 7 to detect/set IPMI settings, so commissioning is failing for all my nodes - any ideas?
<LiftedKilt> it worked perfectly in 1.9
<kiko> LiftedKilt, do you mean the auto-enlist functionality is failing?
<LiftedKilt> kiko: when I pxe boot the nodes, they enlist in maas and show in the New state w/o any configuration for the power options
<LiftedKilt> in 1.9 the power settings were automatically configured for IPMI, and maas created the user and password for it
<jhegge> lifterKilt: are you commissioning with xenial or trusty?  we had better luck commissioning trusty with maas 2.x (just in a few cases, no idea why)
<LiftedKilt> it isn't doing that in 2.0, and thus the commissioning fails because it can't control the nodes
<LiftedKilt> trying everything in xenial
<LiftedKilt> jhegge: importing 14.04 now
<LiftedKilt> err 14.10
<LiftedKilt> err should have been 14.04 and I just selected the wrong release haha
<kiko> yes, it should have indeed
<jhegge> yeah, I stuck with the 2 LTS releases
<jhegge> i learned my lesson on Azure with 15.10 disappearing
<kiko> jhegge, MSFT knows how to keep one honest always!
<mpontillo> arosales: valeech: when you look at the subnet configuration in MAAS for the PXE network, is there a default gateway defined? if so, can it be removed?
<valeech> mpontillo: arosales: there was a gw on the pxe subnet and I removed it and it did not help.
<mpontillo> valeech: could you use the MAAS CLI to do something like, "maas <profile> machine get-curtin-config <system-id>" so we can see how the deployed node is being instructed to set up its network?
<valeech> mpontillo: yes, I will need to tear down my ha setup to put one of the machines back into ready stateâ¦ one sec.
<mpontillo> valeech: that command should be run on a deployed or deploying node
<LiftedKilt> jhegge: still not working
<valeech> mpontillo: ok, so I ran this: maas 20-juju machine get-curtin-config 4y3h7s and it returned âMachine overrighteous-concetta is not in a deployment state."
<mpontillo> valeech: also, I assume you've redeployed after changing the subnet configuration
<LiftedKilt> it won't even let me try to commission them without a power type selected
<mpontillo> valeech: yeah, that command only works if you're deploying or deployed. it will tell us what the configuration we sent to the installer was, including the network config
<valeech> mpontillo: yes I tried redeploying again. same situation, I had to ssh in to the node getting deployed and change the gw in order for juju to finish bootstrapping.
<LiftedKilt> I just can't figure out why it isn't configuring the power type
<mpontillo> valeech: can you run the get-curtin-config command while you're having trouble bootstrapping?
<valeech> mpontillo: am I using the wrong system-id?
<mpontillo> valeech: I'm not sure. what state is the node in? I assume when juju starts bootstrapping it will move to "Deploying". at which point you should be able to run the command on the bootstrapping node.
<mpontillo> valeech: juju probably won't try to do anything with it until it moves to "deployed", come to think of it
<valeech> mpontillo: right now, because I was able to change the gw, I have juju bootstrapped to 3 VMs all in the delpoyed state and juju status showing the 3 nodes running
<valeech> 3 nodes because I ran juju enable-ha
<mpontillo> valeech: then, yeah, please double check the system_id; you can run something like this: maas 20-juju nodes read | jq '.[] | {hostname:.hostname, system_id: .system_id, status:.status}' --compact-output
<mpontillo> valeech: the deployed nodes should have status = 6
<valeech> mpontillo: looks good: http://pastebin.com/xZdsM1Sk
<valeech> overrighteous-concetta is the first of many bare metal boxes.
<valeech> mponitllo: I was using the wrong system id. Here is the output of the get-curtin-config on one of the juju bootstrap nodes: http://pastebin.com/r6u0cya4
<mpontillo> valeech: ah, that node is in failed commissioning state
<mpontillo> valeech: thanks, taking a look
<mup> Bug #1595304 opened: add Subnet fails with "Extra data: line 1 column 8 (char 7) <MAAS:New> <https://launchpad.net/bugs/1595304>
<mpontillo> valeech: is 10.131.107.2 the correct gateway? that's the only one I see in the curtin config. I do have a theory though: in this configuration, you can either have successful deployments or successful commissioning. the reason being that, based on MAAS configuration, you can *either* have internet access during commissioning/enlistment (on the PXE network)
<mpontillo> or after deployment (on the deployment network)
<valeech> mpontillo: yes, 10.131.107.2 is the correct gateway. the pxe network is 10.0.0.0/23
<valeech> mpontillo: when you say either successful deployments or commissioning, is that something specifc in my setup or a general concept with juju/maas?
<mpontillo> valeech: with MAAS, when a machine first boots and registers, it is called enlistment. enlistment is a minimal statement of "this machine is on the network; we saw it with a MAC address and an IP address". when the machine enlists, it must contact MAAS to report this information back
<mpontillo> valeech: commissioning is similar but more involved; the machine will inventory its disks and network cards, hardware, etc, and send it back to MAAS.
<mpontillo> valeech: both commissioning and enlistment happen in what we call an "ephemeral" environment, in which we "live boot" over the network; no data is written to the machine's disks
<mpontillo> valeech: however, in order for successful commissioning (and in some cases, enlistment) to occur, generally the node must DHCP boot from MAAS and get a correct default gateway, in order to access the Ubuntu archive and download whatever software is necessary to complete the commissioning (or in the case of enlistment, enough to just contact MAAS)
<mpontillo> valeech: that all would work fine with MAAS knowing about the gateway on your PXE network. but if you start deploying machines and expecting them to not use the PXE network, we may have a problem
<mpontillo> valeech: when the machine deploys, we take the network configuration in MAAS and write it to disk - generally statically. (we assign a static IP address, etc)
<valeech> mpontillo: thatâs exactly whatâs happening. Would a local repository allow elisting/commissioning to work without a gw?
<valeech> mpontillo: I didnât realize elisting/commissioning required a gw.
<valeech> mpontillo: I thought the nodes would simply use the maas-proxy to get whatever they needed.
<mpontillo> valeech: well, I think there may be a better workaround. why not set the PXE interface to be unconfigured (no IP address), so that it won't be possible for the deployed node to accidentally use the gateway on the wrong network?
<valeech> mpontillo: That should work. I have no need for the PXE network outside of deploymentâ¦
<mpontillo> valeech: I would tend to agree that the proxy should allow enlisting in this situation. I'm not sure why that wouldn't be working.
<mpontillo> (and commissioning)
<valeech> mpontillo: Enlisting works great! Its commissioning that is having issues.
<valeech> and deployment
 * mpontillo needs to do more testing with the proxy; I usually don't use it in my test environment
<mpontillo> yeah, commissioning is what accesses the archive.
<valeech> ok
<valeech> whatâs the quickest way to tear down a juju ha? :)
<mpontillo> valeech: I would just release the nodes and delete the juju environment, but I haven't done much with juju 2.0. you might ask in #juju
<valeech> mpontill: fair enough
<valeech> mpontillo: That worked! Setting a gw on the pxe subnet so that machines can boot and enlist/commission but then leaving that interface unconfigured for deployment was the trick. Thanks for all of the help. Thank you to arosales also!
<mpontillo> valeech: great! glad you got it to work. thanks for the feedback and for trying MAAS.
<valeech> mpomtillo: you are welcome. my team is installing maas/juju/openstack. this is the first go I am having at it with. so far it is awesome! my understanding of everything is something less than awesome :)
<arosales> mpontillo: rocks, thanks for the help :-)
<arosales> valeech: many thanks for baring with us while we get these beta released
<arosales> maas 2.0 and juju 2.0 GA are going to be a solid pair
<valeech> arosales:  no worries at all. thanks for the quick responses to questions and excellent support!
<bdx> or
<mup> Bug #1595339 opened: Boot failed on enlist <MAAS:New> <https://launchpad.net/bugs/1595339>
#maas 2016-06-23
<mup> Bug #1595379 opened: 2.0 beta7 : having to recommission some nodes multiple times before they're finally ready  <oil> <MAAS:New> <https://launchpad.net/bugs/1595379>
<mup> Bug #1595381 opened: 2.0 beta7: inconsistent naming of the network interfaces -  en0X and renameX - after commissioning <oil> <MAAS:New> <https://launchpad.net/bugs/1595381>
<shewless> Hi there. Is there a way to transfer ownership of a node in MAAS 2.0
<shewless> I don't want to have to destroy and recreate
<shewless> I'm going to update the database... maybe change the maasserver_node table.. I hope that won't explode!
<shewless> it looks like updating the owner_id column in the maasserver_node table did the trick.. so far anyways
<mup> Bug #1595381 changed: 2.0 beta7: inconsistent naming of the network interfaces -  en0X and renameX - after commissioning <oil> <MAAS:Won't Fix> <https://launchpad.net/bugs/1595381>
<LiftedKilt> let's have a go at this again - MAAS 2.0 isn't detecting or configuring the IPMI settings of my nodes during enlistment, so nothing will commission. I did not have any problems with this in 1.9
<LiftedKilt> any ideas of where to look to resolve this?
<mup> Bug #1595633 opened: [2.0RC1] Commissioing/deploying OS doesn't configure the default domain to search <MAAS:Triaged> <https://launchpad.net/bugs/1595633>
<andrewglass3> evening all
<andrewglass3> Guys I need your help please?  Id like to use MAAS in my datacentre however I cant see if it supports Debian 8? We are a mixed Ubuntu/Debian/Centos house.....can I use Debian 8 with this? Appreciate any assistance :) Thanks
<mpontillo> LiftedKilt: are you using the latest version of the MAAS 2.0 beta? (beta 8?) can you take a look at the system console during enlistment to get an idea where it might be going wrong?
<mpontillo> LiftedKilt: is MAAS configured to serve DHCP, and can is the MAAS region IP address reachable via the addresses and gateway handed out by DHCP on the network the nodes are enlisting on?
<newell_> LiftedKilt: I suspect that you don't have MAAS configured to serve DHCP (this must be reset on upgrade/install)
 * newell_ was just informed that we try and make it so you don't have to do anything on upgrade .... i.e. same settings
<LiftedKilt> mpontillo, newell_: I am serving dhcp - the nodes are grabbing IP's on pxe boot
<LiftedKilt> I am using beta7 - I will upgrade to beta 8 now
<newell_> LiftedKilt: Was this an upgrade or a fresh install?
<LiftedKilt> newell_: fresh install
<LiftedKilt> upgrade to beta8 in progress
<LiftedKilt> deleted the failed nodes and will re-enlist
<mpontillo> LiftedKilt: just to confirm, can the enlisted nodes hit the MAAS region IP with the route they get from DHCP? I had a similar problem in a test environment before, where I was using two NAT networks, and the nodes couldn't reach the region
<mpontillo> LiftedKilt: the workaround in that case is to "dpkg-reconfigure -plow maas-rack-controller" and ensure the region URL is set to an IP address the nodes can reach.
<LiftedKilt> mpontillo: the region/rack controller (same box for now) is on the same subnet as the nodes
<mpontillo> LiftedKilt: yes, that was the case for me as well, but the region+rack was multihomed and the nodes were trying to contact the wrong IP. (anyway, I won't go down that rat hole any more for now)
<mpontillo> LiftedKilt: can you keep an eye on the system consoles during enlistment to see if the enlistment scripts report any errors?
<LiftedKilt> took a video of the whole enlistment process
<mpontillo> LiftedKilt: oh, awesome, thanks.
<LiftedKilt> scrolling through now looking for errors
<LiftedKilt> first error I see is "ubuntu pollinate[915]: Network Communication Failed * Hostname was NOT found in DNS cache
<LiftedKilt> this looks more promising - FATAL: Module ipmi_ssif not found
<LiftedKilt> no such file or directory : ipmitool
<LiftedKilt> mpontillo: https://www.youtube.com/watch?v=tzE_mVyyEq4
<LiftedKilt> the error in question flashes for just a split second at 2:15
<newell_> LiftedKilt: try installing ipmitool
<newell_> but....you should have had this installed before right?...hmm
<newell_> watching video now....
<mpontillo> LiftedKilt: at https://youtu.be/tzE_mVyyEq4?t=146 I see there are some errors from apt, reporting that the IPMI package could not be authenticated and was not installed. are you using a local mirror? (if so, are you sure it is updated?)
<LiftedKilt> mpontillo: not using a local mirror
<LiftedKilt> newell_: I installed impitool on the maas server
<LiftedKilt> re-enlisted, same error
<mpontillo> LiftedKilt: newell_: ipmitool is installed on the enlisting node in the ephemeral environment as well, not just the server. that seems to be where the failure is
<newell_> LiftedKilt: yeah for soem reason in your enlistment environment it is not getting installed
<mpontillo> LiftedKilt: I also noticed you're using trusty for enlistment; do you have the Xenial images synced as well? could you try enlistment/commissioning using xenial? (check that it is selected on the settings page)
<LiftedKilt> mpontillo: happy to - I was using xenial yesterday and jhegge recommended I try trusty
<LiftedKilt> switched to xenial - I'll go try again
<mpontillo> LiftedKilt: ah, okay, were there issues with xenial? guess I should go read the scrollback =)
<mpontillo> LiftedKilt: I would say that for MAAS 2.0, xenial has had much more runtime under test. the reverse is true for MAAS 1.x.
<mpontillo> LiftedKilt: if enlistment doesn't work with xenial, the thing I would do next is edit /etc/maas/preseeds/enlist_userdata and change the script near the bottom so that you can log in and figure out why the IPMI packages aren't authenticated.
<mpontillo> LiftedKilt: just be careful because you're getting into dangerous territory now ;-)
<LiftedKilt> mpontillo: on Xenial it hangs on "Temporary failure resolving" for each of the ubuntu archives
<newell_> LiftedKilt: is this during enlistment that you see this?
<LiftedKilt> newell_: yes
<LiftedKilt> it's like DNS isn't set
<LiftedKilt> but it is on the subnet in MAAS
<newell_> LiftedKilt: can you do to the MAAS UI and check the controllers page
<newell_> click on the controllers and make sure that all the services have a green icon
<LiftedKilt> newell_: everything is green
<mpontillo> LiftedKilt: that's troubling. it seems like there is an issue with the Ubuntu archive, for both releases. so - you said no local mirror - do you have the default archive URL configured?
<LiftedKilt> mpontillo: http://archive.ubuntu.com/ubuntu
<LiftedKilt> both main and ports are set to the default values
<LiftedKilt> oh wait...
<mpontillo> LiftedKilt: newell_'s question reminded me... I think maybe we're using MAAS as a default DNS server whereas we weren't before. can you check that your upstream DNS is correctly configured on the settings page? check the DNSSEC setting too; it's common for some DNS forwarders to not properly handle DNSSEC, so you have to disable it
<LiftedKilt> I have dns configured in the network settings, but nothing in the Upstream DNS settings on the main maas settings page
<LiftedKilt> let me re-enlist and see if that fixes it. I'm going to be so annoyed with myself if it was that simple
<newell_> LiftedKilt: I suspect that it will be :)
 * newell_ has been there
<LiftedKilt> nope
<LiftedKilt> same problem
<LiftedKilt> disabling dnssec as per mpontillo and trying again
<LiftedKilt> nope. still nothing
<LiftedKilt> still hanging at the same point
<newell_> LiftedKilt: can you authenticate packages on MAAS server?  I presume so since you upgraded etc
<LiftedKilt> let's just pretend this never happened :|
<LiftedKilt> turns out if you are in a /21 and you try to create a /24 in maas, you can't just arbitrarily create a default gateway that doesn't exist
<newell_> LiftedKilt: ha, glad you found the reason :)
<LiftedKilt> newell_: mpontillo you both were more than helpful - if you are ever in the LA area I owe you a beer
<newell_> LiftedKilt: no worries :)
<mpontillo> LiftedKilt: hah, glad you got it working. you had me worried ;-)
<LiftedKilt> if it comes down to a problem in the software or a problem in my implementation, always bet on the software -_-
<valeech> Does maas 2.0 beta 9 have any issues deploying a machine with multiple nics and setting them up properly? I am trying to create 2 bonds, add vlan interfaces to those bonds and then assign IP addresses out of the subnets.
<valeech> While the machine boots and finishes deploying but it does get hung on waiting for network configuration. Once its up I can ssh in and see that one of the addresses on the first bond worked but nothing on the second bond is active. and the way the bonds are setup looks a little weird to me.
<mpontillo> valeech: we would need the results of "maas <profile> machine get-curtin-config <system-id>" and the contents of /etc/network/interfaces on the deployed node to debug. are there any errors in syslog?
<valeech> mpontillo: I will get those and see if there are any errors.
<mpontillo> valeech: it would also be good if you could explain in what way they're "a little weird", so we can understand the expected vs. actual result
<mpontillo> valeech: I did fix an issue with "weird bond configurations" in curtin, which I do not believe has been released yet. https://code.launchpad.net/~mpontillo/curtin/fix-improper-bond-parameters-bug-1588547/+merge/296469
<mpontillo> valeech: that only occurred if you had a bond with an alias interface. there was a similar issue with IPv6 that was fixed at the same time
<valeech> mpontillo: I am deploying the machine now (again) so I can get that info.
<mpontillo> valeech: ah, well, you shouldn't have had to redeploy if it was already deployed, but thanks
<valeech> mpontillo: I was messing around with some other stuff and used that machine.
<valeech> mpontillo: Here is the output from get-curtin-config, interfaces file and an ip link command: http://pastebin.com/E5W2nFny
<valeech> in addition to the network issues, the system is not showing 2 NVMe drives that I ahve installed in the system with fdisk. I can parted them and add partitions and mount them but they will not show up in fdisk.
<valeech> checking syslog for errors...
<mpontillo> valeech: from the pastebin, I see that commissioning has discovered nvme0n1 and nvme1n1, and you have 3 bonds configured from the six physical interfaces. I'm not seeing anything obviously incorrect yet. what is different from your expectation?
<valeech> mpontillo: syslog output for eth4, eth5 which make up bond10 : http://pastebin.com/zCeV9PBD
<mpontillo> valeech: so syslog saying "bonding: bond10: Warning: No 802.3ad response from the link partner for any adapters in the bond" seems to indicate that the bond is not configured as a LACP bond on the switch side
<mpontillo> valeech: the other thing that concerns me is "8021q: VLANs not supported on bond10"
<mpontillo> valeech: seems the NIC doesn't support VLANs? O.o
<valeech> mpontillo: I saw that. They do support vlans, so I am not sure whats up with that.
<valeech> mpontillo: double checking switch config...
<mpontillo> valeech: looks suspiciously similar to https://bugs.launchpad.net/ubuntu/+source/ifenslave-2.6/+bug/881379
<mpontillo> valeech: it seems to *maybe* correct itself later in the boot process; not sure if those log messages are red herrings.
<valeech> mpontillo: yeah, the last lines in the syslog seem to show that the bonding links came up
<valeech> mpontillo: Ok, just physically verified what switch ports this host is connected to. I verified the switch is configured properly and the switch shows the LACP channel is up on both interfaces.
<mpontillo> valeech: ok, great. so maybe just a bit of a race condition in ifenslave/ifupdown/systemd that sorts itself out eventually..?
<valeech> mpontillo: output of the bond interface: http://pastebin.com/UJtRH7zu
<valeech> mpontillo: perhaps, but the vlan interfaces never come up.
<valeech> mpontillo: as you can see in the ip link status
<valeech> and what are these?? br-bond10.400    br-bond10.450
<mpontillo> valeech: was this a node involved in a juju bootstrap? guessing juju put that there
<mpontillo> valeech: MAAS does not configure bridges in this release.
<valeech> mpontillo: it may have been as I was working through some testing. Doesnât maas put on a whole new OS though at deployment?
<mpontillo> valeech: MAAS 2.0 would, by default, enlist and commission with a Xenial image. this image is ephemeral (booted via iSCSI, nothing written to disk) -- when you deploy the node, at that time we'll install whatever the user selects (generally xenial by default in MAAS 2.0, but juju may have requested trusty?)
<valeech> mpontillo: yes, I have juju set to use trusty as the default series.
<valeech> mpontillo: and this node is fullt deployed according to maas. so there shouldnât be any left over artifacts from an old juju bootstrap, right? the box should now have a new os from maas.
<valeech> fully*
<valeech> mpontillo: and those br-bond10 interfaces have the IPs that are configured in MAAS for the machine in the interfaces file.
<mpontillo> valeech: that is very strange. it looks like the OS that booted is not the OS that MAAS was instructed to deploy. otherwise you would not have that bond interface
<valeech> mpontillo: interestingâ¦
<mpontillo> valeech: is the currently-deployed machine Xenial or Trusty?
<valeech> mpontillo: trusty
<mpontillo> valeech: can you try redeploying with Xenial?
<valeech> mpontillo: I was just typing that :)
<mpontillo> valeech: maybe this is related to your missing storage too, in that an [older] trusty kernel might not support those devices. (you could try specifying a hardware enablement kernel, such as hwe-x)
<valeech> mpontillo: before I put these machines (thereâs is 8 of them) in my maas/juju pool, they were production running trusty and it did see the nvme drives thenâ¦
<mpontillo> valeech: perhaps with the newer kernel?
<valeech> mpontillo: thatâs possible, yes
<valeech> mpontillo: ugh, these boxes are soooo slow to boot :)
<mpontillo> valeech: the price you pay for ECC memory, I guess...
<valeech> mpontillo: haha very true
<valeech> mpontillo: just to be clear on what I am doing. I set the minimum kernel version for commissioning in maas to hwe-x. I then recommissioned the node and now I am having juju deploy the machine using xenial.
<mpontillo> valeech: well, hwe-x will be needed for deployment as well as commissioning. let me check on my test MAAS
<valeech> mpontillo: I didnât see a spot to set it for deploy in the gui. Is that a cli only thing right now?
<mpontillo> valeech: if you click "Edit" inside the "Machine summary" on the node details page, you should be able to set it per-node
 * mpontillo guesses it would be nice to update that to whatever the version used to commission was, if it was greater
<valeech> mpontillo: got it!
<valeech> mpontillo: ok, it gets hung deployment. It installs and then does the final reboot then just sits at the login prompt. On a trusty deployment I would normally see it apply ssh keys and do some other stuff. It hasnât done that. And there is no way for me to get into the system to see the cloud-init log to see why it has died.
<mpontillo> valeech: check /var/log/maas/rsyslog/<host>
<valeech> mpontillo: not sure what I am looking for but I see this: Jun 23 23:16:25 overrighteous-concetta cloud-init[1970]: Installation finished. No error reported.
<valeech> mpontillo: I donât see any other errors
<mpontillo> valeech: hmmm.. I just realized something. if you are deploying xenial there is no need to set a minimum kernel. because xenial already comes with the xenial kernel. ;-)
<valeech> mpontillo: that makes sense.
<mpontillo> valeech: so I think the valid choices are "trusty with hwe-x as minimum kernel" and "xenial with no minimum kernel". If you set a minimum kernel for Xenial, it might not work, since there are no hardware enablement kernels for xenial yet
<valeech> mpontillo: since I can get trusty to work, let me try it with trusty and hwe-x
<mpontillo> valeech: I would scroll up in the rsyslog output to see if there are any other clues, but yeah, that sounds like a good plan
<mpontillo> valeech: once you're back up and running, my other question regarding the VLAN interfaces would be, once the node is deployed, are you able to do "ifup <interface>" to bring the missing VLAN up?
<mpontillo> valeech: in other words, is it a race condition in the config? maybe there is something we could do when we write /etc/network/interfaces to work around the problem.
<valeech> mpontillo: I think I tried that and it didnât work, but I will give it a go.
<valeech> mpontillo: That did it! both bond interfaces are up and the nvme drives are now showing up.
<mpontillo> valeech: very good. thanks for confirming
<valeech> mpontillo: the only thing that is a problem, is the MTU is not set on those 2 bond interface VLANs
<mpontillo> valeech: MTU in Linux is an attribute of a physical interface. you cannot set it on a VLAN without setting it on the parent
<valeech> mpontillo: Understood. the mtu is set on the bond10.400 and bond10.450 interfaces but not on the physicals eth4 or eth5.
<valeech> mpontillo: does maas not set the physical interface mtu?
<mpontillo> (if you think about it, if I have eth0 with MTU 1500, I can't really have eth0.100 with an MTU of 9000. because I can't get more than 1500 bytes out on eth0, and ultimately the traffic is exiting eth0)
<mpontillo> valeech: we should be able to set it
<mpontillo> valeech: I think right now MAAS allows it to be inconsistent, which might not end well.
<valeech> mpontillo: There needs to be a place in the GUI to set the physical interface mtu.
<mpontillo> valeech: agreed, either that or setting it on the child should automatically update the parents
<mpontillo> valeech: could you file that as a bug?
<valeech> mpontillo: exactly. I can. To be honest, I donât know how to file a bug, but I will figure it out :)
<mpontillo> valeech: https://bugs.launchpad.net/maas/+filebug
<valeech> mpontillo: there it is! :) thanks!
<mpontillo> np, thanks for testing MAAS 2.0 beta ;-)
<valeech> mpontillo: and thanks for all of the help! I really really appreciate it
<mpontillo> valeech: it's a good distraction from the spec I'm supposed to be writing for the next release, lol ;-)
<mpontillo> (but seriously, happy to help.)
<valeech> mpontillo: haha! well I have plenty of questions :)
#maas 2016-06-24
<mup> Bug #1595753 opened: [beta8] HMC power driver regression -- Not able to connect via SSH. <MAAS:In Progress by newell-jensen> <https://launchpad.net/bugs/1595753>
<valeech> mpontillo: bug opened: https://bugs.launchpad.net/maas/+bug/1595755
<mup> Bug #1595755 opened: MTU does not get set on bonded interfaces <MAAS:New> <https://launchpad.net/bugs/1595755>
<mpontillo> valeech: thanks.
<mup> Bug #1595755 changed: [2.0b8] MTU does not get set on bonded interfaces <MAAS:New> <https://launchpad.net/bugs/1595755>
<mup> Bug #1595755 opened: [2.0b8] MTU does not get set on bonded interfaces <MAAS:New> <https://launchpad.net/bugs/1595755>
<mup> Bug #1595996 opened: Create bond interface in the official documentation <MAAS:New> <https://launchpad.net/bugs/1595996>
<mup> Bug #1595996 changed: Create bond interface in the official documentation <MAAS:New> <https://launchpad.net/bugs/1595996>
<mup> Bug #1595996 opened: Create bond interface in the official documentation <MAAS:New> <https://launchpad.net/bugs/1595996>
<mup> Bug #1596018 opened: (doc) missing instructions on how to enable SSL support <MAAS:New> <https://launchpad.net/bugs/1596018>
<newell_> blakeh: ping
<mup> Bug #1596046 opened: maas 2.0 pxeboot fails on power 8 <MAAS:New> <https://launchpad.net/bugs/1596046>
<newell_> blakeh: you around?
<blakeh> here
<newell_> Have you downloaded images for your MAAS 2.0 server?
<blakeh> yes
<blakeh> ppc and amd
<newell_> blakeh: and you have DHCP configured for this MAAS server?
<newell_> blakeh: from the bug report it seems like you are deploying a 2nd MAAS server on one of the MAAS 1.9 nodes that you deployed?
<blakeh> newell - yes
<blakeh> newell - i have the first one shut donw
<newell_> Or is this an isolated MAAS 2.0 installation (from MAAS 1.9)
<newell_> ?
<blakeh> newell  -- i have a 14.04  with maas 1.9 that is currently powered off on same subnet-
<blakeh> newell - to prevent dhcp problems
<newell_> blakeh: okay, just wanted to make sure
<blakeh> newell -- deploying an x86 node works fine
<newell_> when you try and boot your power nodes with MAAS 1.9 is it serving pxelinux.0 as well?
<blakeh> newell - yes- no issue
<newell_> blakeh: okay power should be getting served something other than pxelinux.0
<newell_> so that is concerning
<blakeh> newell - i think pxelinux.0 is correct for ppc64el--- ppc64 is served by boot ppc64.bin
<blakeh> newell -- lookin in /var/lib/maas/dhcpd.conf seems to indicate this is correct - on both 1.9 and 2.0
<newell_> blakeh: ah maybe I am thinking powerVM
<newell_> what type of power is this?
<newell_> powerNM
<newell_> NV*
<blakeh> yes nv
<newell_> okay that is why
<newell_> thanks for verifying
<blakeh> power 822L
<newell_> are you using trusty images or xenial?
<newell_> or something else?
<blakeh> did put in a bug
<blakeh> xenial
<newell_> let me ping the powerNV guy and see if he has tried this with MAAS 2.0
<blakeh> sure -- Bug #1596046 has some logs attached
 * newell_ looking now
<newell_> blakeh: so this is simply manually turning the power on for the machine correct?
<newell_> blakeh: have you tried entering the power parameters and trying to commission the node directly?
<blakeh> yes -- power on and do a pxe boot
<blakeh> have not tried entering manually yet
<newell_> blakeh: try that for now
<blakeh> ok - will take a few minutes
<wililupy> crazy question here...Is there a way to get MAAS's DNS server to sync with a regular bind9 DNS server?
<blakeh> newell:  so at this point when i select commission for the power 8 node it does power it on an bring it up to petitboot, but does not load a new os on sda - hangs on commissioning- i expect it to time out and error.
<newell_> blakeh: so it was able to find the pxelinux.cfg unlike before?
<blakeh> no -- it was unable to install an os -- no pxe boot  ---
<newell_> blakeh: no OS is installed during commissioning
<blakeh> perhaps i picked the wrong action >  i selected commission- should it be acquire?
<newell_> commissioning is what you want
<newell_> do you have console log?
<blakeh> correct-- i am trying again-- net info was not correct last time
<newell_> ack
<newell_> wililupy: not sure.  mpontillo any idea if we can get MAAS DNS server to sync with regular bind9 DNS server?
<mpontillo> wililupy: newell_: not sure what is meant by "sync", but I understand that a common configuration is to delegate a subdomain to MAAS
<wililupy> Reason for asking is that I don't want to have to continue to have to manually add the servers provisioned by maas manually in my DNS server.
<mpontillo> MAAS only partially takes control of the bind9 configuration. it's likely that you could configure bind9 however you like, and MAAS will mostly stay out of your way
<mpontillo> wililupy: what I do for the MAAS I test with is I tell my local dnsmasq that my MAAS DNS server is authoritative for the .maas (or whatever) domain
<mpontillo> wililupy: in short, yes, it should be possible. alternatively you could just rsync the zone files MAAS generates to your other bind9 server
<mpontillo> wililupy: there are any number of ways you could tackle this problem
<wililupy> mpontillo: Gotcha. That makes sense. I could add a subdomain in my DNS server such as maas.domain.net and have it be the authority for that.
<wililupy> it being MAAS
<mpontillo> wililupy: yeah, just update the domain in MAAS to be whatever your chosen subdomain is
<blakeh> newell -- so it loaded the os on the hard drive- ub 14.04 instead of installing the commissioning os - u 16.04
<bdx> hey whats up all? Is there a reason why a space can't be assigned to a fabric with no subnets defined?
<sjl> how does one log into a failed centos deploy?  the iDRAC console shows login with 'localhost login' and my normal ssh key doesn't apply since the node instance apparently did not successfully get on the network
<pacavaca_> 3
<pacavaca_> oops, accidentally pressed enter
<pacavaca_> Is there anybody around to answer some questions?
<mpontillo> sjl: did the machine boot into CentOS? IIRC, since MAAS don't do advanced network configuration on the CentOS images, you must use DHCP. not sure if that helps.
<mpontillo> sjl: normally I would say that you may able to modify the cloud-init user data (see /etc/maas) to configure a password, but it sounds like your deployed nodes aren't able to reach the MAAS metadata server anyway
<mpontillo> sjl: did you try an Ubuntu deployment? if so, any issues?
<mpontillo> pacavaca_: just ask your question; if someone knows the answer they will jump in
<sjl> mpontillo: the problem is intermittent with about 20% failure rate with centos,  I haven't seen any errors with ubuntu but haven't tried that much to be sure
<mpontillo> sjl: so it works one out of every ~5 times -- on the same machine, or different machines? it would be good to isolate why it fails, when it does...
<sjl> mponntillo: yes the machine did boot to centos, so it was able to communicate to maas at some stage
<mpontillo> sjl: if you are using MAAS 2.0 you might check /var/log/maas/rsyslog/<node> to see if there are any clues. cloud-init should configure the MAAS server as an rsyslog endpoint
<sjl> mpontillo: same machine
<mpontillo> sjl: though I don't think that rsyslog feature persists after deployment is done
<sjl> i'm using the standard centos66 from http://images.maas.io/ephemeral-v2/daily/  Is there a preseed user account ('root' or 'centos') with a static password to get me in?
<mpontillo> sjl: no, there are no static passwords. if cloud-init is failing to get the machine's configuration, I doubt that an additional cloud-init preseed with a static password would help you
<pacavaca_> maas2: on the controller machine I've changed some network configuration. Basically I broke a bridge and gave the IP back to the physical NIC. Now how can I tell maas to re-detect the network configuration on controller? And actually, why can't I run DHCP on a bridge?
<mpontillo> pacavaca_: if you are on MAAS 2.0 it should auto-detect within ~30 seconds
<mpontillo> pacavaca_: DHCP on a bridge should work fine. which version of MAAS 2.0 are you running? if you didn't use the PPA, you probably have beta 3. if that is the case, you want to get the latest beta from ppa:maas/next
<sjl> how about a backdoor method into centos66?  I've tried without success: https://maas.ubuntu.com/docs/troubleshooting.html?highlight=backdoor#debugging-ephemeral-image
<pacavaca_> mpontillo: hmm. it doesn't seem to happen http://d.pr/i/17W8y I'm on beta8 (from ppa). Should I look into some logs? I haven't found a way to enable DHCP on a bridge through UI..
<mpontillo> pacavaca_: though I am not sure what you mean by "broke a bridge and gave the IP back to the physical NIC"; if you have a bridge configured, the IP does need to be on the bridge.
<pacavaca_> mpontillo: it was. but now I removed that bridge
<pacavaca_> Uh oh. Seems it has resolved. I did not remove a bridge but just brought it down and brought the NIC back up and it didn't work. But now I did 'brctl delbr br0' and MAAS detected the new config right away.
<pacavaca_> So, false alarm. Thank you for the attention mpontillo!
<mpontillo> pacavaca_: no problem, glad it's working ;-)
<mpontillo> pacavaca_: DHCP is enabled per-VLAN in MAAS 2.0, not per interface. this change was made so that we can support HA. you'll find the DHCP on/off options on the VLAN details page under Networks. (a but buried, I admit.)
<pacavaca_> mpontillo: yeah, just found it.. seems my bridge got destroyed for nothing :)
<bdx> general note: when enlisting virsh instances, ensure you don't configure them to '--autostart' :-)
<pacavaca_> mpontillo: how does MAAS figure out when node is deployed? I just tried using my custom image and the machine got re-imaged correctly, I can log in and use it. However, MAAS still shows it as deploying. Should there be some maas agent running on machine or what's the mechanism?
<mpontillo> pacavaca_: we consider MAAS to be agentless. but cloud-init acts as our agent, and I *think* (but would have to verify) that we can tell the machine is deployed due to the notification from cloud-init
<pacavaca_> mpontillo: ok, thanks. Is there any way I can verify cloud-init is functionalble? Because this is a custom image I've created in some hacky way and things may be broken.
<mpontillo> pacavaca_: well, I confirmed that the cloud-init request for user-data is what hands the node off to deployed state. does your custom image include cloud-init?
<pacavaca_> It should, but how do I verify that?
<bdx> :-)
<bdx> pacavaca_: why/how should it?
<mpontillo> pacavaca_: well, besides the obvious, I might do a packet capture from the rack controller during deployment and compare what happens for a supported image to yours.
<pacavaca_> I initially created the image using curtinator and I assume it installed cloud-init. But then I did some modifications to the contents of tarball, so It might broke something
<pacavaca_> that's to bdx :)
<mpontillo> pacavaca_: okay. well, I would suggest testing with the official Ubuntu image to see if you can isolate the problem. I can't dig too deeply into custom images since that's something Canonical sells support for, as it can be a bit involved to get it right, as you see =)
<pacavaca_> mpontillo: sure, I understand that. Just looking for some hints to simplify debugging. Ok, I will try using the official image just to see if it actually works.
<f1gjam> hey guys, I am trying to install openstack via autopilot, but juju fails to boot strap a node - here is the log file http://paste.ubuntu.com/17779796/
<f1gjam> any ideas?
<bdx> flgjam: ask in #ubuntu-solutions
<bdx> things are looking sweet you guys!
<bdx> just gave the experimental3 a go
<bdx> beautiful
<bdx> http://imghub.org/image/nKE0
<bdx> http://imghub.org/image/nccF
<bdx> http://imghub.org/image/n9ZB
<bdx> http://imghub.org/image/nuMw
<bdx> that is a beautiful thing
<bdx> thanks!
<bdx> you guys are killlling it!
#maas 2016-06-25
<f1gjam> ok ill ask there
<mpontillo> bdx: looks great; glad MAAS is working well for you =)
<f1gjam> hey guys i tried #ubuntu-solutions - no joy
<f1gjam> also trying in ubuntu-server
<mpontillo> f1gjam: the traceback is saying juju tried to deploy a MAAS node but it failed to change state to "Deployed". Can you try using MAAS to deploy a node manually and see if it works? Which version of MAAS did you install?
<f1gjam> Maas version 1.9
<f1gjam> MAAS Version 1.9.3+bzr4577-0ubuntu1 (trusty1)
<f1gjam> going to deploy OS now
<f1gjam> so the machine looks like its done
<f1gjam> but MaaS is saying deploying still
<f1gjam> is there a log i can follow to see what MaaS is waiting on?
<mpontillo> f1gjam: hmm, that may indicate a lack of connectivity to the MAAS server from the deployed node
<mpontillo> f1gjam: is there an installation log on the node details page? (it would be in the drop-down box alongside the commissioning results and summary)
<mpontillo> f1gjam: are you able to SSH to the machine and check the cloud-init logs?
<f1gjam> ok
<f1gjam> so its finsihed but still saying deploying
<f1gjam> do you want commissioning oupiut
<bdx> sure would be cool if dns was modifiable from the gui .....
<bdx> i guess it wouldn't really make sense though
<bdx> pushing for container hostname specification capibility ... I guess that would be on the juju side of things though eh?
<f1gjam> ok i am going to sleep - will try again in the morning
<bdx> f1gjam: yea, the more logs the better
<bdx> you can easily get them shareable with the tool 'pastebinit'
<bdx> flgjam: e.g. `cat /var/log/apache2/error.log | pastebinit`
<bdx> flgjam: would return a url with your log
<bdx> flgjam: you should try to compile a list of logs, and paste them on #ubuntu-solutions
<bdx> flgjam: if no one is around, they will answer you when they come back
<mpontillo> f1gjam: if there is commissioning output, that's good, but I was more interested in if there was an install log
<wililupy> Using MAAS 2.0 with an external DHCP server, I keep running into deployment issues. The servers start up, they get an IP, but when they try to go online to setup, they try an APIPA address and then default back to my gateway's IP address, but never successfully commision. Anyone else expirience this?
<wililupy> If I have MAAS act as DHCP/DNS server, I can commission, but I don't want to have to create a seperate routeable network segment if I don't have to.
<f1gjam> hey mpontillo - Are you around. The installation of the OS failed. There isnt an install available either.
<f1gjam> this is all i get:  [ERROR] OS-LEN1: Marking node failed: Node operation 'Deploying' timed out after 0:40:00.
<f1gjam> is there a proper up to date guide which tells you how to setup MaaS step by step?
<mpontillo> f1gjam: the docs linked off of http://maas.io/ should be relatively up to date.  can you take a look at the system console during the deployment to see if there are any errors? is there an installation log available in the node details page?
<mpontillo> f1gjam: you might also check /var/log/maas/rsyslog/<node> for clues
 * mpontillo won't be watching IRC closely, as it's the weekend, just so you know
<f1gjam> hey mpontillo
<f1gjam> ok going to check now
<f1gjam> let me re-provision
<f1gjam> since i released all the nodes
<f1gjam> mpontillo, appreciate any help however slow it is :)
<f1gjam> one question
<f1gjam> do i need to do any iptables configuration
<f1gjam> my networks are - 192.168.10.0/24 (External) - 192.168.20.0/24 (Internal)
<f1gjam> when i look at the system being provisioned it seems to provision all the way.... albeit i cant logon ...
<mpontillo> f1gjam: well, the deployed node must be able to access the MAAS server (to fetch SSH keys and send the indication that the node is now "Deployed"). Not sure about iptables; depends on your setup, but you'll want to verify that if a host is configured on that network it is able to reach MAAS
<f1gjam> its installed
<f1gjam> currenlty at
<f1gjam> Source MaaS http://192.168.10.1/MAAS/metadata/curtin
<mpontillo> f1gjam: often times if there is a firewall or NAT involved and the deployed node cannot reach MAAS, what you need to do is "sudo dpkg-reconfigure -plow maas-rack-controller" and make sure the URL to the MAAS region is set to one that will be able to be reached from the network the nodes are deployed on
<f1gjam> rsyslog is empty
<f1gjam> ok - so my network setup is as follows:
<mpontillo> f1gjam: so what I would do is make sure that the 192.168.20.x IP address is being used for the MAAS URL. it's possible that the deployment network cannot reach the "external" IP you are using for the MAAS region?
<f1gjam> 192.168.10.0/24 - MAAS IP 192.168.10.1 / GATEWAY = 192.168.10.254 (REAL FIREWALL/ROUTER) -This is my external network
<f1gjam> 192.168.20.0/24 - MAAS given adress of  192.168.20.1 on interface eth1
<mpontillo> f1gjam: also make sure that all the proper subnet gateways are configured in MAAS (check the details page for each subnet under "Networks" in MAAS)
<f1gjam> thats one quesion
<f1gjam> so lets say external network 192.168.10.1
<f1gjam> has the correct gatwway etc...
<f1gjam> but in 192.168.20.1
<mpontillo> f1gjam: okay, and how are the deployed nodes expected to get off-subnet? what is their router supposed to be? the MAAS server?
<f1gjam> MAAS
<f1gjam> 192.168.20.1
<mpontillo> f1gjam: okay then yes you'll need to ensure routing is enabled on the MAAS server and whatever iptables rules necessary are in place
<f1gjam> why?
<mpontillo> f1gjam: my guess is that the deployed nodes are trying to hit MAAS on the external IP address and it isn't working due to routing issues
<f1gjam> I mean the document statues to set
<mpontillo> f1gjam: alternatively you can do the dpkg-reconfigure command I suggested earlier to make sure the deployed nodes hit MAAS on the internal network interface
<f1gjam> and change the MaaS IP?
<f1gjam> to the internal address?
<mpontillo> f1gjam: yes
<f1gjam> https://screencloud.net/v/lVjd
<f1gjam> https://screencloud.net/v/xzDb
<f1gjam> https://screencloud.net/v/377
<f1gjam> brb
<f1gjam> if i change the router ip to the REAL router for the .20 netowkr
<f1gjam> i get
<f1gjam> calling 192.168.20.254//latest/meta--data/instance-id sp im guessing it has to be the address of the MaaS server
<mpontillo> f1gjam: ah, you are on MAAS 1.9? for some reason I was assuming you were on 2.0
<mpontillo> f1gjam: some of my instructions were wrong then. "sudo dpkg-reconfigure -plow maas-cluster-controller" (not rack-controller)
<mpontillo> f1gjam: did you send a screenshot of the configuration of the eth0 interface?
<mpontillo> f1gjam: there is no requirement that the router IP is the MAAS server. you can define the router IP on the cluster interface settings page
<f1gjam> im back sorry was afk
<f1gjam> the problem is
<f1gjam> if i set the GW IP of eth0 which is the private interface to 192.168.20.254
<f1gjam> it those that error about
<f1gjam> https://screencloud.net/v/3m9g <-- eth0 configuration (private network)
<f1gjam> https://screencloud.net/v/64N
<f1gjam> if i put the gateway as
<f1gjam> 192.168.20.254
<f1gjam> I get that issue
<mpontillo> f1gjam: oh, that's interesting. are you redirecting MAAS to an SSL-enabled site?
<mpontillo> f1gjam: I would try running "dpkg-reconfigure -plow maas-cluster-controller" and make sure the MAAS URL is set to http://192.168.20.1:5240/MAAS/
<mpontillo> f1gjam: that way you'll ensure you are talking directly to MAAS without any thing in between
<f1gjam> ok
<f1gjam> but..
<f1gjam> if you look from that pick thats when i point it to the router as the default GW for the 192.168.20.x network
<f1gjam> that issue doesnt come up
<f1gjam> if i make the gateway 192.168.20.1 = MAAS
<mpontillo> f1gjam: yeah, that is weird and I can't explain that right now (though I've seen it in the past, I wonder if it's some kind of a fallback); I'm pretty sure it should be using the MAAS URL configured on the cluster, not the gateway, to attempt to contact the cloud-init data source
<f1gjam> mpontillo, sorry i disconnected
<f1gjam> back now
<f1gjam> https://screencloud.net/v/QUf
<boxo> Hello
<boxo> i ned more info for mass
#maas 2016-06-26
<mpontillo> f1gjam: ah, is it working now? looks good?
<f1gjam> mpontillo, nope
<f1gjam> i just blew away the isntall doing it again
<f1gjam> i think you were right about change the IP of the MAAS server
<mpontillo> f1gjam: ok, the screenshot seemed to show it properly finding the MAAS data source
<f1gjam> yes
<f1gjam> but it doesnt do anything
<f1gjam> just sits there for 40 mins
<f1gjam> and then MAaS server marks as failed :(
<mpontillo> f1gjam: really weird. are you able to get a packet capture of the deployment?
<f1gjam> how would i do that
<f1gjam> wireshark?
<f1gjam> or soethjing?
<mpontillo> f1gjam: I use a command like: tcpdump -s 0 'port not 22' -n -w deployment.pcap
<mpontillo> on the rack controller
<mpontillo> the 'port not 22' is so it doesn't capture your SSH traffic.
<f1gjam> so isnbt the rack controller and maas server one box now
<mpontillo> that will write a packet capture to 'deployment.pcap'. you'd probably need to be root or add "sudo" first.
<mpontillo> f1gjam: yeah, by default it's the same machine but you can put the rack/cluster controller (we renamed it from cluster to rack in MAAS 2.0) on a separate machine if you want
<f1gjam> ok so its all on one machine right now
<f1gjam> give me two secs
<f1gjam> im just waiting for it to install again
<f1gjam> shouldnt take long
<mpontillo> f1gjam: that's fine. if you get a pcap it'll unfortunately be pretty big, as you'll capture the entire image download (~200 MB or so I think)
<mpontillo> f1gjam: but it may help to debug...
<mpontillo> f1gjam: you can filter by http and see if the deployed node is attempting to contact MAAS. just to verify, which version of MAAS do you have installed?
<f1gjam> no issues with that i can put it on dropbox
<mpontillo> cool
<mpontillo> I might not be able to dig into it until next week though
<f1gjam> sure
<f1gjam> where are you ?? in the world i mean
<mpontillo> california
<f1gjam> h wow
<f1gjam> i go there all the time
<f1gjam> Newark
<f1gjam> well once a quarter for work
<f1gjam> :)
<mpontillo> ah cool, never been there
<f1gjam> well its near SF
<mpontillo> oh, yeah, well I've been in the area. I live near Sacramento but I used to work in SF a couple days per week
<f1gjam> yeah sacremento is far from there
<f1gjam> i havent had a chacne to go there yet
<f1gjam> been LA
<f1gjam> and to
<mpontillo> too hot. I don't recommend it =)
<mpontillo> cheaper housing though
<f1gjam> ssualito
<f1gjam> yes very hot in LA
<f1gjam> but it was a tick in the box
 * mpontillo -> afk (good luck with MAAS, sorry it isn't going more smoothly!)
<f1gjam> ill cathch you next week :)
<f1gjam> mpontillo, i know your gone, but i wanted to let you know it WORKED!!!!
<f1gjam> its 4am here in UK so im going to sleep :) satisfied. thanks for your help. I will create updated docs which might be helpful to post somewhere, if you can recommended where that is. thanks again
<f1gjam> mpontillo, feature request - https://bugs.launchpad.net/maas/+bug/1596287
<mup> Bug #1596287 opened: Feature request Installation tracking via MaaS <MAAS:New> <https://launchpad.net/bugs/1596287>
<f1gjam> mpontillo, almost worked, landscape installed failed. I am now going to try out Ubuntu 16.04 with MaaS 2.0 - since i have the install working im guessing its better to try the latest version
<f1gjam> mpontillo, just incase - this is the network layout - https://screencloud.net/v/jsk7
<f1gjam> updated screenshot: https://screencloud.net/v/v5ms
<mpontillo> f1gjam: as far as I know, landscape does not work with MAAS 2.0 yet =( ... the API in MAAS is different.
<mpontillo> f1gjam: what was the issue with landscape?
<f1gjam> let me paste the error
<f1gjam> http://paste.ubuntu.com/17924168/
<f1gjam> and
<f1gjam> sorry thats the error
<f1gjam> ive now installed 16.04
<f1gjam> and trying to get it to install autopilot
<f1gjam> going to kick off install now
<f1gjam> might go abck to 14.04
<f1gjam> especially if you are saying landscape doesnt work with 2.0
<f1gjam> mpontillo, do you work for canonical?
<f1gjam> quick question, is there a way to point ubuntu cd installer to a http://kickstart file which will just pass all the parameters needed for install? secondly is there an auto generated kickstart file located on the box installed manually?
<mpontillo> f1gjam: yes, I work for Canonical on the MAAS team. ;-) not sure about kickstart (thought that was a RH thing), MAAS used to support a way to install Ubuntu using the Debian installer, but it's much slower
<f1gjam> well, is there anyway to do a similar thing to kickstart :)
<f1gjam> the reason i asked if you worked for canonical was because i wanted to ask about the docs, which all seem to be out of date. It takes too much effort to install everything correctly
<mpontillo> f1gjam: yeah we are planning on improving the docs for the next release... sorry for the inconvenience
<f1gjam> mpontillo, its cool, if i have this working, i can help :)
<mpontillo> f1gjam: are you installing Ubuntu to metal or a VM? If a VM, I would use the cloud image and the local cloud-init provider
<f1gjam> metal
<f1gjam> I have 3 * HP110 G6 severs and 2 * Lenovo TS140 Servers
<f1gjam> all of them have 3 network cards
<f1gjam> 1 * 64GB USB and 1 * 250GB or greater HDD
<mpontillo> f1gjam: thanks; if you find errors in the docs and want to send us some minor patches, I'm sure it would be appreciated.
<f1gjam> well I am going to create a new one from scratch
<f1gjam> There are things which need more explanation
<f1gjam> but that would be easier to discuss on a skype call
<f1gjam> especially the networking pieces
<f1gjam> ok - so just finishing off setting up 14.04 with MaaS for like the 20th time :)
<mpontillo> f1gjam: thanks for any feedback you can provide!
<f1gjam> no worries :)
<f1gjam> next time im out in California
<f1gjam> we should meet up :)
<mpontillo> Let me know - I'm often busy with family so no guarantees.
<f1gjam> yeah no worries, it was supposed to be in July
<f1gjam> but not 100%
<f1gjam> would be great if you could choose which hdd to commission on, however you have to commission for it to detect the drives then recommission
<mpontillo> f1gjam: commissioning does not touch your drive at all. It boots a read only image over iscsi with an overlayfs
<f1gjam> well, how do i get it to use USB - you only get to choose after you have done commissioning
<f1gjam> then when you - recommission it shows as the usb drive with the parition
<mpontillo> f1gjam: you can configure the storage layout any way you want after it commissions. MAAS resets the storage layout to default upon recommissioning (though it can optionally retain it)
<mpontillo> f1gjam: in other words, delete the filesystem and configure the volumes/partitions to your liking after you commission.
<f1gjam> ah i see
<f1gjam> well i just wanted it to use USB whole
<f1gjam> and not touch the slow hd
<f1gjam> as i want that for ceph
<mpontillo> f1gjam: you should be able to configure it to do that.
<f1gjam> yeah, as you said after commisson
<f1gjam> that makes sense. The way i was thinking it worked wasnt the case
<f1gjam> need some more clarification :)
<f1gjam> ill try and write up a list of stuff
<f1gjam> and ping it over
<f1gjam> tht might be better
<f1gjam> im really good at making things idiot proofish
<mpontillo> f1gjam: thanks!
 * mpontillo -> afk
<ccit-spence> Is it ok to use MAAS within a vCenter / vSphere setup?
#maas 2017-06-19
<mup> Bug #1698766 opened: Pod remaining cores can go below zero  <pod> <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1698766>
<mup> Bug #1698767 opened: Pod local storage gets added up even if on the same disk <pod> <MAAS:Triaged> <https://launchpad.net/bugs/1698767>
<pmatulis> ew, robust-maggot
<cnf> so where does one add subnets one doesn't use or have any control over, whatsoever?
<cnf> in maas?
<cnf> on a separate fabric?
<roaksoax> cnf: no, you can add them were they are, and you can mark them as 'unmanaged'
<cnf> roaksoax: but they are not on any of my switches etc
<pmatulis> catbus, did you get the bcache? you need to create a 'cache set' and then a 'bcache'
<cnf> i just need them to add routes to them
<roaksoax> cnf: it doesn't matter
<roaksoax> cnf: you can create a new fabric yes
<catbus> pmatulis: ah, that works! Thanks.
<mup> Bug #1636933 opened: Can change storage of an allocated node <docteam> <MAAS:New> <https://launchpad.net/bugs/1636933>
<mup> Bug #1636933 changed: Can change storage of an allocated node <docteam> <MAAS:New> <https://launchpad.net/bugs/1636933>
<mup> Bug #1636933 opened: Can change storage of an allocated node <docteam> <MAAS:New> <https://launchpad.net/bugs/1636933>
<mup> Bug #1698891 opened: [web UI] non-admin can appear to create logical volume <docteam> <MAAS:New> <https://launchpad.net/bugs/1698891>
<mup> Bug #1698889 opened: maas badly needs dep8 tests that check for django compatibility  <maas (Ubuntu):Triaged> <https://launchpad.net/bugs/1698889>
<mup> Bug #1698895 opened: [web UI] admin can create logical volume for an allocated node <docteam> <MAAS:New> <https://launchpad.net/bugs/1698895>
#maas 2017-06-20
<mup> Bug #1567176 changed: [2.0b1] django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_space_pkey" <MAAS:Invalid> <https://launchpad.net/bugs/1567176>
<mup> Bug #1567176 opened: [2.0b1] django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_space_pkey" <MAAS:Invalid> <https://launchpad.net/bugs/1567176>
<mup> Bug #1567176 changed: [2.0b1] django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_space_pkey" <MAAS:Invalid> <https://launchpad.net/bugs/1567176>
<mup> Bug #1660743 changed: [2.1.3] When commissioning a 'NEW' machine from the Machine details page, it incorrectly shows 'MAAS is not providing DHCP.' message <maas-at-home> <oil> <MAAS:Invalid> <MAAS 2.1:Won't Fix> <https://launchpad.net/bugs/1660743>
<mup> Bug #1681611 changed: Page must be refreshed when scripts updated via the API/CLI <MAAS:Fix Released by ltrager> <https://launchpad.net/bugs/1681611>
<xygnal> does PXE mode auto-clear when a node reaches deployed state?I noticed a problem with my curtin late command to turn off PXE, and only just fixed it now.
<xygnal> need to understand if I need to go through those other nodes and manually clear that to avoid re-builds
<mup> Bug #1699222 opened: [2.2] Updateing virsh power parameters for one node affects other node <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1699222>
<roaksoax_> xygnal: howdy! maas only tells the BMC to PXE boot on next boot
<roaksoax_> xygnal: so uf the machine's bios is set to pxe boot, on reboot it should pxe
<roaksoax_> xygnal: if the machine is set to boot from disk, on reboot it should just boot from disk
<roaksoax_> in EFI is the same behavior, although, there was a bug in curtin that if maas wasn't available, it wouldn't boot from local disk
<xygnal> how do you avoid an accidental PXE causing it to re-deploy?
<roaksoax_> xygnal: if the machine is 'deployed' maas will tell it to localboot if the machines PXE's
<xygnal> ok that was what I was looking for, confirmation that once deployed status it will respond differentl
<xygnal> the problem was that the statement telling it to not pxe boot during DEPLOYING was not triggering, so it would just continue to re-pxe deploy :)
<roaksoax> xygnal: yeah maas should never tell the machine to re-deploy when it is already deployed
<mup> Bug #1699286 opened: [2.2, trunk] test__renders_ntp_servers_as_comma_separated_list fails randomly <MAAS:Triaged by mpontillo> <MAAS 2.2:Triaged by mpontillo> <https://launchpad.net/bugs/1699286>
<mup> Bug #1605312 changed: Unhandled failure in updating lease. django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key" <MAAS:Invalid> <https://launchpad.net/bugs/1605312>
<mup> Bug #1699308 opened: [2.2] TestDeviceHandler.test_list_num_queries_is_the_expected_number fails randomly <MAAS:New> <https://launchpad.net/bugs/1699308>
<xygnal> anyone tried to deply esxi using maas? i know its unsupported.  curious if anyone has tried.
<ikonia> actually deploy esx, or deploy to esx
<seanhoughton> hi all, we're using MaaS to provision KVM vms but on our production server we get an error "No user data registered for node named ct-vm-01001" just after the node transitions to DEPLOYED state. I've looked through the code but it's getting tricky to figure out how to debug. Any ideas? We're on MaaS 2.14
#maas 2017-06-21
<mup> Bug #1699479 opened: test__dhcp_configurations_rendered fails unable to resolve localhost when in ipv6 network <MAAS:In Progress by danilo> <https://launchpad.net/bugs/1699479>
<nan_> Hello, I'm investigating MAAS and have some questions.
<nan_> Can machines be provisioned persistently, or is it on a usage-needed basis?
<nan_> As in, when machines are Allocated or Deployed, is that to a specific user or can MAAS be used to run an application server?
<pmatulis>  nan_ a MAAS-deployed machine must be allocated to *some* user but once it's up it's available to the network
<nan_> pmatulis: so that user could be "admin" etc.?
<pmatulis> nan_, sure
<nan_> pmatulis: awesome, thanks for that clarification. So does MAAS strictly control hardware allocation? Like, can I run an application on top of it as though it's one large machine?
<pmatulis> if you are deploying directly from MAAS (not indirectly via Juju) then that user needs to import public SSH keys if you care about MAAS installing those keys on the deployed machine automatically
<roaksoax> nan_: the main goal of MAAS is to deploy machines
<roaksoax> nan_: you can use juju, chef, puppet, ansible with MAAS as well
<pmatulis> nan_, re hardware allocation, i don't follow. MAAS deploys machines and you do whatever you want with them
<pmatulis> you don't install applications on MAAS itself but on the machines it deploys
<nan_> pmatulis: roaksoax: so once I've deployed X number of machines, do the provisioned servers appear as a single mega-machine, or still as X individual servers?
<pmatulis> individual
<pmatulis> nan_, what are you planning?
<nan_> pmatulis: my organization runs library software (evergreen-ils.org), and we are looking at new ways to deploy/manage our production cluster
<nan_> Individual machines are ideal because we use multiple application servers and DBs
<pmatulis> nan_, sweet
<nan_> pmatulis: roaksoax: thank you both for your help
<nan_> pmatulis++
<nan_> roaksoax++
<lrensing> hi everyone
<lrensing> anyone seen this error before?
<lrensing> http://paste.openstack.org/show/613315/
<roaksoax> /win 16
<roaksoax> lrensing: that seems like some moduls are not installed ?
<roaksoax> never seen that error before
<mup> Bug #1699555 opened: [Test] Bug to test new jenkins lander bug resolver <MAAS:Triaged by newell-jensen> <https://launchpad.net/bugs/1699555>
<mup> Bug #1699622 opened: [2.2] dhcp-acquired IP address of a node in rescue mode disappears from cli output and web ui <MAAS:New> <https://launchpad.net/bugs/1699622>
<hel0tsp> ahoy
<hel0tsp> do i need impi for maas?
#maas 2017-06-22
<tamasgp> Hi! Anyone here?
<exoduswtf> Anyone had success deploying cent6 over uefi on maas 2.2.0?
<exoduswtf> I see the install succeed, but once the box reboots it just drops straight into the grub shell
<roaksoax> exoduswtf: centos 6 didn't support uefi until 6.8 i think
<roaksoax> hence maas doesn't support it yet
<exoduswtf> Yeah, that's what I thought on support not being there til 6.8
<exoduswtf> Snippets in the maas code make it sound like the is version available is always the latest minor release of the major version
<exoduswtf> I know the releases are tagged as centos66/70
<roaksoax> yeah that's a bug
<roaksoax> should be centos6/centos7
<exoduswtf> I guess I'm expecting centos66 to actually be centos 6.9, but I might be misunderstanding
<roaksoax> exoduswtf: it should be the latest yes
<exoduswtf> If it's 6.9, then uefi should be supported, I'm just not sure if there's wrapper code missing in the provisioningserver to support it
<roaksoax> the image needs enablement to support efi
<exoduswtf> Yeah, I guess it needs efibootmgr and the right boot partitions setup
<exoduswtf> I just haven't been able to figure out how to bootstrap all that in
<exoduswtf> Might try to see if I can get support through advantage on that
<exoduswtf> I expect I'll have to build a custom image with image builder
<xyz12355676> help
<day1000> I am unable to conjure-up landscape, and continue to hit this issue.  Not sure how-to troubleshoot this direct issue, nor able find if it's a configuration issue. (http://paste.ubuntu.com/24924556/)
<day1000> Wondering if anyone has enountered this with MaaS, and Juju.
<mup> Bug #1699833 opened: MAAS doesn't include link speed in interface_set output <cpe> <MAAS:New> <https://launchpad.net/bugs/1699833>
<mup> Bug #1699833 changed: Include link speed in interface_set output <cpe> <MAAS:Triaged> <https://launchpad.net/bugs/1699833>
<mup> Bug #1699833 opened: MAAS doesn't include link speed in interface_set output <cpe> <MAAS:New> <https://launchpad.net/bugs/1699833>
<mup> Bug #1699855 opened: maas should include topology information about network interfaces <cpe> <MAAS:New> <https://launchpad.net/bugs/1699855>
<mup> Bug #1699855 changed: maas should include pci card topology information about network interfaces <cpe> <MAAS:New> <https://launchpad.net/bugs/1699855>
<mup> Bug #1699855 opened: maas should include topology information about network interfaces <cpe> <MAAS:New> <https://launchpad.net/bugs/1699855>
<mup> Bug #1648190 changed: IPMI problem with Tyan system (AST2400) <MAAS:Invalid> <https://launchpad.net/bugs/1648190>
<mup> Bug #1699864 opened: [2.3 UI] Check all box in Nodes listing page does not select all nodes <MAAS:Triaged> <https://launchpad.net/bugs/1699864>
<vlad__> Hey guys I've got a quick question. I did a fresh install of maas for a region because I was getting weird errors and wanted to start over. However, I've done so on a fresh ubuntu install and now I'm getting a new error. Can anyone give me a lead on this? [critical] Error on request (2315) controller.check_images: 6mqbgr
<roaksoax> vlad__: seems like that the rack/region connection is failing ?
<roaksoax> vlad__: what version is this ?
<vlad__> roaksoax: Checking now
<vlad__> roaksoax: {         'version': '1.8.0',         'subversion': 'alpha10+bzr3750',         'capabilities': ['capability1', 'capability2', ...]     }
<vlad__> Seems like I may need to change my ppa?
<roaksoax> oh wow, version 1.8.0, that's deprecated
<roaksoax> it has been deprecated by a long time now
<vlad__> Yeah that came in the default ubuntu apt repos... I've added the stable branch just forgot to do so on this newest run lol thanks for helping!
<roaksoax> vlad__: what ubuntu version are you using ?
<roaksoax> vlad__: there's no 1.8.0 here: https://launchpad.net/ubuntu/+source/maas
<vlad__> roaksoax: Should be 16.04.2
<vlad__> It is that version
<vlad__> Upon boot up I ran update and upgrade
<roaksoax> vlad__: dpkg -l | grep maas
<vlad__> roaksoax: After adding the new ppa and updating all maas is 2.0.2
<roaksoax> 2.2.0
#maas 2017-06-23
<Guest60905> hi can i ask for help?
<Guest60905> i installed mass configured dhcp on second nic i put a notebook on that pxe boot so it booted up from pxe maas but after it booted up it says: "Can not apply stage final, no datasource found!" ....
<Guest60905> nobody?
<Guest60905> please can somebody help me?
<rbasak> Guest60905: I suggest you wait a while - most channel users are still asleep.
<Guest60905> thanks
<Guest60905> the erro is here: http://imgur.com/a/6fheb
<Guest60905> *error
<gimmic> For some reason maas isn't detecting some new racks of gear I added. In the past I statically assigned ipmi interfaces and the nodes would show up
<gimmic> what's the discovery mechanism?
<vlad__> Hey guys how can I keep maas from running certain tests during commissioning?
<vlad__> I'm trying to run this command but it just comes back not found (I have the correct system_id): maas $PROFILE machine commission -k $SYSTEM_ID testing_scripts=none  how far off is this from correct?
<vlad__> Nevermind had to change script for finding system_id
<mup> Bug #1700134 opened: default minimum HWE kernel preventing deploying  <4010> <MAAS:New> <https://launchpad.net/bugs/1700134>
<roaksoax> gimmic: /win 14
<roaksoax> err
<roaksoax> sorry
<roaksoax> gimmic: can those machines pxe boot of maas ?
<roaksoax> maybe they are not pxe booting
<gimmic> Yeah, I think that is the new issue
<mup> Bug #1700161 opened: Rescue Mode fails and can't exit <MAAS:New> <https://launchpad.net/bugs/1700161>
<vlad__> Are interfaces limited to having one fabric per interface?
<darinavbt> Hi all
<darinavbt> I'm spinning up an OpenStack cloud using Ubuntu Autopilot. I'm setting up MAAS. When I commission machines, should I check "Allow SSH access and prevent machine from powering off"?
<darinavbt> I'd think I'd want to be able to SSH into it, but I don't know why that's linked with "prevent powering off".
<darinavbt> Or, if that checkbox is unchecked, can I still log in as ubuntu using the SSH key I put into MAAS?
<jamesbenson> darinavbt: That's just commissioning it..
<jamesbenson> after it is commissioned, you'll have to deploy.
<jamesbenson> for commissioning, it powers off  once complete.  It's not in a good state to use at that point.
<jamesbenson> Once deployed, the power will stay on.
<darinavbt> Ah, gotcha. Yeah, I was just reading the MAAS docs and was heading towards that conclusion. Thanks!
<mup> Bug #1700180 opened: [2.2] Power querying failures related to SSL/TLS are not propagated to logs and UI <cpe> <cpe-sa> <https> <security> <ssl> <tls> <MAAS:New> <https://launchpad.net/bugs/1700180>
<ltrager> darinavbt: The enable SSH and prevent power off option on commissioning and testing is there to help with debug
<ltrager> darinavbt: As jamesbenson mentioned you still need to deploy an operating system to use it
#maas 2018-06-18
<mup> Bug #1777443 opened: Got 500 http error when using MAAS Api <MAAS:New> <https://launchpad.net/bugs/1777443>
<PatrickD_> Hi, trying to get MAAS HA working, with SSL. But it looks like the address is converted to ipv6 and used in the URL (https://[::ffff:10.10.10.10]/MAAS/) which fails because the cert is not valid. Any idea ?
<roaksoax> PatrickD_: https://docs.maas.io/2.4/en/installconfig-network-ssl
<PatrickD_> yes, I did that. But when accessing the interface, rackd tries to connect to the API using https://[ipv6]/MAAS/rpc and fails the cert check.
<roaksoax> PatrickD_: what's maas_url on rackd.conf ?
<PatrickD_> https://domain.name/MAAS
<roaksoax> PatrickD_: that's probably because domain.name is resolving to the IPv6?
<PatrickD_> no it doesn't. It resolves to ipv4 only. in rpc/clusterservice.py it looks like it transfers the name to IP (v6). ~line 1043
<roaksoax> PatrickD_: does the region have any IPv6 address ?
<PatrickD_> link local
<PatrickD_> no, only loopback
<roaksoax> PatrickD_: oh so maas uses: [::ffff:10.10.10.10]
<roaksoax> PatrickD_: that's fine
<roaksoax> PatrickD_: that's expected
<roaksoax> that doesn't mean we are using ipv6, but for maas to support Ipv6 we need to do that
<PatrickD_> So how it is supposed to check the cert validity using https://[::::ffff:10.10.10.10]/MAAS/rpc ?
<roaksoax> PatrickD_: you are welcome to file a bug on that, although, we currently only support ssl for front-facing users and not inter-controller communication
<roaksoax> PatrickD_: but please do file abug
<PatrickD_> Ah, I see :) Makes sense now. We will file a bug. It means we will need 2 IPs for API. 1 for inter-controller and 1 for front-facing, with redirection of 80 to 443.
<roaksoax> PatrickD_: yeah, so for inter-rack communication it would seem we would just have to use the domain instead of changing as you described
<PatrickD_> We will file the bug tomorrow :) Thanks for your help ! (And thanks for MAAS too ;)
#maas 2018-06-19
<c06> hi all
<c06> i have two nodes both have MAAS with dhcp enabled on different local networks. when i am trying to find the other machines dhcp ip are getting conflicted
<c06> is there anyway where i can specify only to fetch ip from particular server
<c06> any suggestions.?
<mup> Bug #1777712 opened: SSL doesn't work for inter-controller communication <MAAS:New> <https://launchpad.net/bugs/1777712>
<mup> Bug #1777719 opened: [2.4] Bad traceback when an image is not in the image mirror <error-surface> <MAAS:Triaged> <https://launchpad.net/bugs/1777719>
#maas 2018-06-20
<mup> Bug #1777742 opened: POD VM fails deleting on AARCH64 <MAAS:New> <https://launchpad.net/bugs/1777742>
<PatrickD_> Hi, in https://github.com/maas/maas/blob/master/src/metadataserver/user_data/templates/snippets/maas_enlist.sh, line 239. The sed should transform http://servername/MAAS/api/2.0/machines/ into MAAS/api/2.0/machines/, like in the default, 2 lines above ? In my tests, it looks like it is doing nothing, and uses full serverurl as api_url, making the enlist_node call on the last line fail... Am I missing something ?
<PatrickD_> Ok, found the problem. The regex doesn't like our name, because it contains a "-". Looks like the regex should match all valid domain names, and it doesn't. I will report the bug.
<mup> Bug #1777924 opened: maas-enlist regex doesn't match all valid domain names <MAAS:New> <https://launchpad.net/bugs/1777924>
<mup> Bug #1777924 changed: maas-enlist regex doesn't match all valid domain names <MAAS:New> <https://launchpad.net/bugs/1777924>
<roaksoax> PatrickD_: http://paste.ubuntu.com/p/Gpn7rN4MR7/
<roaksoax> PatrickD_: try that ?
<mup> Bug #1777924 opened: maas-enlist regex doesn't match all valid domain names <MAAS:Triaged> <MAAS 2.4:Triaged> <https://launchpad.net/bugs/1777924>
<PatrickD_> That regex works.
<PatrickD_> Bug was opened by me :)
<PatrickD_> We changed our name to maaslb for now, and it fixed the problem.
#maas 2018-06-21
<mup> Bug #1777967 opened: dpkg-query: package 'maas-cluster-controller' is not installed and no inn is available <maas-maintainers> <MAAS:New> <https://launchpad.net/bugs/1777967>
<naturalblue> hi. I am trying to deploy a juju maas controller and it starts to deploy but i notice later on it looks for an IP (that was MAAS's old IP address) and then fails to deploy. Is there a way to get the details of what it is looking for. I have changed all ips and restarted all services on MAAS. I am not sure why it is still looking for the old address.
<naturalblue> would puttting the node into rescue mode and then logging in allow me to see the error
<mup> Bug #1778047 opened: No rack controllers can access the BMC of node <MAAS:New> <https://launchpad.net/bugs/1778047>
<naturalblue> i thought if you uploaded your ssh public key to maas then it was compied over to all machines it deploys so you can login. is this only on rescue mode?
<PatrickD_> not only rescue mode.
<PatrickD_> on commissioned and deployed
<mup> Bug #1777967 changed: dpkg-query: package 'maas-cluster-controller' is not installed and no inn is available <maas-maintainers> <MAAS:Invalid> <https://launchpad.net/bugs/1777967>
<PatrickD_> Having problems with Dell 640, with a Megaraid (perc) controller. fails to run smartctl (I guess it should either detect that it should use "-d megaraid,<diskid>" to scan the disks, or simply return that smartctl is not supported...
<PatrickD_> this is at commissioning, when it runs maas-run-remote-scripts
<PatrickD_> {"1.0": {"testing_scripts": [{"name": "smartctl-validate", "path": "testing/smartctl-validate", "script_result_id": 89, "script_version_id": 1, "timeout_seconds": 300, "parallel": 1, "hardware_type": 3, "parameters": {"storage": {"type": "storage", "value": "all"}}, "packages": {"apt": ["smartmontools"]}, "for_hardware": [], "has_started": true}]}}
<mup> Bug #1777967 opened: dpkg-query: package 'maas-cluster-controller' is not installed and no inn is available <maas-maintainers> <MAAS:Invalid> <https://launchpad.net/bugs/1777967>
<mup> Bug #1777967 changed: dpkg-query: package 'maas-cluster-controller' is not installed and no inn is available <maas-maintainers> <MAAS:Invalid> <https://launchpad.net/bugs/1777967>
<polto__> hi
<polto__> is MAAS packaged with the latest LTS ubuntu (18.04) is considered stable ?
<polto__> I am facing errors trying to deploy servers from MAAS
<roaksoax> polto__: 2.4b2 is in the archive
<roaksoax> polto__: 2.4 final is on a ppa
<PatrickD_> @roaksoax Did you get a chance to read my question from 2 hours ago ? We have an issue with smartctl. We are digging, trying to find how the index.json is generated. ours seem to be missing stuff in the type:storage value: all... Should be a table there instead of "all".
<ltrager> PatrickD_: MAAS uses 'all' as a place holder when a new machine is added.
<ltrager> PatrickD_: What happens is the machine starts commissioning, sends commissioning results to MAAS, then maas-run-remote-scripts redownloads the tar which has the discovered values
<ltrager> PatrickD_: when smartctl failed did commissioning successfully complete and discover disks?
<PatrickD_> no. It sees the size of the disk (/dev/sda, 1 single disk because it is a RAID5 on a Dell server PERC controller). and then fails with this error : https://paste.ubuntu.com/p/6CsfwRztds/
<ltrager> PatrickD_: Whats the output of the API for the machine? - maas $PROFILE machine read $SYSTEM_ID
<PatrickD_> https://paste.ubuntu.com/p/wwtQWHhcbK/
<PatrickD_> maas-run-remote-scripts downloads the index.json, right ? And this json should not contain "all" in that spot. It should contain a hash. So maas-run-remote-scripts fails because .get method on a non-has fails ?
<ltrager> PatrickD_: yes maas-run-remote-scripts downloads index.json in a tarball with all scripts that are pending
<ltrager> PatrickD_: the "all" placeholder is fine during commissioning but not during testing
<ltrager> PatrickD_: What should be happening is commissioning finishes, maas-run-remote-scripts redownloads the tar containing index.json and executes the tests
<ltrager> PatrickD_: what version of MAAS are you running
<PatrickD_> 2.4 from ppa
<PatrickD_> so what script gathers the information necessary to build the proper storage information ?
<ltrager> PatrickD_: The builtin commissioning script 00-maas-07-block-devices
<PatrickD_> https://paste.ubuntu.com/p/q9cj2sQd5g/
<PatrickD_> result from the 00-maas-07-block-devices script
<ltrager> PatrickD_: is there anything in the machines event log?
<ltrager> PatrickD_: What should be happening is 00-maas-07-block-devices is received which causes MAAS to regenerate storage parameters.
<ltrager> PatrickD_: That should replace 'all' with the actual data
<PatrickD_> event log ?
<ltrager> PatrickD_: In the UI select the machine and click on the Events tab
<PatrickD_> Nothing special there... PXE Request - commissioning, then Node changed status - From 'Commissioning' to 'Testing'
<PatrickD_> then to Node changed status - From 'Testing' to 'Failed testing'
<PatrickD_> you think the output from the 07-block-devices script is fine ? (I haven't seen any outputs to really compare to :)
<ltrager> PatrickD_: yeah it seems fine and the output from the API shows the disk is being added
<ltrager> PatrickD_: could you file a bug at https://bugs.launchpad.net/maas/+filebug and include the logs from /var/log/maas?
<ltrager> PatrickD_: to get around this bug you can commission without running any tests. In the UI you can just deselect smartctl-validate
<ltrager> PatrickD_: Using the API you can change what runs by default by running
<ltrager> PatrickD_: maas $PROFILE node-script remove-tag smartctl-validate tag=commissioning
<PatrickD_> Ok. Trying a few things, then I will file the bug with more info.
<PatrickD_> the command to remove smartctl is the same as doing it per machine in the UI ?
<ltrager> PatrickD_: The API command stop its from being selected by default
<ltrager> PatrickD_: This way if you use something like JuJu to commission it won't run
<jaypipes> hi folks. I'm hoping a MaaS expert might be able to take a look at https://bugs.launchpad.net/maas/+bug/1778047 and provide some advice as to what I should try to get my machines able to be commissioned by the rack controller. thx in advance!
<arkaroo> I can't seem to get maas to comission nodes. I have my setup at the point where nodes will PXEboot from the maas region controller, but no matter what I do, the PXEboot ends with the error "mounting /root on /media/root-ro failed: Invalid argument" followed by a bunch of other errors about not being able to mount to /root, and then just the busybox initramfs terminal prompt
<arkaroo> it looks like squashfs is trying to mount to a path that doesn't exist, but I can't figure out why, anyone know what's going on?
<arkaroo> for the record: it looks like if the PXEboot can't download the image it wants, it doesn't stop and tell you the network is incorrectly configured, it tries to extract a zip that isn't there and gives you mounting errors
<arkaroo> not sure yet why I can pxeboot from the region controller but can't download files from it, something is very weird there
<Wilee> Hello
#maas 2018-06-22
<adham> hi everyone
<adham> I have an urgent question I have installed kubernetes on top of maas, then a lot of vms has been created and these nodes can be in MAAS, however these nodes are not available in kubernetes I'm not sure if these 2 are related
<adham> I installed The canonical distribution kubernetes
<adham> The vms that has been created are as following: aware-code aware-yak casual-corgi casual-whale casual-mole clear-hound close-liger cool-troll decent-beetle divine-bug driven-drake easy-cod equal-frog equal-swan exotic-earwig expert-cow expert-slug fair-bee first-dog frank-monkey gentle-racer good-koi grown-bunny guided-eft handy-wahoo hip-hornet holy-bass holy-hen intent-bear large-kit
<adham> do anyone have any idea about those spam vms on maas?
<adham> or local vm i'd say?
<adham> details I have added to the question here https://stackoverflow.com/questions/50970133/installed-kubernetes-on-ubuntu-and-i-see-a-lot-of-nodes-are-getting-created-in-m
<adham> but it seems that noone have knowledge
<adham> if anyone here have any idea, it would be very appreciated if you can really assist
<mup> Bug #1778258 opened: Commissioning script names cut off in UI <MAAS:Triaged> <https://launchpad.net/bugs/1778258>
<mup> Bug #1778260 opened: [UI] Scipt result actions dropdown has cutoff text <ui> <MAAS:Triaged> <https://launchpad.net/bugs/1778260>
#maas 2018-06-24
<Guest0> Q: Can maas-image-builder build SLES images? https://docs.maas.io/2.4/en/installconfig-images reads " Ubuntu Advantage is needed in order to use Windows, RHEL and SUSE images or in order to build a custom image for any operating system." but maas-image-builder -h only displays opensuse,windows,centos,rhel
<Guest0> Can MaaS.io use vmware virtual machines as "Machines" to deploy os ontop?
#maas 2020-06-15
<mup> Bug #1883269 changed: UI traceback adding alias to nic <ui> <MAAS:Fix Released> <maas-ui:Fix Released> <https://launchpad.net/bugs/1883269>
<mup> Bug #1883294 changed: maas changepassword does not have automatable options <cdo-qa> <foundations-engine> <MAAS:Invalid> <https://launchpad.net/bugs/1883294>
<mup> Bug #1879561 changed: 2.7 snap install fails on lxc containers <cdo-qa> <field-high> <MAAS:Invalid> <https://launchpad.net/bugs/1879561>
<d0ugal> ltrager: You can probably do /msg ChanServ OP #maas ltrager
<ltrager> d0ugal: I can :)
<mup> Bug #1847794 opened: Pod must be on a known host if interfaces are specified <field-high> <MAAS:New for bjornt> <MAAS 2.6:New> <MAAS 2.7:New> <https://launchpad.net/bugs/1847794>
#maas 2020-06-16
<mup> Bug #1842896 changed: [VM Provisioning] If constraints: zones=<ZONE> causes Juju and MAAS to provision new VMs on same node of correct zone in disrespect of overcommit restrictions <juju:Invalid> <MAAS:Expired> <https://launchpad.net/bugs/1842896>
<mup> Bug #1842896 opened: [VM Provisioning] If constraints: zones=<ZONE> causes Juju and MAAS to provision new VMs on same node of correct zone in disrespect of overcommit restrictions <juju:Invalid> <MAAS:Expired> <https://launchpad.net/bugs/1842896>
<mup> Bug #1842896 changed: [VM Provisioning] If constraints: zones=<ZONE> causes Juju and MAAS to provision new VMs on same node of correct zone in disrespect of overcommit restrictions <juju:Invalid> <MAAS:Expired> <https://launchpad.net/bugs/1842896>
<mup> Bug #1883735 opened: deb package leaks /run/lock files on purge <MAAS:In Progress by ack> <https://launchpad.net/bugs/1883735>
<mup> Bug #1883743 opened: Add chassis suggests power types which aren't chassis <ui> <MAAS:New> <https://launchpad.net/bugs/1883743>
<mup> Bug # opened: 1883747, 1883748, 1883750, 1883752
<mup> Bug # changed: 1883747, 1883748, 1883750, 1883752
<mup> Bug # opened: 1883747, 1883748, 1883750, 1883752
<mup> Bug #1883747 changed: Problem while MAAS commissioning a Power system (witherspoon) with NVMe drive <ppc64el> <MAAS:New> <The Ubuntu-power-systems project:New for maas> <https://launchpad.net/bugs/1883747>
<mup> Bug #1883747 opened: Problem while MAAS commissioning a Power system (witherspoon) with NVMe drive <ppc64el> <MAAS:New> <The Ubuntu-power-systems project:New for maas> <https://launchpad.net/bugs/1883747>
<mup> Bug #1883747 changed: Problem while MAAS commissioning a Power system (witherspoon) with NVMe drive <ppc64el> <MAAS:New> <The Ubuntu-power-systems project:New for maas> <https://launchpad.net/bugs/1883747>
#maas 2020-06-17
<mup> Bug #1883824 opened: Support LXD projects in power control <MAAS:New> <https://launchpad.net/bugs/1883824>
<mup> Bug #1883830 opened: Snap enterprise proxy support <MAAS:New> <https://launchpad.net/bugs/1883830>
<mup> Bug #1847794 changed: Pod must be on a known host if interfaces are specified <MAAS:Fix Released by bjornt> <MAAS 2.6:Fix Released> <MAAS 2.7:Fix Released> <MAAS 2.8:Fix Released> <https://launchpad.net/bugs/1847794>
<mup> Bug #1847794 opened: Pod must be on a known host if interfaces are specified <MAAS:Fix Released by bjornt> <MAAS 2.6:Fix Released> <MAAS 2.7:Fix Released> <MAAS 2.8:Fix Released> <https://launchpad.net/bugs/1847794>
<mup> Bug #1847794 changed: Pod must be on a known host if interfaces are specified <MAAS:Fix Released by bjornt> <MAAS 2.6:Fix Released> <MAAS 2.7:Fix Released> <MAAS 2.8:Fix Released> <https://launchpad.net/bugs/1847794>
<mup> Bug #1847794 opened: Pod must be on a known host if interfaces are specified <MAAS:Fix Released by bjornt> <MAAS 2.6:Fix Released> <MAAS 2.7:Fix Released> <MAAS 2.8:Fix Released> <https://launchpad.net/bugs/1847794>
<mup> Bug #1847794 changed: Pod must be on a known host if interfaces are specified <MAAS:Fix Released by bjornt> <MAAS 2.6:Fix Released> <MAAS 2.7:Fix Released> <MAAS 2.8:Fix Released> <https://launchpad.net/bugs/1847794>
<mup> Bug #1883986 opened: MAAS should retreive the the amount of space used in a storage pool from the Pod/VM host <MAAS:New> <https://launchpad.net/bugs/1883986>
<mup> Bug #1883986 changed: MAAS should retreive the the amount of space used in a storage pool from the Pod/VM host <MAAS:New> <https://launchpad.net/bugs/1883986>
<mup> Bug #1883986 opened: MAAS should retreive the the amount of space used in a storage pool from the Pod/VM host <MAAS:New> <https://launchpad.net/bugs/1883986>
#maas 2020-06-18
<mup> Bug #1884051 opened: Can't add a machine without specifying the mac address <ui> <MAAS:New> <https://launchpad.net/bugs/1884051>
<mup> Bug #1884075 opened: No network device detected on commissioning <MAAS:In Progress by ack> <https://launchpad.net/bugs/1884075>
<mup> Bug #1884075 changed: No network device detected on commissioning <MAAS:In Progress by ack> <https://launchpad.net/bugs/1884075>
<mup> Bug #1884075 opened: No network device detected on commissioning <MAAS:In Progress by ack> <https://launchpad.net/bugs/1884075>
<mup> Bug #1884075 changed: No network device detected on commissioning <MAAS:In Progress by ack> <https://launchpad.net/bugs/1884075>
<mup> Bug #1884075 opened: No network device detected on commissioning <MAAS:In Progress by ack> <https://launchpad.net/bugs/1884075>
<mup> Bug #1884087 opened: MAAS Notifications should require user=True, admin=True or both. <MAAS:Triaged> <https://launchpad.net/bugs/1884087>
<mup> Bug #1884088 opened: Maas notifications should support usernames <MAAS:Triaged> <https://launchpad.net/bugs/1884088>
<mup> Bug #1884099 opened: If two notifications are collapes in the UI the "dismiss all" label is visible even if the notifications are not dissmisable <ui> <MAAS:New> <https://launchpad.net/bugs/1884099>
<mup> Bug #1884112 opened: MAAS keys count in user list is bogus <ui> <MAAS:New> <https://launchpad.net/bugs/1884112>
<mup> Bug #1884157 opened: Creating a machine with an existing IPMI IP results in db contraint error <MAAS:New> <https://launchpad.net/bugs/1884157>
#maas 2020-06-19
<mup> Bug #1884051 changed: Can't add a machine without specifying the mac address <ui> <MAAS:Fix Released by ya-bo-ng> <maas-ui:New> <https://launchpad.net/bugs/1884051>
<mup> Bug #1884250 opened: The MAAS `users create` API should allow True/False for is_superuser <api> <MAAS:Triaged> <https://launchpad.net/bugs/1884250>
<mup> Bug #1884250 changed: The MAAS `users create` API should allow True/False for is_superuser <api> <MAAS:Triaged> <https://launchpad.net/bugs/1884250>
<mup> Bug #1884250 opened: The MAAS `users create` API should allow True/False for is_superuser <api> <MAAS:Triaged> <https://launchpad.net/bugs/1884250>
<mup> Bug #1884276 opened: Terrible user experience adding existing LXD host <MAAS:New> <https://launchpad.net/bugs/1884276>
<mup> Bug #1884276 changed: Terrible user experience adding existing LXD host <MAAS:New> <https://launchpad.net/bugs/1884276>
<mup> Bug #1884276 opened: Terrible user experience adding existing LXD host <MAAS:New> <https://launchpad.net/bugs/1884276>
<mup> Bug #1884278 opened: Can't commission without a test <MAAS:Incomplete> <https://launchpad.net/bugs/1884278>
<mup> Bug #1884310 opened: Failed to determine /dev/nvme0n1 partition number <MAAS:New> <https://launchpad.net/bugs/1884310>
#maas 2020-06-21
<mup> Bug #1814164 changed: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
<mup> Bug #1814164 opened: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
<mup> Bug #1814164 changed: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
<mup> Bug #1814164 opened: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
<mup> Bug #1814164 changed: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
<mup> Bug #1814164 opened: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
<mup> Bug #1814164 changed: Unable to remove DNS records where name is in the format <name>.<domain> <dns> <MAAS:Expired> <https://launchpad.net/bugs/1814164>
