#maas 2012-05-29
<czajkowski> that was fast :)
<czajkowski> ubuntulog2: welcome and logging
<Soekris> I can't run juju bootstrap. I get the error http://paste.ubuntu.com/1012908/ http://paste.ubuntu.com/1011995/ the error message and the juju config
<allenap> Soekris: I think the maas-oauth line in your juju config looks wrong. Try removing the leading ${ and the trailing }
<allenap> Soekris: Do you know how they got there? Was that line presented to you that way? Even if it was a mistake I'd like to know how it happened so that we can adjust the UI.
<Soekris> allenap: i've followd the https://wiki.ubuntu.com/ServerTeam/MAAS/Juju
<Soekris> allenap: thanx 2012-05-29 13:55:30,124 INFO 'bootstrap' command finished successfully Thas nicer :D
<allenap> Soekris: Ah, that's interesting. The token ${maas-api-key} is meant to be replaced in its entirety. I'll have a think about how that could be better represented.
<allenap> Soekris: I've updated the docs so hopefully you'll be the last to get stuck on this. Sorry about that, and thanks for letting me know.
<Soekris> allenap: if i can get it smooth to work i will try to make a youtube movie. som people can pause and run.
<allenap> Soekris: That could be very helpful.
<allenap> Hello rvba!
<rvba> Hello allenap!
<Daviey> ooo, hello chaps!
<Soekris> next problem http://paste.ubuntu.com/1013374/ i have done ssh-kegen -t dsa copied de id.rsa.pub in to the dashboard
<Soekris> hello i'm staring from skratch. When i boot with a unknow server. With image must i chose ? maas-precise-arch-commision ? t
<Daviey> Soekris: no
<Daviey> Soekris: just let it timeout
<Soekris> haha patience something i doen't know :D
<Daviey> Soekris: 20 seconds? :)
<Daviey> You can just go down to maas-enlist if you are really impatient
<Soekris> haha thanx.
<Soekris> Yes i got 1 blue node :D
<Soekris> juju bootstrap did work. but juju status didn't work. I got server refused mesages http://paste.ubuntu.com/1013772/\
#maas 2012-05-30
<allenap> Morning all.
<rvba> Morning allenap.
<rvba> allenap: do you have time for a quick pre-imp call?
<allenap> rvba: OTP, but after that sure.
<rvba> allenap: cool.
<allenap> rbasak: Wow, long email :) I look forward to reading it.
<Soekris> juju bootstrap did work. but juju status didn't work. I got server refused mesages http://paste.ubuntu.com/1013772/
<Soekris> I don't know where to look futher
<allenap> Soekris: How long after bootstrap did you try this? Bootstrap can take a long time (it has to install Ubuntu for one thing).
<Soekris> allenap: is there a command to see of is finished ?
<allenap> Soekris: juju status ;)
<allenap> Soekris: It's been a while since I tried this, so I can remember exactly what to expect :-/
<Soekris> i have installed a new virtual server so everty time each step goes smoother :D
<allenap> Soekris: Have you tried ssh'ing to ubuntu@s4-cl1-maas.jwvwcomputing.com?
<Soekris> yes that wil work
<Soekris> in the verbose it looked that er was missing a kind of service
<Soekris> I will try it from skrath with more pacient :D
#maas 2012-05-31
<jtv1> bigjools: still not seeing any decent way to test my virsh power script, or to make it quite trivial.
<jtv> One thing I could do is allow the caller to override the virsh executable it uses, so that a test can inject âechoâ instead.  But that leaves baggage in normal execution paths.
<jtv> I could make it a power parameter, I guess.
<bigjools> jtv: sorry was caught up with things.  I don't think you can test the scripts.
<bigjools> the approach I took with the wol stuff was to test the templating and to test that the script returns with a 0 code.
<jtv> Thanks â I've got some things I can do now.
<bigjools> jtv: since the scripts are intended to be customised, I think unit testing them is not useful
<bigjools> they should be QAed instead
<jtv> I'd like to know that at least it makes some kind of syntactic sense to the shell.
<jtv> Which I can, actually, test to some extent.
<bigjools> jtv: exit code!
<jtv> Alas, no, not that easy.
<bigjools> not for all, no
<bigjools> and we really don't want to start up VMs
<bigjools> so like I said, I'd really leave it to QA
<bigjools> this is why I made this level of separation
<jtv> Well there's one thing I can do in tests that makes it not start up VMs and yet exercises most of the script.
<jtv> One simple thing, that is.  Complicated things can do more, I'm sure.  :)
<bigjools> :)
<jtv> And there.  Just found a bug thanks to my test!
<jtv> bigjools: I was just saying that we could stay on and chat about the anonymous-metadata requirement
<bigjools> jtv: ah ok I have a call with gavin first, will call you right after
<jtv> OK
<rvba> allenap: I see you've got a call now, so just ping me when you will be available to talk about this "migration" problem.
<allenap> rvba: Cool, ta.
<bigjools> jtv: ok, wanna call?
<jtv> bigjools: otp
<jtv> w/someone else
 * bigjools is hurt
<bigjools> you said you'd only ever otp me
<jtv> uh-oh
<bigjools> jtv: I need to head off - maybe you want to talk to Daviey about the requirements for this?
<jtv> Yes, I think I will.
<Daviey> o/
<Daviey> jtv:
<jtv> Daviey: Hi.  It's about the anonymous metadata accessâ¦ I wanted to be clear about the purpose.  Is it _just_ for debugging?  Because several of the things that were said sort of suggested that it might be needed for commissioning.
<jtv> Which I hope is not the case, really.
<jtv> But if it's for debugging, we might be able to require run-of-the-mill UI admin authentication.
<Daviey> jtv: so..
<Daviey> jtv: We have some hardware that can't boot as maas can expect to..
<jtv> I see.  This is indeed entirely different from the use-case we had before.
<Daviey> Therefore, for 'demo / testing' if we can avoid having to push data to the node, we can get a working setup
<Daviey> jtv: yeah, seems to fit the same model tho?
<jtv> Wait â I'd like to comment on those two lines separately.
<Daviey> ok
<jtv> About âseems to fit the same model,â I think admin auth makes more sense _for debugging_ than opening up the whole thing â whereas for difficult hardware it may not be very user-friendly.
<Daviey> jtv: well, injecting a known user/pass is MUCH easier than a generated on demand oauth key
<jtv> The other thing is: what do you mean by âavoid having to push data to the nodeâ exactly, so that I understand correctly?  Do you mean that we don't want to push the credentials for the metadata service to the node through its boot params?
<Daviey> So.. using admin auth does help.
<jtv> We could probably have a separate key for this.
<Daviey> jtv: we don't have access to boot params at deploy time.
<jtv> But it's a security hazard if it's open in production.
<Daviey> Crappy hw
<jtv> But then how does the node even know where to get its metadata?  We push that in the same way.
<Daviey> jtv: yeah, this is for debug / demo (closed network) workflow
<Daviey> jtv: Yeah, knowing where the metadata service is = known constant
<Daviey> It's really injecting runtime generated auth is the problem
<Daviey> (although, could be really smart and use avahi for auto detecting where the metadata service is... but really.. overkill.)
<jtv> I don't suppose we could have a single, shared key and differentiate by MAC address?  Bit of a hack, and only good for a limited class of networks.
<Daviey> hmm
<jtv> Then again, if you're running in the demo configâ¦
<Daviey> the mac address isn't normally exposed in the http request.
<jtv> Argh!  How does the node even know how to identify itself?  It doesn't receive its system_id either!
<Daviey> So.. the MAAS server would need to arp
<jtv> Maybe the node could do it.
<Daviey> X-FORWARDED_FOR: $(hostname) .. seems sane?
<jtv> How does it know its hostname at that stage?
<Daviey> X-Forwarded-For: rather.
<Daviey> jtv: well.. when using the model of dhcp allocated hostname.. MAAS is informed of the hostname, and the node knows it
<jtv> DHCP-allocated hostname?  Isn't that the node telling the DHCP server what hostname it wants?  If so, it'd have to know first â quod non.
<Daviey> jtv: hmm.. there are two models.. MAAS controls dhcp, and an existing dhcp
<Daviey> this fits existing dhcp quite well.
<jtv>  Ouch ouch ouch you want to make it rely on a given DHCP setup as well?  That's just making things worse & worse.
<Daviey> it's not all that bad IMO
<Daviey> remember, this is a non-prodcution setup
<Daviey> jtv: to contrast, this is how openstack does it.. https://github.com/openstack/nova/blob/master/nova/api/ec2/__init__.py#L253
<jtv> So you're talking about setting up a DHCP server with prepared leases with hostnames, for demo purposes?  It feels a lot like designing a whole feature specifically for just one demo.
<Daviey> jtv: no, this also fits the debug model quite well.
<jtv> Only if you have the hostnames.
<Daviey> i can do an out-of-band install of ubuntu server, apt-get install cloud-init, providing i installed with the hostname MAAS expects, we are GOLD
<Daviey> right?
<Daviey> jtv: the other solution is to do a reverse dns lookup based on the ip address, and post that.. That caters for MAAS controlling the dhcp
<Daviey> client side ^
<jtv> But why not have the client look up the mac address it uses for the metadata access?
<Daviey> also valid
<jtv> In fact it could just loop over its interfaces until it found a hit.
<Daviey> jtv: technically, i'd say mac address lookup is more insecure :)
<Daviey> but security isn't an issue here
<Daviey> jtv: yeah, mac address seems reasonable :)
 * jtv refreshes his memory on that topic
<Daviey> arp -na
<jtv> No I mean, ISTR MAC addresses were significant somehow to the EC2 metadata service.  Let me see if it matters to us at all.
<jtv> (As you say, this is identification, not authentication â and yes, we'll still need to worry about how to fit this in with actual authentication)
<Daviey> jtv: nah, mac address isn't interesting to the meta data service afaik.
<jtv> You're right.  We ditched that.
<jtv> Still, it's one of the few things the node does know when it starts commissioning.
<jtv> Daviey: how would we do enlistment on this hardware?
<jtv> (So I get a picture of what process we'd be working towards)
<Daviey> jtv: using the cmd line tool, maas-enlist
<jtv> OK so MAAS knows about its MAC addresses anyway.
<jtv> Will it need to have a custom(ized) metadata client?
<jtv> I guess it will â hardcoded base metadata service URL
<jtv> And it can embed its MAC address in the URL as well.
<Daviey> jtv: sounds good to me.
<Daviey> i think it will be a wrapper around cloud-init TBH
<Daviey> running cloud-init post-boot isn't as straight forward as calling a command, sadly
<jtv> Daviey: I think we'd have to make it a whole separate http tree.  Maybe a hidden /metadata/<version>/node/<mac>
<Daviey> jtv: hmm, mac can't just be passed as a parameter through urls.py to the same function which defaults to None, and cecks django settings if METADATA_MATCH_MAC = True ?
<Daviey> ahh.. unauth'd url
<Daviey> frack
<jtv> Yeah.  Want separate URL anyway, I think.
<Daviey> would be nice to be able to reuse the same code..
<Daviey> but i have every confidence you'd do the right thing :)
<jtv> Well you have to get the URL from a different source anyway, right?
<jtv> So that might as well include a mac address.
<Daviey> yuppers
<jtv> And I take it the enlistment program sends all of the node's MAC addresses, in which case we don't even need to worry about which one we use.
<Daviey> right!
<Daviey> maas-enlist defaults to pushing all mac's
<jtv> So then whatever code picks up the metadata URL from your static setup will append one of its mac addresses as a path component, and presto.
<Daviey> \o/
<Daviey> sounds like a plan
<jtv> (And it'll skip the oauth bit, but that much was obvious)
<jtv> I can add a setting for just the dev/demo configs.
<Daviey> jtv: you rock my world.
<jtv> Well let's wait until it works.  :)
<jtv> Daviey: I'll work it out in more detail tomorrow.  Until then I remain, your loyal servant &c.  :)
<Daviey> jtv: thanks.. nn sir
<jtv> nn!
<cheez0r> hey folks, this isn't a dev question, but I'm struggling with MaaS/dnsmasq right now and I was hoping you might be able to point me in the right direction.
<cheez0r> I'm trying to stand up a new MaaS, I add 11 nodes to the MaaS, but when I run juju bootstrap, I get an ssh error. When I run verbose, it is trying to connect to the hostname of the node, which fails to resolve.
<cheez0r> Shouldn't MaaS be adding the hostname to cobbler to add to dnsmasq automatically when the node is finished commissioning?
<cheez0r> The strangeness comes in where one of my 11 blades seems to resolve but the other 10 do not.
<roaksoax> cheez0r: can you ping the bootstrap's node hostname from where you are running juju?
<cheez0r> roaksoax: no. that's part of the problem. It should have automatically been added via cobbler when I commissioned the node.
<cheez0r> I'm trying to understand both why the dns hostname adds aren't working and where I could manually add them
<roaksoax> cheez0r: are you running an external dns server or are you using maas-dhcp?
<cheez0r> using maas-dhcp
<cheez0r> because of that I was under the impression that MaaS would add hostname entries for each of the nodes as they were commissioned.
<roaksoax> cheez0r: is the machine where you are running juju, using the maas server as DNS server?
<cheez0r> yes, should be
<cheez0r> right- I'm following the howto at https://wiki.ubuntu.com/ServerTeam/MAAS/Juju#MAAS:_getting_started_with_Juju
<roaksoax> cheez0r: make sure that the machine were you are running juju from is using the maas server as DNS server (you probably have to do it manually)
<cheez0r> how can I do that- the node is set as 'ready' in MaaS, it's fresh out of commissioning. I'm running juju bootstrap from the MaaS node.
<cheez0r> It looks like a resolution from the MaaS node issue, not from the node itself
<cheez0r> and none of the nodes resolve from the MaaS node except for one, for whatever reason.
<cheez0r> They were all added identically.
<roaksoax> cheez0r: I think I've hit the issue before but can't recall how to fix it but just editing /etc/resolve.conf and nameserver W.X.Y.Z
<cheez0r> right, except that I don't know the IP addresses MaaS has handed out to my servers to manually configure their hostnames
<cheez0r> it's resolving via dnsmasq; I just need to add the hostnames to the dnsmasq configuration
<roaksoax> cheez0r: that's added automatically, unless there's a bug in MAAS where it is not updating cobbler correctly
<roaksoax> cheez0r: try: sudo cobbler sync
<cheez0r> right, which I think is the problem
<cheez0r> I've done cobbler sync about a hundred times so far
<cheez0r> :p
<roaksoax> uhmmm i'll try to reproduce as soon as I can to further troubleshoot this
<cheez0r> Do you know where the cobbler or dnsmasq configuration is located for the hostnames?
<cheez0r> I can try manually adding them and see if it gets me past this point
<roaksoax> cheez0r: /var/lib/cobbler/cobbler_hosts
<roaksoax> cheez0r: what does that file show?
<roaksoax> cheez0r: does it show hostname/ip combination?
<cheez0r> it's empty.
<roaksoax> cheez0r: so if you do: sudo cobbler system dumpvars --name node-XYZ(of whatever node) | grep dns
<cheez0r> no output returned
<cheez0r> system not found <hostname>
<roaksoax> cheez0r: so maybe MAAS is not setting the hostname
<cheez0r> doesn't matter if I use the hostnames I specified or the default
<roaksoax> cheez0r: you could try: sudo cobbler system edit --name node-XYZ --dns-name node-hostname
<roaksoax> and then sudo cobbler sync
<cheez0r> ok let me try that
<roaksoax> and try to ping it by hostname
<cheez0r> how can I get a list of nodes cobbler knows about?
<roaksoax> cheez0r: sudo cobbler system list
<cheez0r> that's really odd, they all seem to have the same mac address
<cheez0r> names are node-<stuff>-d485645878c8
<cheez0r> the systems in that list seem to reflect the correct dns_name in the config it outputs
<cheez0r> okay Im guessing this is a bug related to my specifying hostnames when adding nodes to MaaS
<roaksoax> yes apparently so
<cheez0r> let me delete and readd a node with no specified hostname and see what it does
<roaksoax> i'd have to setup a physical cluster to troubleshoot
<cheez0r> not asking you to do that, but thanks for the thought ;)
<roaksoax> cheez0r: i know :) I just need to make sure it works well :)
<cheez0r> well, I'm specifying amd64 arch and was specifiying a hostname in the format city_name-dc_name-enclosure#-blade#
<cheez0r> might be throwing errors on all of the hyphens or some such, but I dunno
<roaksoax> for hostnames?
<cheez0r> yes
<roaksoax> cheez0r: hostnames don't accept underscores, but does accept hypens
<cheez0r> no underscores in the actual names
<cheez0r> like paris-champselysee-enclosure1-blade12
<roaksoax> alright, yeah I think it might be realated to not updating cobbler correctly
<cheez0r> well I've got one node re-commissioning with the default hostname node-<MAC> so we'll see if that fixes it
<roaksoax> cool
<cheez0r> no change- the newly recommissioned node with the default hostname is still not resolving.
<cheez0r> cobbler_hosts is still empty
<cheez0r> the newly recommissioned node still shows up with a funky name with cobbler system list
<cheez0r> re-commissioning with architecture set to i386 to see if that fixes anything
<cheez0r> no change, still doesn't resolve
<Soekris> Hello I've run juju bootstrap. This was good. I have seen apt-get running. But juju status doens work. I get ERROR Invalid SSH key. ssh ubuntu@hostname works
<Soekris> http://paste.ubuntu.com/1016722/
<Soekris> Where goes it wrong at my setup ?
<cheez0r> Soekris: the hostname resolves?
<cheez0r> or are you ssh ubuntu@IP?
<Soekris> cheez0r: it resolves
<Soekris> ubuntu@s1-cl1-maas works
<Soekris> ssh ubuntu@s1-cl1-maas works
<cheez0r> hrm interesting
<cheez0r> I have the same SSH issue but mine is because cobbler isn't adding the DNS names for some rason
<Soekris> I have configued dns and dhcp on a other server
<Soekris> It's strange that something like so easy can be so difficult :D
<cheez0r> agreed
<Soekris> cheez0r: is it for production or for test purposel ?
<cheez0r> kind of both
<Soekris> you can quick make the entries in the /etc/hosts
<cheez0r> yeah I think that'll be my workaround right now
<Soekris> Strange one of the two servers are showing in the juju status. But how it's a MAAS mistery
#maas 2012-06-01
<bigjools> jtv: what else did you need to know?
<jtv> What exactly it is we're not doing.  Multiple worker packages?
<bigjools> jtv: right, single worker per rack
<jtv> Regardless of arch?  Good.
<bigjools> indeed
<bigjools> it's a little mad to have multiple ones
<bigjools> but I suspect that thought arose due to intimate lack of knowledge of our new architecture
<jtv> Nicely put
<bigjools> s/intimate lack of/lack of intimate/
<jtv> Shame.  You had a nice pattern going there.
<jtv> Oh, about the anonymous-metadata issue: my preference is to have a config setting (dev/demo, not production) that would open up a mirror to the metadata tree keyed by MAC address: /metadata-by-mac/<version>/<mac>/â¦
<jtv> The way I see it, the MAC address is about the most reliable knowledge the node has at the beginning of the provisioning stage, and conversely MAAS knows all its MAC addresses, so it makes a good key.
<bigjools> jtv: I asked Francis for confirmation on this bug
<bigjools> ie. why was it escalated
<bigjools> and we'll fix that problem now rather than one that doesn't exist (yet)
<jtv> What is the existing problem and what is the non-existing problem?
<bigjools> exactly what I want to clarify
<jtv> Soâ¦ put this one on ice until we have an answer?  Or start doing work that'll be needed either way?
<jtv> Actually, one thing that needs to be done either way is proper coffee.
<bigjools> jtv: if you can see common work, by all means do it
<jtv> but
<jtv> no but?
<allenap> Morning.
<jtv> Hi there allenap
<allenap> \o jtv
<cheez0r> criminy, on a freshly loaded MaaS server node I can't even get cobbler system dumpvars to run
<cheez0r> "cannot marshal None unless allow_none is enabled"
<allenap> Daviey: Do you have experience with iPXE? James Page was saying that it's Good Stuff.
<Daviey> it is very good stuff.
<Daviey> And i do.
<Daviey> allenap: The 'maas without controlling infrastructure' story, was initially going to be ipxe based.
<allenap> Daviey: I'm not familiar with that story, unless I know it under a different name.
<Daviey> allenap: user doesn't have control over dhcp
<allenap> Daviey: Ah, right. Is there any reason to not use ipxe for all scenarios?
<Daviey> allenap: doesn't work on arm
<allenap> Daviey: Blast.
<Daviey> it would make me very happy if it did
<Daviey> would provide a nice abstraction
<Daviey> and support iscsi out of the box
<Aaton> so anyone have a good write up for setting up maas in a virtualbox test env.  Setting up the network right is the main thing I'm looking for
<robbiew> Aaton: marcoceppi might
<marcoceppi> Aaton: try this: http://marcoceppi.com/2012/05/juju-maas-virtualbox/ it's not a perfect experience, but you can get MaaS running
<hazmat> marcoceppi, nice!
<Aaton> thanks!
<marcoceppi> Aaton: it's kind of a chronicling, so I'd recommend reading the whole thing through before starting, half-way through I got smart and figured a bunch of things out. Wiki page for VirtualBox should be done this weekend
<Aaton> marcoceppi: yea, just read it over. so did you create a second interface. I want to connect to the bridged interface but boot the other servers inside virtualbox. I don't have control of the dhcp server.
<marcoceppi> Aaton: I didn't create a second interface, I used a VM to put the MaaS main node on
<marcoceppi> If you install the main node as a VM, then install maas-dhcp ( didn't have access to my offices dhcp server, but it worked anyways ) it *should* work
<Aaton> marcoceppi: I'll be giving that a try
#maas 2013-05-27
<bazzer> why does maas-import-isos fail every single time on a fresh 12.04.2 server with errors complaining about failed to find directory /var/lib/tftpboot and /var/log/syslog has apparmor errors of DENIED when cobbler tried to mkdir /var/lib/tftpboot?
#maas 2013-05-28
<bigjools> bazzer: can you try the version in the PPA here instead (https://code.launchpad.net/~maas-maintainers/+archive/stable), the maas shipped with 12.04 is broken in quite a few places.
<bigjools> the one in 12.04 is due to be replaced with the PPA version soon
<bazzer> bigjools: huh good to know...i tried MaaS out once when 12.04 first shipped...trying to pick it up again
<bazzer> bigjools: i got fed up earlier and created /var/lib/tftpboot :)
<bigjools> bazzer: cool, let me know how you get on
<bigjools> the latest maas in raring has a ton more features
<bazzer> bigjools: it completed whatever it was up to doing on the imports...so it can't be a bad thing :)
<bazzer> bigjools: i'm adding th repository now...i just thought sticking with 12.04 with it being a LTS would be the way to go for a server...
<bigjools> normally true, yes :)
<bigjools> the new version is also in -proposed if you want to enable it
<bazzer> i (somehow) semi got it working the other day at work...nodes stuck in commissioning...clients would boot and hang complaining of iscsi targets *shrug*
<bazzer> bigjools: hey that pulled in isc-dhcp-server...not using dnsmasq now?
<bigjools> no
<bigjools> bind9 & isc now
<bazzer> sweet...i'm old school dhcp and bind...dnsmasq confusing me to no end when i'm quickly trying to do something
<bigjools> yeah, dnsmasq is more a desktop solution
<bazzer> yeah, i greatly hated dealing with it even if it was extremely well documented...seems too non-server-ish
<bigjools> the set up of the dhcp and dns config is all done in the maas ui now
<bazzer> nice
<bigjools> it's all organised into a region and multiple clusters
<bigjools> each cluster has its own dhcp
<bazzer> thought that is how it was supposed to be? then again i tinkered with it for a short time like i said
<bazzer> well that sucked...had a lockup
<bazzer> bigjools: uhh, maas-import-pxe-files pulls in armhf?
<bigjools> bazzer: configure what it pulls in /etc/maas/
<bazzer> gotcha
<bazzer> didn't see the mention in the docs
<bazzer> seem commented out to me
<bazzer> ummm I just tried to enlist a node...semi success if one considers success that it booted over the network and is sitting at some login screen
<bazzer> oh wait it shutdown!
<bazzer> seems i got some FQDN issues :(
<bigjools> if it enlisted it should show up in the UI
<bazzer> yeah it is
<bigjools> are you ising IPMI?
<bazzer> as (;; connection timed out; no servers could be reached master
<bazzer> bigjools: dunno?
<bigjools> did you install maas-dhcp and maas-dns?
<bigjools> if you don't know you probably aren't :)
<bazzer> well dhcp yes before you gave me that ppa and i upgraded
<bigjools> these are new packages
<bazzer> yeah maas-dns is missing
<bigjools> you will also need to visit the cluster config page and set its details up
<bazzer> already did that
<bigjools> cool
<bigjools> you set it to manage dns as well?
<bazzer> i sure did
<bigjools> cool, so just install maas-dns and all should be well
<bazzer> i must say this now works oh so much better than the headache i had this afternoon
<bigjools> we should probably detect it's missing and warn in that page
<bigjools> yeah, the default version in 12.04 is, ummm, poor
<bazzer> yes that'd be good to do for sure
<bazzer> lmao
<bazzer> poor
<bazzer> i love that statement
<bigjools> it was demoware
<bazzer> yeah i understand that...but i love a challenge and whatnot
<bazzer> okie...trying this one more again
<bazzer> oh that "semi" working default version i got set up at work...i screwed it up so bad trying to enlist nodes that it would show one and cobbler was telling me there were 10
<bazzer> ok um how do i delete a node i accepted and its in commisioning?
<bazzer> nm
<bazzer> found it
<bazzer> well now a new problem, juju :)
<bazzer> no matching node is available
<roaksoax> rvba: still around
<roaksoax> ?
<rvba> roaksoax: yep
<roaksoax> rvba: so I'm gonna be separating the ipmi autodtection code away from enlist_userdata and user_data.template. However, we discussed with julian that we would like to "inject" that script
<roaksoax> rvba: so any ideas of how I could use/inject that script into the templating?
<roaksoax> rvba: the idea of separating the code away is to make it available in a separate package
<rvba> roaksoax: well, I see two ways to do this: one solution is to read that script in MAAS and include it in the context used to render the main template.  The other is to use tempita's inheritance mechanism to compose the template using the main template and the script.
<roaksoax> rvba: both options would require have it installed in maas region then. Ok, I'll think about what's best and let you know :)
<rvba> roaksoax: it probably needs some thinking indeed.
<roaksoax> indeed
<roaksoax> rvba: btw.. the piston fixes... they are not merged in upstream yet
<roaksoax> rvba: do you still want me to grab those and apply them?
<rvba> roaksoax: IIRC they have been there for a long time and are still not merged upstreamâ¦ right?
<roaksoax> rvba: yeah 1 has been there for months, the other just for days
<rvba> roaksoax: if that's the case then I think we should include them as patches to avoid having to wait for upstream to include them.
<roaksoax> ok
<roaksoax> my point was that we usually cherry pick patches applied upstream (and these haven't)
<roaksoax> but will take care of them
<rvba> ok thanks.
<roaksoax> rvba: btw.. https://bitbucket.org/jespern/django-piston/pull-request/25/compatibility-fix-for-json-emitter-with/diff
<roaksoax> rvba: check the commet by Vladislav
<roaksoax> rvba: do you think we should use what he commented instead of django.VERSION
<roaksoax> ?
<rvba> roaksoax: "Why soooo looong?" ?
<rvba> I think the author of that patch is a core Django dev.
<roaksoax> rvba: look at the diff, there's a comment in a diff with "try: import json except ImportError from .....
<roaksoax> rvba: instead of "import django \n if django.VERSION...."
<rvba> roaksoax: I prefer the usage of django.VERSION tbh.
<roaksoax> rvba: ok
<roaksoax> rvba: uploaded. should be available in the next couple of hours
<AskUbuntu> Install nodes error (could not find kernel image) | http://askubuntu.com/q/301150
<kmyst> quick question...i'm trying to enlist nodes that have two nics (eth0 is connected to the network the maas controller is on and eth1 is connected to another network with internet access) but my commissioning is failing i think since it only brings up eth0 so how do i get around that?
<smoser> hey...
<smoser> kmyst, it *should* be fixed to correctly load only the nic that it booted from.
<smoser> hey. in 'maas-cli admin nodes list'
<smoser> is there any way to get the user-name that has a node allocated ?
<smoser> afaiks no
<kmyst> smoser: yeah that's what i figured but the nic it boots from has no internet access so the commissioning just sits and sits then spews a ton of apt-get errors since it can't get to the internet
<smoser> ah. because it doesn't bring up the other nic
<smoser> ?
<kmyst> right
<smoser> i'd appreciate it if you opened a bug, as your use case seems quite valid to me.
<smoser> or at least i'd like to see  a thoughtful response.
<smoser> you can probably get thorugh this by adding an entry in /etc/network/interfaces inside the ephemeral image
<kmyst> which the end result is the node ultimately shuts down and goes to the ready state but well...ya know :)
<kmyst> good idea
<smoser> shoot.... that might not work*
<smoser> probably wont.
<kmyst> drat
<smoser> as we have some clever code in the initramfs that dynamically writes /etc/network/interfaces
<kmyst> ah
<smoser> based on what BOOTIF was given/
<smoser> on the initramfs kenrel command line
<smoser> let me see thoug
<kmyst> k
<kmyst> i just got done testing out the same thing on virtualbox
<kmyst> same results
<smoser> i think the easiest thing to do will be to make the commissioning user-data bring up eth1
<smoser> kmyst, what version of maas are you using ?
<kmyst> uh let me check
<smoser> you're also going to have to manually handle bringing up the nics in install also.
<smoser> as it is also going to just bring up the nic that it sees on the cmdline
<smoser> (i think)
<smoser> please open a bug
<kmyst> ya that's what it's doing for sure
<smoser> as i honestly think this is a very reasonable setup you have.
<kmyst> maas version 1.2+bzr13373-df
<kmyst> i'll definitely open a bug if that's the way to go?
<smoser> well, we definitely want a bug
<smoser> to fix it properly
<kmyst> will do then
<kmyst> 1.2+bzr1373+dfsg-0ubuntu1~12.04.1~ppa1
<kmyst> from the stable ppa
<smoser> ah. thats what i was looking for.
<kmyst> smoser: only reason i set it up that way was i didn't care if maas ran dhcp/dns but since i already have an authoritative dhcp server on the network segment i isolated maas and the nodes and just provided the nodes with an extra nic to a lan segment that has external access
<kmyst> so after experiencing the problem i fell back to testing the idea out with virtualbox...same results :)
<smoser> yeah.
<smoser> so one way that will probalby work...
<smoser> in the ephemeral images, you can add an upstrat job that runs on 'starting cloud-init'
<smoser> that just does whatever you need to do to 'ifup eth1'
<smoser> (it will be more than just 'ifup eth1' as there wont be an entry in /etc/network/interfaces for eth1, but you could just append one and then ifup)
<smoser> that make sense?
<kmyst> completely
<smoser> do that in the ephermal images, and then also make it happen in the installer
<smoser> so that when it comes up it gets your handywork aplied
<kmyst> so the disk.img in /var/lib/maas/ephemeral/precise/ephemeral/amd64/20121008 right?
<kmyst> ls
<AskUbuntu> How to preserve custom configurations across reboots when using juju and maas? | http://askubuntu.com/q/301241
<AskUbuntu> Juju deploy issue : Connection was refused by other side | http://askubuntu.com/q/301243
<kmyst> smoser: where's the installer?
<AskUbuntu> sudo maas-import-isos fails | http://askubuntu.com/q/301280
#maas 2013-05-29
<rvba> roaksoax: thanks a lot!  I'll test that sometime today.
<Bicyus> Hi there!
<Bicyus> guys i'm taking this error when installing a node from PXE:
<Bicyus> true && in-target sh -c 'f=$1; shift; echo $0 > $f &&chmod 0440 $f $* 'ubuntu ALL=(ALL) NOPASSWD: ALL ' /etc/sudoers.d/maas && wget "http://maas-ip:80/cblr/svc/op/nopxe/system/$system_name" -o /dev/null && true
<Bicyus> failed wit exit 1
<Bicyus> cant find any answer on google...
<Bicyus> any clue why this is happening?
<bigjools> Bicyus: update to the version in ppa:maas-maintainers/stable
<Bicyus> thanks bigjools i'm trying again, without automatic commisioning
<Bicyus> is that PPA production ready? i mean, we are testing to deploy openstack on a small private cloud
<bigjools> it's getting SRUed soon
<roaksoax> jtv: still around?
<jtv> roaksoax: barely!  Hi.
<jtv> I'm in Europe.
<roaksoax> jtv: was just wondering the ETA for having the support to move all the templates to 'etc' ?
<jtv> I just started on it!  Shouldn't take long.
<jtv> Actually I wanted to ask you one thing: would it be OK for the package just to install an entire directory etc/maas/templates as /etc/maas/templates, in one go?  Or would you want me to break that up further?
<roaksoax> jtv: ok cool, yeah I just saw the ETC_DIRECTORY thing, but I was wondeirng if it would make sense to continue to do things like: POWER_TEMPLATE_DIR, PXE_TEMPLATE_DIR, etc
<roaksoax> jtv: i was thinking that the best would be: /etc/maas/templates/<power,pxe,etc>/*.template
<roaksoax> and we also need /etc/maas/preseeds/<all preseeds>
<roaksoax> so a PRESEED_TEMPLATE_DIR variable would be nice too (if it doesn't yet exist)
<jtv> AFAICT what you describe for the templates is exactly what I documented.
<roaksoax> jtv: yeah just reading :)
<jtv> POWER_TEMPLATE_DIR etc. will all be gone â with a provision for previous settings, I suppose, but people won't be invited to edit them individually any more.
<roaksoax> jtv: I think the location of the template dir's should be configurable for each of the templates
<jtv> Can you give me an example of when that would be useful?
<roaksoax> jtv: Personally, I have stronger preference for POWER_TEMPLATE_DIR, PXE_TEMPLATE_DIR, PRESEED_TEMPLATE_DIR, etc
<roaksoax> jtv: that way you are providing more flexibility
<roaksoax> smoser: ^^ thohgts?
<roaksoax> thoughts?
<jtv> I understand the flexibility is nice, but it won't come for free â it means more maintenance burden, more to keep in mind.  So it's important I think to keep an eye on practical uses.
<roaksoax> jtv: if that's easier for you to maintain then I'd suggest TEMPLATE_DIR = '/etc/maas/templates' and then it goes from there
<roaksoax> jtv: however, don;'t we currently have a template dir for each? say the code nows where to find DHCP temapltes, where to find DNS temapltes, etc etc
<roaksoax> so that would mean only extending that to be configurable
<jtv> Writing it is easy â I'm not worried about that.  But this kind of thing does add friction to maintenance.  Can you give me an example of how you'd use this?
<jtv> I need to understand what the purpose is.
<roaksoax> jtv: the purpose is the same as why we currently have POWER_CONFIG_DIR/POWER_TEMPLATES_DIR, DNS_CONFIG_DIR, etc
<roaksoax> it allows more flexibility for the packager
<roaksoax> for the user
<roaksoax> etc
<roaksoax> jtv: that way the user can create custom directory *names* to store their templates
<roaksoax> jtv: so if the user wants, they can do: POWER_TEMPLATES_DIR = /var/lib/whatever/power_management_templates/
<jtv> How do you manage upgrades to those?
<roaksoax> instead of being restricted to only "power"
<jtv> In that kind of situation, if the packaged version changes, how do you detect & resolve the difference?
<roaksoax> jtv: you don't need to, packaging will default to /etc/maas/templates/{power,pxe,etc}
<roaksoax> jtv: TBH, I personally think separate DIR vars for each dir is best due to flexibility, but if it is more convenient to keep them under one single variable, i'm fine with that too
<jtv> I do think it simplifies the celery config, which is quite awkward to get one's head around.
<jtv> There will be separate directories for different kinds of templates, so you can still mess with the config at that level if you really want.
<roaksoax> jtv: well the celery config already has the concept of separate DIR's
<jtv> Only some.
<roaksoax> jtv: and yes, the idea is that configs are meant to be user configurable
<roaksoax> but if it is one dir, then make it TEMPLATES_DIR instead of ETC_DIRECTORY
<jtv> BTW right now the only templates directory that I see a configuration option for is the one for the power templates.
<roaksoax> jtv: i know :) that's why I thought we were gonna add 1 var per temaplte type
<jtv> I guess you could always create your own version and then softlink to the regular one in /etc for the things you don't want to change...
<roaksoax> jtv: so to make things simpler, I think we should just go for TEMPLATES_DIR (instead of ETC_DIRECTORY)
<roaksoax> jtv: so that would allow us to do: 'etc/maas/templates' or 'etc/maas/xyz', etc
<jtv> Yes, we can do that...  It'll cost us one simplification I was hoping to make, but I can live with that.
<roaksoax> jtv: so the idea is not really that templates are to be limited to live in ETC< but rather, that you can decide where you want your templates regardless of whether it is on /etc/<someplace> or not
<roaksoax> jtv: and in the case of power, note that power has both, templates and config, in that case, it does make sense to have separate variables for the templates and the config
<jtv> Not a pleasant thought to have separately configurable template and config directories just for the power methods.  :/
<jtv> Is it really that much of a problem to have a single directory that's mostly softlinks to the packaged version, with a few customized files?
<roaksoax> no it is not
<roaksoax> again i'm fine with TEMPLATES_DIRECTORY = '/etc/maas/templates'
<roaksoax> i mean default can be TEMPLATES_DIRECTORY = None which will look for the templates the place tjhey are now
<roaksoax> but if set, then it will look for the templates in the currently set directory
<jtv> We can't look for the templates the place they are now â the point of all this is to move them.
<jtv> Right now, the code just looks in its own location.
<jtv> The move takes that easy default away.
<roaksoax> right
<roaksoax> I think there should *always* be a default
<roaksoax> that you can configure later on
<roaksoax> currently, the POWER_TEMPLATE_DIR allows you to have a sane default, but also allows you to move the templates and place them something else
<roaksoax> and I think that's how it should be designed
<roaksoax> the idea is to allow the change of location of the templates, making it configurable. The idea is not to restrict them to 'etc'
<jtv> I do think it makes sense to keep all those customized directories together, so that there's a single standard layout wherever you put them.
<jtv> Separating the power templates from the power config does make that harder.
<roaksoax> jtv: i would be fine having the config reside at the same location of the template
<roaksoax> jtv: at the end, the file extension will differ file.{template,config}
<jtv> That would simplify it a lot...  technically it's not a "template" but it seems close enough.
<roaksoax> yeah
<jtv> It could be argued that it's not "config" either because it's just stuff that's got to be sent over, not something that controls MAAS's behaviour.
<roaksoax> yeah
<roaksoax> in reality it is a config template
<jtv> Okay, I'll revise tomorrow to make the split at the /etc/maas/templates level.  Can I make maas-cluster-controller.install just install that whole directory in one go?
<roaksoax> jtv: yes
<jtv> Great.  Thanks.
<jtv> And now I'll be off!
<roaksoax> have a good one
<jtv> nn
<kumquat> hi there everyone.  i hate to ruin the silence by asking a question, but i'm running 1.3 in development at the moment and my nodes time out when trying to access metadata during cloud-init and after pxeboot (169.254.169.254/meta-data/instance-id/ timeout).  i don't have any oauth credentials defined in the enlistment preseed; could this be the source of the problem?
#maas 2013-05-30
<Bicyus> Hello everyone!
<Bicyus> guys wich package is responsible for DHCP in PPA maas-maintainers/stable?
<Bicyus> can't make it to work on 12.04
<roaksoax> Bicyus: maas-dhcp
<roaksoax> install that and maas-dns
<Bicyus> roaksoax: Thanks. they are installed but cant see wich procces is responsible for DHCP,
<Bicyus> nothing on  "ps aux|grep dhcp" after "service maas-dhcp restart"
<roaksoax> Bicyus: did you configure dchp/dns on the webui?
<Bicyus> webUI? seem there is nothing about dhco/dns!
<Bicyus> double checking...
<roaksoax> Bicyus: Go to the settings and there you will see the cluster controller, edit the cluster controller and config the DNS/DHCP stuff for an interfae of it
<Bicyus> thanks roaksoax!
<Bicyus> i'm reinstalling from 0 again ;-)
<smoser> roaksoax, is there a way to move a node from 'available' to commissioning?
<smoser> oh. wow, it looks like rvba is near there
<smoser> https://code.launchpad.net/~rvb/maas/en-masse-delete/+merge/166513
<smoser> rvba, can you confirm that ? that your branch there plans on letting me move nodes into commissioning (or any valid state transition even!) from the api ?
<rvba> smoser: that branch implements bulk actions in the UI.  But all the logic is in a form that will be reÃ¼sed by the API to let you do the same thing using the API.  I'll do that as a follow-up branch.
<smoser> so that is "in plan" ?
<smoser> is there a blueprint task or something  for that ? or a bug ?
<smoser> i can open a bug if you'd like
<rvba> There is a blueprint.
<roaksoax> smoser: yes it is if you play with the db
<rvba> smoser: I thought it was mentioned on the blueprint https://blueprints.launchpad.net/cdo-dev/+spec/s-cloud-maas-scale-ui â¦ but it's not there :)
<rvba> smoser: if everything goes according to plan, I'll do the API stuff tomorrow.
<smoser> rvba, hoorah
<rvba> smoser: would you mind filing a bug still?  I have the impression that the method you need is a bit different from what I'm planning to add.
 * rvba is off.
<smoser> rvba, bug 1185897
<ubot5> bug 1185897 in MAAS "expose ability to re-commission node in api and cli" [Undecided,New] https://launchpad.net/bugs/1185897
<Bicyus> bye guys! ;-) have a nice day!
#maas 2013-05-31
<roaksoax> jtv:
<roaksoax> jtv: ping
<jtv> Hi roaksoax
<roaksoax> jtv: howdy!! is the MAAS_CONFIG_DIR only for the provisioningserver?
<jtv> No, it's meant for all of MAAS.
<jtv> Got a use for it?  :)
<roaksoax> jtv: well I think that's kind of non desired
<roaksoax> jtv: each service/daemon should have its own variable to find its config
<jtv> Note that this isn't what we talked about with custom templates.
<jtv> It helps, but it's more of a first step.
<roaksoax> jtv: yes I understand! However, I do still believe that each service should be able to find its config on its own
<jtv> If we move something from the source tree to /etc, then that introduces the very basic problem of how to find the right one when we're running from branch.
<roaksoax> jtv: if you implement thjis, this now means that you will have to deprecate celery_common
<jtv> Why that?
<roaksoax> jtv: no nevermind about that, but yo will have to make the MAAS_CONFIG_DIR a common variuable for the cluster and the region
<roaksoax> jtv: it would have to work the same way it works with celerycommon_config.py
<roaksoax> isn't that adding more complexity than required?
<jtv> In normal test/demo usage where you run from a single branch, it would be a single variable yes.
<roaksoax> jtv: yes, but MAAS in real world will not run in one single node, would it?
<jtv> The celery config works slightly differently.  It makes a distinction between a production-or-test run vs. a demo run.
<jtv> In the real world you don't run MAAS from a branch.  You run it from an installed package.
<jtv> That's the case where you _don't_ have MAAS_CONFIG_DIR.
<jtv> But when you're running tests or a demo, you _do_ want MAAS_CONFIG_DIR.
<jtv> If you wanted to run a demo cluster and a demo region with different setups, on the same machine, you would either run them from separate branches, or override MAAS_CONFIG_DIR for one or the other, to point to a different config directory.
<jtv> Not sure you'd want that, of course.
<roaksoax> jtv: right, so what if you set MAAS_CONFIG_DIR in a system where you instlaled MAAS via package to use /user/share
<roaksoax> jtv: then the code will pick that up
<roaksoax> jtv: and use that as a config dir
<roaksoax> right?
<jtv> You could make it pick that up if you wanted to, I suppose, but would you want to?
<jtv> If you install a MAAS package, you will simply not have a MAAS_CONFIG_DIR.  If you do set one, that allows you to run one or more of the MAAS processes from a different config.
<roaksoax> jtv: ok
<roaksoax> i guess what I'm trying to understand is if this is also meant to be used when we package this up
<jtv> The main thing here is to provide a uniform way for MAAS code to ask, "where is /etc/maas?" â and get a branch-local version when running from a branch.
<jtv> No, it's not meant to be used by a package.
<roaksoax> jtv: right, but "/etc/maas" is a packaging convinience
<roaksoax> is a safe default
<jtv> Yes, that remains unchanged.
<jtv> It's normally only when running from a branch, instead of from a packaged install, that you'll want MAAS_CONFIG_DIR.
<jtv> So it creates a kind of "lightweight chroot" effect for (1) tests, and (2) demos.
<jtv> The problem with using the celery config for this was that it throws tests and production into one kind of config, and demos into another.  Not the right fit.
<roaksoax> jtv: right, but what I mean is that currently we run the pserv by passing it the --config-file option that tells where to load the pserv.yaml right?
<jtv> It's optional, I think.
<roaksoax> no
<roaksoax> that's how the daemon is run
<roaksoax> and that's how it should be run
<roaksoax> daemons should allow you to tell them where the configuration is and if not sent, then it would look for a sane default
<jtv> So you're saying it's not looking for a sane default?  That seems wrong.
<jtv> And we do in fact have code in there that looks for a sane default, /etc/maas/pserv.yaml.
<jtv> Unfortunately it *also* looks for an insane default, ./etc/pserv.yaml.
<jtv> That's for branch runs.
<jtv> With MAAS_CONFIG_DIR, that will simplify to: if an explicit file was selected, use that.  Otherwise, look for $MAAS_CONFIG_DIR/pserv.yaml.
<roaksoax> jtv: ok so MAAS_CONFIG_DIR is only for branch runs and not when you actually *install* and run maas?
<jtv> Exactly.
<roaksoax> jtv: ok, so as long as nothing else changes, then I'm fine with that
<jtv> Good, because I think the code already landed.  :)
<roaksoax> because I also read that this is the basis for dropping POWER_{CONFIG,TEMPLATE}_DIR
<jtv> Eventually, yes.  Not exactly dropping it, I think, because that'd break compatibility.  But the important point is that templates by  default would live in $MAAS_CONFIG_DIR/templates.
<jtv> If we don't have $MAAS_CONFIG_DIR, then we'd have to make that "if we're running from a branch, use <branch>/etc/maas/templates.  Otherwise, use /etc/maas/templates."
<roaksoax> right
<jtv> So this change allows us to say: the default location for templates is $MAAS_CONFIG_DIR/templates.  Once we have that, we can allow people to customize that location.
<roaksoax> right
<jtv> And then if we do that, you can move the entire template directory (and softlink the parts that you don't want to change, I guess) and _then_ there should be no more need for those fine-grained configuration items.
<jtv> So yes, this is the basis, but it's not a simple direct step.  :)
<roaksoax> jtv: ok so as far as the templates then
<roaksoax> jtv: this means that MAAS_CONFIG_DIR/templates would also work in production environments and not only when running from a branch right?
<jtv> Right â and in a production environment it would be /etc/maas/templates, whereas in a branch it would be <branch>/etc/maas/templates.
<roaksoax> right
<roaksoax> which means that in a production environment
<roaksoax> we still have to set MAAS_CONFIG_DIR
<jtv> You don't have to  set it in a production environment â unless you're trying to run MAAS with a different config directory.  But as discussed we'd rather do that for just the templates.
<roaksoax> exactly
<roaksoax> that's my point
<roaksoax> my point is that by default MAAS_CONFIG_DIR will default to /etc/maas and will make all of the daemons/services to look for their configs in MAAS_CONFIG_DIR
<roaksoax> this means that MAAS_CONFIG_DIR is a common variable for both the cluster and the region
<jtv> Yes, that won't change.
<jtv> Right now it's hard-coded.  You're not meant to customize it in production, although you could if you wanted to.
<roaksoax> jtv: right, that's my point
<jtv> So, no worries then?  Great!
<roaksoax> jtv: regardless that MAAS_CONFIG_DIR is set to /etc/maas, then a daemon/service should be allowed to specify *where* is config file is located
<jtv> Yes, separate issue.
<jtv> You're currently keeping me from realizing that dream.  :)
<jtv> Okay, not fair â what I mean is, there's more groundwork to be laid so that we can move templates to /etc, and there's more design work to be done to satisfy various people's requirements.  And I'm eagerly implementing the groundwork right now.
<roaksoax> jtv: ok, well to keep it simply, i'm just concerned that implementing this will affect allowing each service/dameon specify where its config are located
<jtv> If you really wanted to tweak  this per daemon, then just pass each daemon a different value for the variable.  It's an environment variable, not a configuration item.  But it's not really meant for that â it's an implementation detail, if you will.
<roaksoax> ok
<AskUbuntu> Maas enlist nodes with pxe timeout | http://askubuntu.com/q/302479
<kumquat> hi all, when cloud-init runs its final userdata on a VM node, i'm getting the "error inserting ipmi_si: no such device" fatal exit after installing freeipmi-tools and trying to set up.
<kumquat> any help would be appreciated, since the node is hanging on commissioning and keeps sending messages to nodegroups/register
<roaksoax> kumquat: do your nodes have ipmi cards?
<kumquat> ah, no. the fatal error also didn't impact the bootup - after a while the node registers itself successfully anyway.
<roaksoax> kumquat: yeah
<roaksoax> rvba: still around by any chance?
<jtv> smoser: still here?
 * smoser missed jtv
#maas 2014-05-26
<dchem> Hello everyone
<AskUbuntu> MaaS + JuJu on a single PC, MaaS as host for VM Nodes (LXC + KVM) | http://askubuntu.com/q/472230
<rvba> bigjools: jtv:  I just installed MAAS 1.5.1+bzr2269-0ubuntu0.1 on a canonistack instance and the 'architecture' dropdown isn't populated ('power_type' is)â¦ The result is that I can't add a nodeâ¦
<jtv> That means RPC hasn't bootstrapped, doesn't it?
<jtv> Ah, no, arch is derived from boot images â power type is gathered over RPC.
<jtv> So... no boot images?
<rvba> Ah, that's right, I thought the import had been doneâ¦ but it failed for some reasonâ¦
<bigjools> rvba: doesn't it say something about boot images?
<rvba> bigjools: yes, it was the problem indeed.  I was messing with the import config (in the DB) and the import script failed.
<rvba> My fault.
<jtv> Oh dear.  Got a bunch of spurious test failures in test_ucsm.py, and the setup is undocumented & hard to make sense of.
<jtv> And when I say "spurious" I mean "one-off."  They may be justified or they may not be.
<jtv> jhobbs: that looks like a bug in get_first_booter â I think it takes the alphabetically first item, when it's meant to take the numerically first item.  That test setup really needs some documentation, and if it matters at all, we need to test against this.
<jtv> Ordering is tricky.
<jtv> Adage: write your code as if it's going to be maintained by a lazy, irritable axe murderer who lives next door from you; write your tests as if the code is going to be written by a smartarse whom you'd love to prove wrong in any way you can.
<jtv> The worst things in humans can bring out the best things in programming.
<jtv> jhobbs: filed 1323307
<jtv> Bug 1323307, that is.
<ubot5> bug 1323307 in MAAS "Spurious test failure: get_first_booter sorts lexicographically, test assumes numerically" [High,Triaged] https://launchpad.net/bugs/1323307
<jtv> rvba: thanks for that lint branch â I had the same branch written up, but got held up by that spurious test failure ^
<rvba> jtv: heh
<jtv> rvba: you have some reviews.  Blocked one on a question.
<rvba> jtv: on it
<jtv> rvba: I didn't tag that bug as "tests" because it may simply be a bug in the code which the tests occasionally expose.
<rvba> jtv: I see; judging by your explanations, if seems likely that this is a bug in the test itself but you're right, it's probably too early to tell.
<jtv> I'm really not sure which is more likely...  the function under test is clearly expecting an ordinal, as if it's trying to extract a number from the string.
<jtv> Some very easy branches up for review:
<jtv> https://code.launchpad.net/~jtv/maas/resources-message-tweaks/+merge/220992
<jtv> https://code.launchpad.net/~jtv/maas/import-boot-images-with-mandatory-sources/+merge/220993
#maas 2014-05-27
<jtv> Large branch up for review: https://code.launchpad.net/~jtv/maas/wean/+merge/221030
<rvba> jtv: I'll take it.
<jtv> Thanks!
<jtv> Small documentation branch for review: https://code.launchpad.net/~jtv/maas/doc-import-sources-option/+merge/221037
 * jtv re-reviews rvba's branch
<rvba> Thanks.
<rvba> jtv: I just built a package from your "wean" branch and, when I run the import script, it still defaults to using /etc/maas/bootresources.yamlâ¦ I guess I did something wrongâ¦
<jtv> Lemme look...  May be a merge problem.
<jtv> rvba: are you quite sure you've got my branch?  r2374?  I see no way it could be reading that file.
<rvba> jtv: yes, that's the right branch, revision 2374.
<rvba> jtv: arg!  I think the package from trusty took over :/
<rvba> Damn.
<jtv> That'd do it.
<rvba> bigjools: shouldn't the packaging reference 1.5.1 (or even 1.5.2) now that 1.5.1 has been released?
<rvba> In debian/changelog I mean.
<bigjools> rvba: it does
<bigjools> 1.5.3 in fact :)
<rvba> bigjools: I seeâ¦ the Trusty packaging.
<bigjools> see the packaging.trusty branch
<bigjools> the mainline needs updating to 1.6
<rvba> So packaging should reference 1.6
<rvba> Right :)
<bigjools> yes
 * rvba updates.
<bigjools> thanks
<bigjools> needs doing sooner so there's no archive conflicts
<rvba> I couldn't agree more.
<rvba> https://code.launchpad.net/~rvb/maas/packaging-1.6/+merge/221042
<rvba> jtv: now I've got the right package instead and I get this when I run the import script: http://paste.ubuntu.com/7528019/
<bigjools> rvba: just bump the version
<rvba> bigjools: so, what does "New upstream release" means?
<bigjools> rvba: exactly what it says!
<bigjools> there has been no release on the 1.6 branch
<bigjools> (aka trunk})
<rvba> bigjools: ah, okay, I see.
<jtv> rvba: funnily enough I was just writing an MP for the fix for that...  Graham left a small bit out there.  The --sources-list option is mandatory, and the way this is handled was broken.
<jtv> rvba: https://code.launchpad.net/~jtv/maas/mandatory-sources-list/+merge/221050
<rvba> jtv: okay, I guess I'll review that now :)
<jtv> Thanks.  :)
<jtv> Super-easy branch for review: https://code.launchpad.net/~jtv/maas/don_t-talk-about-the-bootresources.yaml/+merge/221053
<rvba> jtv: on it
<jtv> Thanks.  By the way, I documented the sources yaml in the manpage, in an earlier branch.  :)
<rvba> jtv: ah cool.  Maybe we should add a reference to that in what the user sees when he runs `<import-script> -h`.
<jtv> Good idea.
<rvba> I'll do it
<jtv> I'm seeing some landing failures because of a rabbit error.  :(
<jtv> Whoopsie.  Another Rabbit error, with clearer output this time: out of memory!
<rvba> jtv: small branch up for review if you feel like it: https://code.launchpad.net/~rvb/maas/bootsource-doc-fix/+merge/221091
<jtv> rvba: done
<rvba> Thanks.
<jtv> rvba: I have two branches up for review as well.  And with that I leave you for the night.  :)
<rvba> jtv: I'll have a look now.
<onezero1> With maas 1.5 I can no longer set           Default distro series used for commissioning
<onezero1> The only option is Trusty
<roaksoax> onezero1: that's correct
<onezero1> Is this change documented somewhere I missed?  Why the change?
<bigjools> onezero1: it changed because that is now the minimum series you need for commissioning, it has dependencies only in trusty
<bigjools> it has no bearing on what you actually deploy
<jerobaal> hello
<jerobaal> i need a book for ubuntu 14.04 mass and juju for a cloud lab
<jerobaal> ?\
#maas 2014-05-28
<bigjools> jtv, rvba: did either of you have a chance to look at my schema changes?
<jtv> Not yet, not yet...
<rvba> bigjools: not yet :/
<bigjools> no worries, I need allenap to look too anyway :)
<bigjools> I made a load of changes to the design doc
<jtv> There goes the lander again!  Something is wrong.
<jtv> Notice how many rabbit brokers get reported when this happens?  I wonder if we're leaking any.
<rvba> jtv: looks like you got a different error this time!  FAIL: maasserver.tests.test_js.YUIUnitTestsLocal.test_YUI3_unit_tests
<jtv> Gah!  Three in one branch!
<rvba> I suspect there is something wrong with the lander's machine.
<jtv> Yes, that last one sounds like it might be related to memory leaks also.
<jtv> After all, browser tests are normally the most taxing for the system.
<rvba> Yep.
<rvba> jtv: Branch landed \o/
<jtv> rvba: thanks â I hope the G figures it out because this is intowewwabwe
<jtv> I shouldn't have named the branch after the Punic wars â those took several attempts too.
<AskUbuntu> Unable to access MAAS nodes | http://askubuntu.com/q/473350
<libsysguy> Hi all. I was wondering if someone could help me troubleshoot.  I have a fresh maas install and I am getting a metric *ton* of oops logs when pxe tries to run on the nodes.  I generated an sos report and put it here: https://drive.google.com/file/d/0BwO0l99GoKIxeUFxTHgxaGR0Sk0/edit?usp=sharing
<libsysguy> but mostly the oops logs just repeat Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure.
<libsysguy> actually guys, I just got it.  Thanks anyway
<ophuk> is it possible to have the MAAS management interface be different than the PXE/Provision interface? Say I want to control the box through the web interface through 172.16.103.92 and I want the cluster to work through a second ethernet interface with an ip of 10.0.0.1. Is this possible?
<jtv> ophuk: if you're talking about power management through e.g. IPMI, then yes, it's a matter of choosing the node's "power address."
<jtv> That's for turning the node on or off.  Is that the type of control you have in mind?
<jtv> AMT should be able to do the same, from what I hear, although by default it simply sits on the same interface.
<ophuk> jtv: not really. I have one box with two network interfaces, one public and one private. I want to be able to access the web interface through the public interface and I want the private interface there for the PXE, DHCP, and DNS for the cluster
<jtv> Oh, we'er talking about the network interfaces _on the server_?
<ophuk> jtv: yes, sorry
<jtv> Sorry, I thought you meant the network interfaces on the node.
<jtv> Yes, those are meant to be different things (which can of course be the same NIC if you want).
<ophuk> jtv: no worries. I feel like it should be possible but my googling is coming up with anything. It could be because I'm looking for the wrong thing
<jtv> I'll have to refresh my memory, but at a guess I would say that Apache (which runs the region controller) listens on all interfaces.
<rvba> Yes, Apache listens on all interfaces.
<jtv> In addition to that, there's a setting (again, rusty memory) for the URL which nodes should use when talking to the region controller.
<rvba> ophuk: this should work out of the box.
<jtv> The installation does have a question for "which interface on the region controller faces the nodes," but I don't think you typically get to see that.
<jtv> I'll shut up for a bit and refresh that memory.
<ophuk> jtv: ok, so if I do sudo dpkg-reconfigure maas-cluster-controller and set that to the public IP, this allows me to view the web interface from the public IP. But how do I tell it to use the other interface for PXE,
<jtv> It's sort of the other way around.
<rvba> ophuk: you want to use the private IP for that.
<jtv> What rvba said.
<jtv> The UI is on any of the server's addresses; what needs to be known is which address is reachable for the nodes.
<jtv> That's going to be the "private" address in your case.
<ophuk> rvba: ok - so that should be set to my internal IP. That's what I had originally but going to the public interface through a web browser wouldn't bring up the MAAS web ui. I was getting a server 500 error
<jtv> That sounds like you _were_ reaching it, but something else went wrong.
<rvba> What jtv said.
<jtv> Gotta love symmetry.
<ophuk> jtv: yeah apache was working - my public IP is 172.16.103.92 but going to 172.16.103.92/MAAS/ wasn't. I just attributed that to the wrong interface
<ophuk> great minds think alike:)
<jtv> Thank you... :)
<jtv> Shall we have a look at the original problem then, and see if that solves the rest?
<jtv> First place to look is the Apache error log.
<ophuk> jtv: yeah. I did the the reconfigure maas back to the internal IP
<ophuk> jtv: one sec - uploading a pastebin
<ophuk> jtv: rvba here you guys go - this is the error log, http://paste.ubuntu.com/7536756/
<ophuk> I moved the original and cleared it out and tried to reconnect. That is what I got
<rvba> Looks like the connection to RabbitMQ timed out.
<jtv> Innnteresting.
<ophuk> also as I side not I'm not terribly attached to this install, if you feel like something might of gotten messed up during the install I can do it again - it just takes forever to reimport the isos
<jtv> Well that can be solved by moving /var/lib/maas/boot-resources/cache out of the way somewhere, and moving it back into the new install.  :)
<ophuk> jtv: well that is good to hear:)
<jtv> ophuk: rabbit does tend to get upset when its IP address changes.  I don't know if that's related.
<jtv> (In this case that'd be a little silly since from the sound of it, its communication will be entirely local to the machine.)
<ophuk> jtv: it is quite possible because it originally pulled an IP from DHCP in the 172 range and then I had to change it to the public and private IP's. I don't remember an option during the install for setting up both nics
<jtv> Ah!
<jtv> Then I think that explains it.  I've had some luck in the past just editing the rabbitmq config myself, but that wasa before doing much actual work with the setup.
<ophuk> jtv: hmm...ok. Where is this config file?
<jtv> Looking for it...  Curses, I've done this before.
<rvba> jtv: I'm not sure where the piece of documentation that I'm writing belongsâ¦ The best place is probably the online documentation but you've documented the bootsource/bootsourceselection stuff in the import script's manpage so I'll have to redo/copy that part to add the relevant contextâ¦ that's a bit suboptimal.
<ophuk> jtv: /etc/rabbitmq/rabbitmq.config ? I don't have that file
<ophuk> o.O
<jtv> ophuk: same here, and I could have sworn I did just a few months back.
<jtv> This may just be a rabbit change.  Change is great, but one wishes the rest of the world would ask one's permission before making any.  :)
<ophuk> jtv: lol - is it possible to do this during install, i.e. configure a static IP on both nics?
<jtv> rvba: the man page should ideally cover only the command-line script, not the API/UI/CLI-triggered "proper" import.
<rvba> jtv: agreed.  What's suboptimal is that I'll have to copy part of the documentation from the script to the online doc.
<jtv> ophuk: I seem to remember some confusion when reconfiguring afterwards, along the lines of "is this a URL or just an IP address?"  What I did in the past was just grep /etc for my IP address, and see where the fixes need to be.
<jtv> rvba: I don't suppose you can move it, and just make the script refer to the online version?
<rvba> jtv: I'd like to keep the description of the config file's format in the manpage.
<jtv> Ah, that part..  I agree, though it's been a bit of a pain tracking changes.
<newell> I am new to maas development.  I am trying to make it so the maascli (i.e. I will need to make API changes) that allow us to do:  maas <profile> node read <hostname | system_id>
<newell> so we  would be adding it so that we can read a node via the system_id (current) or hostname (new)
<newell> Curious where in the API this change would need to occur
<roaksoax> jtv: ^^
<roaksoax> rvba: ^^
 * newell is happy to be doing it in the right channel finally
<jtv> roaksoax: newell & I have been conversing about this, we just moved to a new channel.  :)
<rvba> heh
<roaksoax> jtv: ah ok :)
<jtv> newell: I think what you'd do at the moment is: maas <profile> nodes list hostname=<hostname>
<jtv> There must be some convenient command-line tool for manipulating/querying JSON...
<roaksoax> jtv: so, what if we would like to change anything that needs system_id to also be possible to specify hostname
<roaksoax> jtv: for example, we have been experiencing difficulty when adding/removing tags
<jtv> roaksoax: then I'd do some broader planning first, just because it's a technical departure.
<roaksoax> jtv: because every time we want to do it, we have to first obtain the system_id, so we would like to move to a world where I can just do API operations specifying a node's hostname, as it is unique
<roaksoax> jtv: so, I'd definitely say that we should support both, system_id|hostname
<jtv> First though, doesn't the Tag API let you do this?
<jtv> Because this is a relationship between Tag and Node, and Node is the side where the scale is.
<jtv> So maybe it'd make the most sense to say something like "add tag X to the nodes I specify as parameters."
<jtv> At that point, there's no technical departure and you no longer need to loop over all nodes for mass updates.
<roaksoax> jtv: so what do you think is more appropriate? Being able to do *any* node related operation (that currently needs system_id) with the hostname of the node instead (or alternatively) of using system_id? or for 1 or 2 methods being able to specify hostnames?
<roaksoax> jtv: it is my perspective that management via the CLI would be highly improved by being able to specify a node's hostname instead of a system_id
<roaksoax> jtv: because in the latter case, we are just doing a "work around"
<jtv> I don't think so.  It's just the more scalable way to phrase the request.
<roaksoax> jtv: obviously, there will be places where we could experience issues, such as when we are to update a node's details
<roaksoax> jtv: right, but right now, for any operation we want to make, we have to do the following: 1. obtain list of nodes. 2. grep for X node. 3. get its system_id. 4. perform operation
<roaksoax> jtv: usability wise, that is *very* painful
<jtv> Since hostnames really only make sense while a node is allocated, I guess another factor that comes into this is under which circumstances the operation makes sense on a given node.
<jtv> I don't think you understand what I said.
<roaksoax> jtv: they don't really only make sense when node is allocated. Hostnames make sense anytime because it is a nice way to identify a node, That's why DNS was created
<jtv> Given that you can tag a bunch of nodes which you pass as parameters, there's nothing against passing them _as hostnames_.
<roaksoax> jtv: right I got your point, however, what I'm trying to explain is that does solve the problem IMHO
<roaksoax> jtv: right now, if I want to update a node, I have to do 1. 2. 3. and 4. above
<roaksoax> if I want to update a tag, I have to do the same thing
<jtv> No you don't, is my point.
<roaksoax> there's many operations that require system_id to be passed
<roaksoax> maas maas tag update-nodes my_tag add="<system_id>"
<jtv> What I'm saying is this needs some more careful thought since it's a technical departure.
<ophuk> how come when you're installing maas from a live CD do you don't have an option to set a static IP when creating a new MAAS server?
<roaksoax> for me to do that, i need to 1. obtain list of nodes. 2. grep for X node. 3. get its system_id. 4.perform oeperaiton
<jtv> And you're just pushing for the solution you have in mind, which may or may not be fine, but we can end up going down a blind alley if we run with the first suggestion.
<roaksoax> jtv: so yes, in this particular case being able to specify hostname instead of system_id does make sense
<jtv> roaksoax: not quite what you need to do.  You can ask explicitly for just the node with the given hostname.
<roaksoax> jtv: right, but you are still asking for a node with a given hostname
<roaksoax> jtv: 1. ask for a node with a given hostname. 2. obtain its system_id. 3. perform any other operation I need
<roaksoax> jtv: why can't we just go to 3 directly by using its hostname instead of obtaining a system_id
<jtv> Think of it as OO if you prefer: you ask for the node, and you get its URL.
<roaksoax> jtv: right, that might be the case, but the CLI is a usability thing
<jtv> I see that you want to cut out that step, and I'm trying to help, but we do need to discuss the broader problem without pushing for the original suggestion.
<roaksoax> jtv: CLI as to make admin's life easier, not more complicated
<jtv> I'm trying to help you, but pushing and pushing instead of going through the design process makes that harder, not easier.
<roaksoax> jtv: I'm not pushing, I'm trying to figure out what is needed to make this happen (which is going through the design process)
<jtv> And I'm sorry for seemingly distracting you from your path to that solution, really I am.  But experience taught us that it's something we have to do.
<jtv> So the problem we have on the table is "I want it to be simpler to write changes to a node."  Right?
<newell> roaksoax, ^^
<jtv> Not just, "I want it to be easier to manipulate tags on a node."
<roaksoax> jtv: for every single node related operation (other than updating hostname), we need to obtain the system_id. That is affecting usability, so basically, I'd say that every single operation that currently requires a node's system_id, should accept the hostname (other than when updating a node's hostname of course)
<jtv> That's much better, thanks.
<jtv> roaksoax: the original architectural guideline for the API, which we've been lax about, was to focus on batch-oriented operations for scalability.  Unfortunately of course scale and friendliness are not good friends, and we do very much want more friendliness in the CLI.
<jtv> Some ideas for things we might want to look into in exploring your suggestion:
<jtv> 1. How would hostnames show up in URLs?  System IDs are "safe ASCII," hostnames have their own restrictions (including hopefully unicode at some point!)
<jtv> I think it's probably OK of course, just noting to check.
<jtv> 2. Is there any chance of ambiguity?
<jtv> Need to check whether a hostname can also be a well-formed system ID or vice versa.
<roaksoax> jtv: agreed.
<roaksoax> newell: did you join the maas-devel ML?
<roaksoax> newell: can you raise some discuss about it?
<roaksoax> jtv: then we would be able to estimate work effort and we can decide from there if this is somethign we can fit this cycle
<newell> hmm not sure let me check
<newell> roaksoax, you want me to send out an email to the list summarizing the conversation here that you and jtv have mostly had?
<jtv> newell, roaksoax: it's currently possible for the same identifier to be one node's system ID and another node's hostname... that's probably something we could just forbid if we have to.  It's a UUID, so not a lot of risk.
<roaksoax> newell: yeah an email talking about the possibility of allowing the API use <system_id | hostname> for all operations that currently require the system_id, and some background, and the ideas raised by jtv
<jtv> (Unless somebody tries to rename nodes as a way of getting one to take over another's role or something)
<newell> roaksoax, okay will do
<jtv> Thanks.  Mailing list is good for this.
<roaksoax> newell: thank you
<newell> npo
<roaksoax> jtv: and agreed, yes we need t consider various variables
<newell> np*
<jtv> Yeah.  Sorry for being difficult about it â we've had some really bad experiences going in to quickly with a solution, and had to force ourselves to do this.
<jtv> It always feels a bit like sabotage...
<jtv> But I'm glad that you're looking into user-friendliness improvements to the CLI!
<jtv> We're OK on the hostname-in-URL issue, I think.  Once we start supporting full unicode I guess we'll have to percent-encode â no drama.
<roaksoax> jtv: oh not at all, please rest assured that me bringing this up was to simply raise some discussion about the issue to reach a solution that would benefit all of us and our users
<ophuk> I've got a node in a ready state but I can't click on start node - there is a blue message up top stating an adequately configured DHCP server can boot this node. I did the DHCP/DNS server stuff through the web UI, is there something I'm missing?
<roaksoax> ophuk: did you add the SSH key for the user?
<ophuk> roaksoax: this part, maas-cli login maas http://10.98.0.13/MAAS/api/1.0 and then the key from the web UI? Yes
<roaksoax> ophuk: nope
<roaksoax> ophuk: go to the webui
<roaksoax> and go to the User Settings
<roaksoax> and add a SSH Key (public key)
<roaksoax> so you can ssh into the nodes after deployment
<ophuk> roaksoax: ok, recommissioning and seeing what happens.
<jtv> ophuk: there should be a hover message on the greyed-out button saying why you can't use it.
<ophuk> jtv: ok, I'll check it
<ophuk> jtv: it is working now
<ophuk> roaksoax: that fixed it i believe...thanks:)
<AskUbuntu> Tips for troubleshooting IPMI in MAAS 1.6 | http://askubuntu.com/q/473587
<jfarschman> Hello... I'm trying to work with preseed_master by making a copy "amd64_generic_trusty"  but when I do this I get an error on line 70
<jfarschman> name 'self' is not defined at line 70 column 3 in file /etc/maas/preseeds/amd64_generic_trusty
<jfarschman> This is something easy :)
<jfarschman> --- was reading about user-provided preseeds here http://maas.ubuntu.com/docs/development/preseeds.html
<jfarschman> Figured it out. Apparently inheritance is odd.
<jfarschman> okay something so basic, anyone can help
<jfarschman> After I have a node allocated and built in MaaS how to I force a rebuild?
<jfarschman> want the OS loaded again to test my preseed scripting.
<blake_r_> jfarschman: stop and start the node
<jfarschman> blake_r_: but how do I put it in a state where it knows to rebuild?
<jfarschman> it should not rebuild on every reboot.
<blake_r_> jfarschman: what do you mean rebuild?
<jfarschman> reinstall the OS, invoking the preseed modifications I just made
<blake_r_> jfarschman: when you stop a node, it is considered no longer in use
<blake_r_> jfarschman: when you start it again it will be re-installed
<jfarschman> I did an install yesterday that was generic... now I need to do an install that adds puppet
<jfarschman> oh... hahaha.
<blake_r_> jfarschman: start means turn on, and give me a clean installed system
<designated> if it still exists in maas, the os will not be reloaded.
<blake_r_> jfarschman: so be careful with that stop button, :)
<designated> otherwise a power failure would cause every server to reload OS
<blake_r_> designated: if the node is allocated and you click stop it will power off, and become unallocated
<blake_r_> designated: next time you click start it will allocate to you and install
<jfarschman> yea... I just assumed it was going to use IPMI and shut it down for me.  but I like this method.
<blake_r_> designated: if you restart the node, from the machine it will reboot, and will not reinstall
<designated> blake_r_, correct, it must be issued from within maas.  simply power cycling the node will not kick off a new OS load.
<blake_r_> jfarschman: yes it will use IPMI and power off the node
<blake_r_> designated: power cycling? if you use ipmi directly or un-plug it and back in, it will nto re-install
<blake_r_> designated: if you use maas to control the node, it will re-install
<blake_r_> designated: so in a data center and the power goes off, they will all restart to where they were before power cycle
<jfarschman> blake_r_: that language, "stop selected nodes" is a little deceptive.  Perhaps deprovision
<designated> blake_r_, we're saying to same thing.  I was trying to make sure jfarschman understood that maas had to be used, simply power cycling the node would not initiate OS load.
<designated> jfarschman, exactly
<blake_r_> jfarschman: https://bugs.launchpad.net/maas/+bug/1311224
<ubot5> Ubuntu bug 1311224 in MAAS ""Start node" and "Stop node" terminology confusing to newbies" [Medium,Triaged]
<designated> the maas documentation isn't entirely clear on a lot of things, like advanced networking for example.  How do you bond interfaces for LACP and use VLAN tagging?
<blake_r_> designated: good question?
<blake_r_> designated: idk, :)
<blake_r_> designated: never done that!
<designated> some things in the documentation are outright wrong.
<blake_r_> designated: are you using the correct version of the documentation, for your version
<designated> blake_r_, yes
<blake_r_> designated: please create a bug, if its wrong
<designated> I've been hesitant because this is the most active I have ever seen this IRC channel.  doesn't seem like a very active community, figured it was a waste of time.
<blake_r_> designated: we are very active, on launchpad
<designated> blake_r_, ahh i see
<blake_r_> designated: will try to be more on here
<designated> having access to more advanced networking features, like LAGG and VLAN tagging, as well as the ability to adjust disk partitions without having to manually write a preseed are essential to success for a project like this, especially if you plan on using juju to deploy software.  juju for example works great when deploying to a cloud but when deploying to bare metal, you need access to hardware, like multiple NICS or bonded and trunked inte
<designated> rfaces.
<blake_r_> designated: I know that VLAN tagging is supported, just don't know to what extent and the ability of MAAS
<designated> i tried deploying openstack on juju and maas, it was a nightmare because of limited network support and disk partitioning options.  who deploys openstack on hardware with a single interface?
<blake_r_> designated: juju neutron uses a second nic interface
<blake_r_> designated: adjust for partition would be hard, who would you handle different size hard drives?
<designated> blake_r_, if the community is truly active, I would definitely be interested in getting involved.  I just don't want to adopt something that isn't actively developed and wind up with problem after problem...
<blake_r_> designated: it is actively developed
<blake_r_> designated: and a key project for canonical
<designated> i did write a bash script to solve the problem of mac to name nic mapping inconsistencies across a pool of servers that gets pulled in during preseed, that was a huge issue i was having when trying to use juju with maas
<designated> blake_r_, i believe it is, it's just knowing how to get ahold of the developers to interact with them :)
<jfarschman> One more.... My provisioning of Trusty stops and [!!] Partition Disks and asks me to make a choice.  This is normal?
<blake_r_> jfarschman: no it should not ask you any questions
<blake_r_> jfarschman: you must have not answered all of the questions required in your preseed
<designated> jfarschman, i would think if the disk partitioning portion of the preseed was complete, it wouldn't do that.  I've never had that issue.
<blake_r_> designated: talking to one now!
<jfarschman> It did on the first install with the default preseed and now with my modified one.  I'll work it out.  Thanks
<designated> blake_r_, i thought maas 1.6 was supposed to be the default on 14.04 but it installs 1.5
<blake_r_> designated: 1.6 is trunk current development
<blake_r_> designated: trusty is 1.5
<blake_r_> designated: we do backport hardware enablement and critical fixes to 1.5
<blake_r_> designated: you can install from a ppa if you want 1.6
<blake_r_> designated: it is not recommended as its in active development
<blake_r_> designated: it can be broken
<blake_r_> designated: depending on when you use it, best to stick with 1.5 unless you really need something from 1.6
<jfarschman> blake_r_: I appreciate that.  When learning a new tool, predictability is important
<blake_r_> jfarschman: np
<designated> blake_r, i thought we would be able to bond interfaces in 1.5 according to https://blueprints.launchpad.net/maas/+spec/t-cloud-maas-advanced-networking
<designated> blake_r, I haven't seen anything in the documentation explaining how to do this.
<designated> blake_r, the ability to do a per node preseed would be extremely beneficial, or the ability to group nodes for a specific preseed, like using a certain preseed based on assigned tags.
<blake_r> designated: https://bugs.launchpad.net/maas/+bug/1254755
<ubot5> Ubuntu bug 1254755 in MAAS "Feature Request - bonding on MAAS provisioned nodes" [Wishlist,Triaged]
<blake_r> designated: you can do a per node preseed
<blake_r> designated: so in /etc/maas/preseeds
<blake_r> designated: if you make a file names liek this
<blake_r> designated: curtin_amd64_generic_trusty_hostname
<blake_r> designated: that is for a custom preseed that uses the curtin install known as "fastpath"
<blake_r> designated: so its {prefix}_{node_arch}_{node_subarch}_{release}_{node_name}
<blake_r> designated: by tag is not possible, but please make a bug for that, I agree that would be helpful
<designated> blake_r, it's to address issues like my storage nodes for example would need different network configuration as well as disk partitioning than say a server that simply runs an application.
<jfarschman> designated: I'm using puppet for that sort of thing.  When the config gets complicated and needs to be granular, puppet.
<designated> jfarschman, that's great but I might as well manually configure those items, if juju can't request a resource from maas that is configured ready to go.
<blake_r> designated: you can also use juju, to do that as well
<roaksoax> /query/win 3
<designated> blake_r, didn't think of juju handling that configuration, i suppose it would have to happen before installing/configuring any software.
<jfarschman> designated: fair enough, but not realistic yet.  One tool to rule them all is still a dream.
<blake_r> designated: yeah, you can modify your charm to perform the configuration
<blake_r> designated: of create a subordinate charm to do it
<designated> crowbar supposedly does all of that, but their dev community doesn't seem very active.  lol  I can't even get crowbar up and running on 14.04 and no one in their irc channel ever speaks...
<designated> plus from what I heard, dell stopped developing it.
<designated> blake_r, i would prefer to use maas/juju, it seems a lot more versatile, the hesitation was quite simply, I couldn't ever get ahold of anyone to troubleshoot/network with and the documentation hasn't been helpful in a lot of situations.
<blake_r> designated: feel free to tag me in a question so I see it, and I can respond
<designated> blake_r, thanks for the chat.  you have inspired me to give maas/juju another chance.
<blake_r> designated: cool, let me know, if you need any help
#maas 2014-05-29
<designated> blake_r, got a 401 response during juju bootstrap, both the node being commissioned and the maas node have the correct time settings in bios.  /var/log/maas/maas.log show the following: OAuthUnauthorized: 'Expired timestamp: given 1401305369 and now 1401326866 has a greater difference than threshold 300'
<designated> i found an old bug that discusses this issue but it was supposedly fixed, plus both nodes should be syncing with ubuntu's NTP server.
<designated> argg...cannot sync with ntp.ubuntu.com.  don't know what's going on there.
<designated> apparently the bootstrap node is being configured with a timezone of UTC which is causing oauth failure.  How do I change it from UTC?
<blake_r> designated: were you able to get ntp.ubuntu.com to work?
<allenap> jtv: Fancy a quick non-MAAS review? https://code.launchpad.net/~allenap/rabbitfixture/rabbitmq-3.3-and-later/+merge/221368
<jtv> allenap: approved
<allenap> jtv: Thanks!
<designated> blake_r, I did get ntp.ubuntu.com to work.  the problem that I was having was maas had a timezone of MDT but was booting nodes with UTC so there was a 6 hour time difference, causing OAUTH to fail.  I altered the preseed to set time to MDT, that resolved the issue.
<blake_r> designated: cool, glad you go it to work
<designated> blake_r, building new preseeds this morning to handle NIC mapping inconsistencies and proper disk partitioning for each node.  I'm assuming a juju charm will have the ability to do NIC bonding but no one in the juju channel was talking last night :)  otherwise I may have to use curtin to bond nics.
<blake_r> designated: well curtin can do it
<blake_r> designated: just "curtin in-target ...."
<blake_r> designated: hopefully that will give you the ability you want
<designated> blake_r, first i have to figure out WTH happened with udev in 14.04.  They changed the default behavior to use firmware/bios index numbers for onboard nics in the naming scheme but pci nicsget named according to a completely different method.  you wind up with something like emX for your onboard NICS and ethX for your PCI nics, only add-on hardware winds up in the 70-persistent-net.rules file.  back to same problem of inconsistencies acros
<designated> s multiple servers.  According to http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ this is supposed to resolve the problems we were seeing but it seems overly complicated for no apparent reason.
<designated> gonna digest this documentation to figure out the best way to proceed.
<designated> they addressed the problems associated with inconsistencies for a single server, but this doesn't make it anymore predictable if I have 100s of servers that need to be consistent.
<blake_r> designated: i am surprised you are having so many inconsistencies with nic naming, the should normally come up in the same order
<designated> blake_r, check this out
<designated> 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
<designated>     link/ether 00:8c:fa:0d:4e:38 brd ff:ff:ff:ff:ff:ff
<designated> 3: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
<designated>     link/ether 00:a0:d1:ed:77:e4 brd ff:ff:ff:ff:ff:ff
<designated> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
<designated>     link/ether 00:8c:fa:0d:4e:39 brd ff:ff:ff:ff:ff:ff
<designated> 5: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
<designated>     link/ether 00:a0:d1:ed:77:e5 brd ff:ff:ff:ff:ff:ff
<designated> went from eth0 to eth2...where is eth1?  on another server, it was completely different.
<designated> I think I'm just going to disregard the changes and write out my own persistent-net.rules file to override all of this nonsense :)
<blake_r> designated: yeah that is wierd
<designated> blake_r, I'm reading http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/utopic/curtin/utopic/view/head:/doc/topics/overview.rst but it's not entirely clear where these commands would go.  it looks like they would go here: /etc/maas/preseeds/curtin_userdata, but the syntax is different.  /etc/maas/preseeds/curtin only contains "{{preseed_data}}", not sure exactly what that is a reference to, unless it's the preseed file but I thought curti
<designated> n didn't use preseeds.  is there a link to more comprehensive curtin documentation?
<AskUbuntu> How can build relation between Rabbitmq-Server (as buffer for request interface) for my owm locla charm( process the requests) | http://askubuntu.com/q/474097
<blake_r> designated: they do go in curtin_userdata
<blake_r> designated: http://astokes.org/customizing-fastpath-curtin-installations/
<designated> blake_r, thanks
<designated> in maas I have the default distro for commissioning and deployment set to trusty, but when i deploy a juju charm, it installs precise...
<designated> i wonder if that has something to do with the charm
<roadmr> designated: not sure if juju's documentation is up to date but: "The default series for MAAS will automatically be set to 'precise'. You can override this setting by adding the optional configuration: default-series: trusty"
<designated> roadmr_afk, yeah i just figured out juju set-env "default-series=trusty" will fix it, but then not all charms exists in the trusty charm store :/
<designated> yay :)
<AskUbuntu> Ubuntu MAAS Setup - Newbie - adding nodes | http://askubuntu.com/q/474140
<designated> blake_r, do you know if variables and conditional statements in a preseed will work?
<designated> something similar to the following:
<designated> SETTIMEZONE=TRUE
<designated> TIMEZONE="US/MOUNTAIN"
<designated> {{if SETTIMEZONE}}
<designated> d-i	time/zone string $TIMEZONE
<designated> {{endif}}
#maas 2014-05-30
<AskUbuntu> MAAS - listed nodes not booting | http://askubuntu.com/q/474372
<blake_r> designated: I do not know
<blake_r> designated: but tempita is used for templating so if it supports it then yes
<blake_r> designated: http://pythonpaste.org/tempita/
<rvba> jtv: would you be so kind as to have a look at https://code.launchpad.net/~rvb/maas/bootsource-cli-doc/+merge/221551 when you get a chance?  (This is definitely not urgent.)
<jtv> Quite possibly.  :)
<rvba> jtv: you're too kind.
<jtv> Maybe.
<rvba> allenap: see what working with Brits is doing to us? ^ ;)
<jtv> Hey, don't say that!  Neither of us has apologised for anything yet!
<rvba> Oh, right.  Sorry about that.
<jtv> Sorry for bringing it up.  Thank you.
<allenap> Gentlemen, the pair of you.
<rvba> Just put a tiny fix up for review: https://code.launchpad.net/~rvb/maas/no-dhcp-write-of-unvalid-config/+merge/221558
<jfarschman> Is anyone adding puppet to the preseed post_scripts in MaaS.  I'm really close, but some thing aren't happening.
<jfarschman> specifically, my /usr/bin/puppet agent --enable;  line is not working, so puppet stays administratively disabled.
<jfarschman> Solved --- you guys probably guessed it.... improper syntax
<designated> I'm a little confused about something in the default preseed.  Why are both grub-installer/skip and lilo-installer/skip set to "false"?  Wouldn't this install both grub and lilo?
<jfarschman> designated: maybe it next line grub-installer/only_debian      boolean true
<designated> miles_, but grub-installer/with_other_os is also set to true.
<miles_> designated: I'm not going to pretend to be anything more than a hacker.  I suppose the OS is a facter in choosing which line is invoked
<designated> blake_r, do you know why?
<blake_r> designated: actaully i do not, but I think miles_ is correct
<blake_r> designated: grub will always be used
<blake_r> designated: i think lilo is an old option in d-i, that must be set, so the option is not asked
<designated> blake_r, okay thanks.
<blake_r> designated: the curtin install is now the prefer method of installation
<miles_> Is it really?  I just finished setting up everything without it.
<blake_r> miles_: yes it is
<blake_r> miles_: it is now the default installer
<miles_> I'll fire it up after Lunch.  Thanks.
<designated> blake_r, the curtin documentation doesn't seem to be comprehensive enough to provide the flexibility needed.  Read http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/doc/topics/overview.rst and the astokes.org blog link you posted yesterday.
<blake_r> designated: agreed that the curtin installer needs better documentation
<blake_r> designated: please make a bug on launchpad.net/curtin
<newell> I created a new lxc ubuntu container and then installed the packages maas, maas-dhcp, and maas-dns
<newell> Created an administrator account and the rebooted the lxc container
<newell> maas server is supposed to automatically start correct?
<newell> nm, figured it out
<newell> Anyone have a good idea of how long to realistically wait for the download of the boot images?
#maas 2014-05-31
<AskUbuntu> Running Openstack on 3 nodes | http://askubuntu.com/q/474911
#maas 2014-06-01
<delair> Hi.. I am new to MAAS and have a basic question that what is the difference between "trusty and precise" boot images
<delair> and if somebody can please point to a link where i can read about it .. Thanks
<jtv> delair: "precise" is Ubuntu 12.04 LTS, "Precise Pangolin."  And "trusty" is the newer Ubuntu 14.04 LTS, "Trusty Tahr."\
#maas 2015-05-25
<pmatulis> is it recommended (best practice) to install a maas server using the (ubuntu server) ISO?
<pmatulis> ("Multiple Server install with MAASâ)
<firl> thanks hatch
<mpontillo> pmatulis: I would recommend you install Trusty (using any method you want) and then use the MAAS stable PPA: https://launchpad.net/~maas-maintainers/+archive/ubuntu/stable
<pmatulis> mpontillo: ah, good point
<mup> Bug #1458662 was opened: List operation returns 414 Request-URI Too Long <MAAS:New> <https://launchpad.net/bugs/1458662>
#maas 2015-05-26
<caribou> Hello, I need to propose a fix to squid-deb-proxy's upstart job
<caribou> and since maas region controller has a dependancy on squid-deb-proxy, I'd like to check with you first
<caribou> so I don't break anything on your side. Anyone around who knows about squid-deb-proxy's usage with maas ?
<caribou> so nobody knows anything particular to squid-deb-proxy startup ?
<jgrassler> I don't know about startup but I can tell you what it's used for:
<jgrassler> Machines being installed by MAAS get the squid-deb-proxy running on the MAAS controller configured as their proxy for apt.
<mup> Bug #1457788 changed: Restarting maas-regiond and maas-clusterd fills the logs with stacktraces <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1457788>
<mup> Bug #1458894 was opened: Cluster service gives up and logs an IOError too soon <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1458894>
<mup> Bug #1458895 was opened: Database deadlock while starting regiond <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1458895>
<pmatulis> my maas nodes are using my maas server as nameserver but the latter is not forwarding and so resolution fails (except for ubuntu archive server names which is another question i have). i am surprised to see that maas uses bind/named. i also see the directory /etc/bind/maas. â  should i be configuring a forwarder there? â¡ shouldn't this be taken care of during maas installation? did i miss something?
<pmatulis> hmm, i have my maas server as forwarder set up in named.conf.options.inside.maas
<roaksoax> pmatulis: did you set the upstream DNS in the Settings page?
<pmatulis> roaksoax: i'm looking there now... got some ssh forwarding to deal with
<pmatulis> roaksoax: i think i remember putting my maas server there. so prolly my mistake
<roaksoax> pmatulis: ok, if you set it correctly, and still doesn't forward it, it might be due to the upstream dns server being misconfigured. You would have to work araound it by doing what's detailed on https://bugs.launchpad.net/maas/+bug/1384334
<roaksoax> pmatulis: which is changing dnssec-validation form auto, to no "dnssec-validation no;"
<pmatulis> roaksoax: working! ok, so i was confused by the field "Upstream DNS used to resolve domains not managed by this MAAS" . i figured, well of course all my domains are managed by maas". i guess in my mind i did s/domains/nodes or something
<pmatulis> roaksoax: which comes back to my other question, why does resolution work for say archive.ubuntu.com ?
<roaksoax> pmatulis: right, so the MAAS dns only resolves DNS it manages, not other DNS
<roaksoax> pmatulis: hence the need to setup a forwarder
<roaksoax> pmatulis: and it works because it goes through the proxy
<roaksoax> pmatulis: only apt is being sent through the proyxy
<pmatulis> roaksoax: but apt doesn't resolve names. wondering how it resolves, say, archive.ubuntu.com
<dpb1> blake_r: roaksoax: is the static range in the gui required to be filled in?
<roaksoax> pmatulis: because of the proxy :)
<roaksoax> dpb1: it should not be, but if you dont, then it is really bad user experience
<dpb1> roaksoax: ok
<roaksoax> dpb1: why would you not want to fill it in? :)
<dpb1> roaksoax: meh, in our environment, we don't really care about the distinction.
<dpb1> roaksoax: it's certainly not a big deal at all
<roaksoax> dpb1: so if you don't have a static range, deployed machines might eventually get a different address, and the understanding of what IP-> dns -> ownership might change
<dpb1> roaksoax: sure, that I need.  but dynamic nodes pulling from that same range doesn't bother me.
<roaksoax> dpb1: i.e. IP leased by node01.maas can be given to node02.maas and you'd have someone who thinks node01 and node02 have the same ip
<roaksoax> or similar
<roaksoax> dpb1: as long as it is a big enough range, it should be ok
<dpb1> roaksoax: yes, agreed, which is why it's not a big deal at all.
<dpb1> roaksoax: just curious
<roaksoax> cool
<dpb1> thx
<pmatulis> roaksoax: right ok
#maas 2015-05-27
<roaksoax> pmatulis: the MAAS machine can access to archive.u.c, hence the proxy software can resolve just fine a.u.c. So when uses the proxy, the proxy knows how to reach a.u.c
<drhalan> i have a maas setup that is using wake on lan to start nodes. however "stop node" doesn't do anything
<drhalan> maas also doesn't seem to be aware of nodes that have been powered on outside of the system. in general it doesn't seem to monitor the power state of machines
<drhalan> are such features only available when using IPMI or similar
<drhalan> ?
<drhalan> oh seems to be answered here: http://askubuntu.com/questions/583635/maas-check-power-state-error-unknown-power-type-ether-wake
<drhalan> nevermind
<mup> Bug #1459088 was opened: Internal Server Error during commissioning <cpe-sa> <MAAS:New> <https://launchpad.net/bugs/1459088>
<mup> Bug #1445482 changed: Nodes in "New" state cannot be powered off <MAAS:Incomplete> <https://launchpad.net/bugs/1445482>
<mup> Bug #1459168 was opened: Start-up stacktrace: IntegrityError: duplicate key value violates unique constraint "maasserver_config_name_uniq" DETAIL:  Key (name)=(rpc_shared_secret) already exists. <stacktrace> <start-up> <MAAS:Triaged> <https://launchpad.net/bugs/1459168>
<mup> Bug #1457879 changed: UnknownRemoteError: Code<UNKNOWN>: Unknown Error raised when calling update_host_maps  <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1457879>
<mup> Bug #1459088 changed: UnknownRemoteError: Code<UNKNOWN>: Unknown Error when calling delete_host_maps <cpe-sa> <MAAS:Triaged> <https://launchpad.net/bugs/1459088>
<mup> Bug #1294711 changed: power management broken after upgrade from saucy <amd64> <apport-bug> <trusty> <MAAS:Fix Released> <maas (Ubuntu):Fix Released by andreserl> <https://launchpad.net/bugs/1294711>
<mup> Bug #1294711 changed: power management broken after upgrade from saucy <amd64> <apport-bug> <trusty> <MAAS:Fix Released> <maas (Ubuntu):Fix Released by andreserl> <https://launchpad.net/bugs/1294711>
<mup> Bug #1457191 changed: Fresh install (from trunk or ppa:maas-maintainers/experimental) results in /var/log/maas/clusterd.log being owned by root <MAAS:Fix Released by andreserl> <https://launchpad.net/bugs/1457191>
<mup> Bug #1459331 was opened: exceptions.AttributeError: 'ServiceParsingError' object has no attribute 'output_as_unicode' <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1459331>
<mup> Bug #1459338 was opened: Service checking on utopic wrongly assumes the system is using systemd. <MAAS:Triaged> <https://launchpad.net/bugs/1459338>
<mup> Bug #1459380 was opened: MAAS logs 503 spurious errors when the region service isn't yet online <MAAS:Triaged> <https://launchpad.net/bugs/1459380>
<mup> Bug #1459380 changed: MAAS logs 503 spurious errors when the region service isn't yet online <MAAS:Triaged> <https://launchpad.net/bugs/1459380>
<mup> Bug #1459380 was opened: MAAS logs 503 spurious errors when the region service isn't yet online <MAAS:Triaged> <https://launchpad.net/bugs/1459380>
<mup> Bug #1459405 was opened: Systems can't be recommissioned as part of releasing <oil> <MAAS:Triaged> <https://launchpad.net/bugs/1459405>
<mup> Bug #1459432 was opened: "could not serialize access due to concurrent update" errors in PostgreSQL logs <MAAS:New> <https://launchpad.net/bugs/1459432>
<mup> Bug #1459432 changed: "could not serialize access due to concurrent update" errors in PostgreSQL logs <MAAS:New> <https://launchpad.net/bugs/1459432>
<mup> Bug #1459432 was opened: "could not serialize access due to concurrent update" errors in PostgreSQL logs <MAAS:New> <https://launchpad.net/bugs/1459432>
<mup> Bug #1459441 was opened: Add Interface is in the lowest left corner of the Network section <MAAS:New> <https://launchpad.net/bugs/1459441>
<mup> Bug #1459441 changed: Add Interface is in the lowest left corner of the Network section <MAAS:New> <https://launchpad.net/bugs/1459441>
<mup> Bug #1459441 was opened: Add Interface is in the lowest left corner of the Network section <MAAS:New> <https://launchpad.net/bugs/1459441>
<mup> Bug #1459444 was opened: Info for the PXE NIC available in MAAS UI but there is no API to get this info <oil> <MAAS:New> <https://launchpad.net/bugs/1459444>
<harushimo> Can MAAS be installed on a VM?
#maas 2015-05-28
<roaksoax> higgins: yes
<roaksoax> err
<roaksoax> sorry
<catbus11> roaksoax: hi, is there documentation about how to build custom image to be imported to MAAS yet?
<Mmike> Hello, lads. When I add images to maas, and then I PXE boot from that image, what is the default username/password that I can use to connect to PXEbooted image?
<mup> Bug #1459607 was opened: Spurious test: maasserver.api.tests.test_node.TestNodeAPI.test_POST_commission_commissions_node <MAAS:In Progress by rvb> <https://launchpad.net/bugs/1459607>
<mup> Bug #1459666 was opened: Wrong cursor when hovering over the "Add interface" button on the node details page <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1459666>
<mup> Bug #1445070 changed: Edit Node pages shows root password in plaintext <ui> <MAAS:New> <https://launchpad.net/bugs/1445070>
<mup> Bug #1442131 changed: 1.8b1 redundant owner column on devices page <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1442131>
<mup> Bug #1459710 was opened: "Set zone" label oddly placed on node listing page <ui> <MAAS:Confirmed for ricgard> <MAAS 1.8:Confirmed for ricgard> <https://launchpad.net/bugs/1459710>
<mup> Bug #1438109 changed: styling for sort on node listing doesn't yet match designs <ui> <ux> <MAAS:Invalid> <https://launchpad.net/bugs/1438109>
<mup> Bug #1459762 was opened: hostname, architecture, disable_ipv4 can be permenantly changed by non-admin user <MAAS:New> <https://launchpad.net/bugs/1459762>
<mup> Bug #1459762 changed: hostname, architecture, disable_ipv4 can be permenantly changed by non-admin user <MAAS:New> <https://launchpad.net/bugs/1459762>
<mup> Bug #1459762 was opened: hostname, architecture, disable_ipv4 can be permenantly changed by non-admin user <MAAS:New> <https://launchpad.net/bugs/1459762>
<mup> Bug #1459865 was opened: System cannot be enlisted with the ephemeral release images <oil> <MAAS:New> <maas-images:New> <https://launchpad.net/bugs/1459865>
<mup> Bug #1459866 was opened: commissioning failing due to no-such-image in boot string <cpe-sa> <MAAS:New> <https://launchpad.net/bugs/1459866>
<mup> Bug #1459876 was opened: MAAS image syncing does not seem to be deleting old images from DB <MAAS:New> <https://launchpad.net/bugs/1459876>
#maas 2015-05-29
<mup> Bug #944565 changed: User details and password are separate forms <ui> <ux> <MAAS:Invalid by carlaberkers> <https://launchpad.net/bugs/944565>
<mup> Bug #1058305 changed: settings page discards multiple changes in different sections <ui> <ux> <MAAS:Invalid> <https://launchpad.net/bugs/1058305>
<mup> Bug #1459886 was opened: Power parameters are sometimes not showing on the webui <MAAS:New> <https://launchpad.net/bugs/1459886>
<mup> Bug #1459888 was opened: Too much spacing between checkboxes/releases in the 'Images'  <MAAS:New> <https://launchpad.net/bugs/1459888>
<mup> Bug #1459889 was opened: Settings page table does not take the whole page <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459889>
<mup> Bug #1459890 was opened: Accounts page table does not take the whole page <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459890>
<mup> Bug #1459892 was opened: User SSH Key is not editable <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459892>
<mup> Bug #1459893 was opened: Delete button for the API Key is inside textbox and does not match SSH key delete button <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459893>
<mup> Bug #1459894 was opened: Cluster settings page is justified to the left and not taking the whole page <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459894>
<mup> Bug #1459896 was opened: Scaling node listing page down, causes Add Hardware text to basically disappear from button <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459896>
<mup> Bug #1459898 was opened: Can't collapse machine output <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459898>
<mup> Bug #1459901 was opened: Can't select, and copy/paste power parameters (IP address, user, password, etc) in node details page <MAAS:Confirmed> <MAAS 1.8:New> <https://launchpad.net/bugs/1459901>
<mup> Bug #1402861 changed: cloud-config-url ignored, install fails <oil> <MAAS:Invalid> <https://launchpad.net/bugs/1402861>
<mup> Bug #1460026 was opened: The color of the headers on the device list page is different from the color of the headers on the node list page <MAAS:New> <https://launchpad.net/bugs/1460026>
<mup> Bug #1460097 was opened: 1.8.0b8 bmc busy, ipmi invalid password, unknown power change: query <uosci> <MAAS:New> <https://launchpad.net/bugs/1460097>
<mup> Bug #1460128 was opened: whitespace above header keeps growing on click for chromium <ui> <MAAS:New> <https://launchpad.net/bugs/1460128>
<mup> Bug #1460128 changed: whitespace above header keeps growing on click for chromium <ui> <MAAS:New> <https://launchpad.net/bugs/1460128>
<voidspace> is there something I need to install for maas to be able to control an apc PDU?
<voidspace> I get failed commissioning with error "No such file or directory"
<voidspace> I assume maas is trying to run a command and that command is not available
<voidspace> allenap: ping
<allenap> newell: ^ can you help voidspace?
<mup> Bug #1460128 was opened: whitespace above header keeps growing on click for chromium <ui> <MAAS:New> <https://launchpad.net/bugs/1460128>
<newell> voidspace, what version of maas are you using?
<newell> allenap, yeah I can
<voidspace> newell: 1.8 from dailybuilds
<allenap> newell: Ta :)
<voidspace> newell: maybe from a few days ago
<newell> voidspace, can you paste me the error?
<newell> voidspace, from the logs
<voidspace> newell: which log file?
<voidspace> APC should be using telnet as far as I can tell from the code
<newell> voidspace, are these your own APC PDUs?
<voidspace> newell: it's my PDU, an APC 7920
<newell> voidspace, the newest code for APC is using SNMP
<voidspace> and it works fine used through the web UI
<newell> voidspace, can you remotely telnet into the PDU?
<voidspace> newell: I have done, yes
<voidspace> newell: which log file do you want me to look in
<newell> voidspace, I just need to understand the error and why you are getting it
<newell> voidspace, is there anyway you can upgrade to trunk?
<newell> or copy the apc.py file that is in trunk?
<newell> and the template as well (you would need that in addition to apc.py)
<voidspace> hmm... I have it installed from the ppa
<voidspace> how easy is it to do a manual upgrade
<newell> voidspace, let's just copy over the files
<voidspace> yep
<newell> let me point you to them, one second
<voidspace> I need to join a conf call
<voidspace> if you post me the files I'll copy them over and ping you when I've done it
<newell> voidspace, http://bazaar.launchpad.net/~maas-committers/maas/trunk/download/newell.jensen%40canonical.com-20150518200138-i9rukj0x1kt3vzs8/apc.template-20150114165444-s6gob4kghfqzss30-1/apc.template
<newell> that is template
<newell> that will be overwritten in /etc/maas/templates/power/apc.template on your maassserver
<voidspace> newell: ok
<voidspace> newell: these are the lines from the log (not terribly informative)
<voidspace> newell: http://pastebin.ubuntu.com/11435450/
<newell> voidspace, http://bazaar.launchpad.net/~maas-committers/maas/trunk/download/newell.jensen%40canonical.com-20150518200138-i9rukj0x1kt3vzs8/apc.py-20150114032625-xrhr9w5csdc3vqdy-1/apc.py
<newell> that is apc.py
<newell> that will be overwritten at /usr/lib/python2.7/dist-packages/provisioningserver/drivers/hardware/apc.py
<newell> on maasserver
<voidspace> cool, thanks
<newell> voidspace, http://bazaar.launchpad.net/~maas-committers/maas/trunk/download/raphael.badin%40canonical.com-20150527145249-y0b04fvm4z1xq3f1/power_schema.py-20140304002952-npz5m4gqcw3gt383-1/power_schema.py
<mup> Bug #1460145 was opened: "sudo maas-region-admin dbshell" fails with "ImportError: No module named testing" error <MAAS:New> <https://launchpad.net/bugs/1460145>
<newell> and that is power_schema.py which will be overwritten at /usr/lib/python2.7/dist-packages/provisioningserver/power_schema.py
<newell> voidspace, make sure you copy with sudo and then when you have done that you need to restart the cluster
<newell> sudo restart maas-clusterd
<newell> voidspace, I assume that the cluster and maasserver are on the same machine above.  Is this correct?
<voidspace> newell: ok
<voidspace> newell: yes
<newell> voidspace, after you do that you will need to reset the power parameters for that node
<newell> voidspace, make sure you have snmp package installed on the maasserver as well
<voidspace> newell: installing snmp may have fixed it...
<voidspace> newell: I don't get that error now... :-)
<newell> voidspace, then you already had the files I was pointing you to
<newell> voidspace, you said you had telnet
<newell> voidspace, could you please verify.  If so, I need to add a bug so we can add the dependency to maas for snmp package.
<voidspace> newell: heh, I have an out of date source tree alongside the installed version :-/
<voidspace> newell: sorry about that
<voidspace> newell: so I verified a power off worked through the web UI
<voidspace> newell: attempting a power on tells me the node can't be powered on
<voidspace> newell: but the  error has gone away
<voidspace> newell: I need to configure these servers to boot on power I think *anyway*
<newell> voidspace, can you see your PDU?
<voidspace> newell: yes
<newell> voidspace, have you configured your MIP?
<voidspace> newell: a power off initiated from MAAS worked
<newell> so that it is writeable?
<voidspace> newell: MIP?
<newell> http://kb.paessler.com/en/topic/653-how-do-snmp-mibs-and-oids-work
<mup> Bug #1460145 changed: "sudo maas-region-admin dbshell" fails with "ImportError: No module named testing" error <MAAS:New> <https://launchpad.net/bugs/1460145>
<voidspace> newell: looking
<newell> voidspace, you should be able to power on and off the outlets from the command line on the maasserver doing something like this:
<newell> $ snmpset -c private -v1 172.16.9.4
<newell> .1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.8 i 1
<newell> you would need to change the ip address (172.16.9.4) to the IP for your PDU
<newell> 'i 1' is for on I believe
<newell> and 'i 2' would be for off
<voidspace> so I get failed object reference
<voidspace> looking up this PDU
<voidspace> looking up the mib for this PDU I mean
<newell> voidspace, well you should be able to use snmpset commands to turn on/off the outlet numbers
<newell> voidspace, you will need to make sure your PDU can be controlled like that for MAAS to work
 * newell makes a note to add documentation for this
<mup> Bug #1460145 was opened: "sudo maas-region-admin dbshell" fails with "ImportError: No module named testing" error <MAAS:New> <https://launchpad.net/bugs/1460145>
<mup> Bug #1460151 was opened: MAAS configured apt proxy gets in the way of Juju configured one <sts> <MAAS:New> <https://launchpad.net/bugs/1460151>
<voidspace> newell: current status http://pastebin.ubuntu.com/11435840/
<voidspace> newell: following the guide here https://tobinsramblings.wordpress.com/2011/05/03/snmp-tutorial-apc-pdus/
<newell> voidspace, you need to program your PDUs.  I haven't done it myself but someone sent me this (no idea if this will help you): http://pastebin.ubuntu.com/11435917/
<voidspace> newell: hmm... I do have an SMP menu
<voidspace> newell: looks like I need to set the NMS IP
<voidspace> I don't have the SNMPv1 menus from there, but I have other interesting looking ones
<newell> voidspace, what snmp version menus do you have?
<voidspace> I have "1. Settings 2. Access Control 1 ..." and then a load more Access Control and Trap Receiver menus
<voidspace> no version specific menus at all
<voidspace> I wonder if it's just old firmware
<voidspace> with snmpset I need the right mib installed, and finding out how to install that mib is problematic
<voidspace> with what I thought was the right mib file in a location it could be found, I got a different error
<newell> voidspace, what firmware version does your PDU have?
<voidspace> 2.6.4
<voidspace> apparently
<newell> never seen that version
<voidspace> that's APC OS version
<newell> when you telnet into the APC what does the firmware version show as?
<voidspace> American Power Conversion               Network Management Card AOS      v2.6.4
<newell> we only support 3.7.4/3.7.4
<newell> 3.7.3/3.7.4
<voidspace> ok
<voidspace> I'll try to upgrade
<newell> voidspace, yeah that might do the trick for you
<newell> voidspace, although snmp is snmp (the version support was mostly for the telnet version of the software)
<newell> voidspace, once you get the snmpset working...MAAS will work
<voidspace> :-/
<voidspace> hoping I don't have to install windows to update firmware
<mup> Bug #1460189 was opened: maas-clusterd must be restarted after installing python-vmomi <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1460189>
<mup> Bug #1460193 was opened: MAAS needs to inject the user's SSH key into the commissioning image for debugging <MAAS:Triaged> <https://launchpad.net/bugs/1460193>
<mup> Bug #1460199 was opened: NoConnectionsAvailable: Unable to connect to cluster 3e375000-9ece-453c-9b5e-f817ab3a51b2; no connections available. <MAAS:Triaged> <https://launchpad.net/bugs/1460199>
<mup> Bug #1460204 was opened: While region is downloading images, cluster is disconnected <MAAS:Confirmed> <https://launchpad.net/bugs/1460204>
#maas 2015-05-31
<mup> Bug #1460485 was opened: Chef Knife broken with MAAS 1.8 - API regression - extra slash <MAAS:New> <https://launchpad.net/bugs/1460485>
#maas 2016-05-30
<jojden> hi roaksoax
<BlackDex> Hello there. I have maas running and using a dhcp-relay with different subnets on the same fabric. This seems to give some trouble because it doesn't responed with the correct subnet. It just picks a subnet (i think the first subnet it sees) and uses that to allocat an IP Address. It should just respond with an IP on the same subnet the relay is on. Is there a special setting i need to do. Add it to a seperate space? Or a sub-fabric or something?
<tarantul> anyone can help with 'failed commissioning' node?
<tarantul> I see many Calling 'http://169.254.168.254/2009-04-04/meta-data/instance-id in boot screen. May be it's a cause?
<D4RKS1D3> hi
<D4RKS1D3> I have one question, it is posible to change automatically the state new to ready?
<kiko> D4RKS1D3, by automatically do you mean "automatically accept and commission every node which has enlisted"?
<kiko> D4RKS1D3, the easiest way to do that is via the API
<kiko> BlackDex, hmmm
<kiko> BlackDex, not sure what you are doing
<kiko> tarantul, could it be because of anoher failure? you'll need to look at the logs to figure it out
<D4RKS1D3> Via api is possible
<D4RKS1D3> but I do not know if with one command similiar to "accept-all" it is possible
<D4RKS1D3> thank for your contribution kiko
<tarantul> kiko: what log I need check?
<kiko> D4RKS1D3, no, you'll need to accept one by one -- in part to avoid the race condition of a rogue node enlisting between you looking and accepting all
<kiko> tarantul, the install log -- turn on the flag when you commission that leaves the machine running to check it
<tarantul> kiko: Node is runing, but I can't login. Does MAAS set default user/pass?
<tarantul> I soled my problem, but have another now. Node boots, do some stuff and shutdown. After reboot it's boot into old linux, not in a new 16.4 setup.
<kiko> tarantul, you now need to deploy -- commissioning does not write to disk
<kiko> tarantul, what was the problem with commissioning, incidentally?
<mup> Bug #1587163 opened: Error: No boot sources provide Ubuntu images <MAAS:New> <https://launchpad.net/bugs/1587163>
<batz77> hi there.. im having trouble with deploying a vivid node. pollinate/seeding appears to fail in the same way outlined in https://bugs.launchpad.net/maas/+bug/1424549 . there's a lot of replies in that bug and im confused as to what the actual fix is, if there is one
<batz77> one side of the threads says "change your image sources to get the newer baked in cert" (I think) and the other says that doesnt matter its a red herring
#maas 2016-05-31
<mup> Bug #1587201 opened: MAAS 2.0 fails to format all bcache partitions <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1587201>
<mup> Bug #1587201 changed: MAAS 2.0 fails to format all bcache partitions <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1587201>
<capslockwizard> I've been looking around the manual of maas but I can't seem to find any documentation on the images that maas uses
<capslockwizard> What I can infer is that is gzip file
<capslockwizard> How can I go about building my own custom MAAS image (Ubuntu and not CentOS)
<capslockwizard> ?
<jojden> how to update interface using maas api
<jojden> I am getting following error with this code
<jojden> self.client.put(u"nodes/%s/interfaces/%s/" % (node_id, interfaces_id))
<jojden> HTTP ERROR:  Cannot update interface interface because the node is not Ready.
<jojden> any idea
<jojden> It means we can change only for the node with ready state ?
<joelio> Hey, I've done a fresh install of 16.04 and just installed MaaS in some real hardware (I tested with vms first). I can't for the life of me get the network interfaces or the controller itself listed in a given subnet
<joelio> it didn't autodiscover what range it was in either
<joelio> not I can't add DHCP resolution as it needs a controller to define that and I can't set the contoller as it's not appearing in the list of available controllers
<D4RKS1D3> Hi, I am trying to add a machine via juju, but i received this message "ERROR juju.provisioner provisioner_task.go:655 cannot start instance for machine "40": cannot run instances gomaasapi: got error back from server: 409 CONFLICT (No available node matches constraints: name=imaginative-attention.datacentre tags=datacentre)"
<D4RKS1D3> Someone know what happens?
<joelio> this just doesn't work.. no rack controller can be applied to a network
<joelio> so can't provision, shame a s this looked like a good tool
<joelio> I've gone through all the docs related to rack controller too
<joelio> it just doesn't work - wonder if it's a bug?
<joelio> yea, just doesnt work - even on a fresh install
<joelio> Node must be connected to a network.
<joelio> in the Controller summary
<joelio> I've applied subnets/spaces etc
<joelio> but there seems to be a bug now
<joelio> it's annoying that the docs don't reference 16.04 properly too - there seems to be a mish-mash of 1.9 and 2.x info
<joelio> as for example, it's showing ethX, not Consistent Device Naming
<joelio> perhaps I'm in a bug
<joelio> as it's beta3 listed in the package upstream in 16.04
<joelio> yet the latest and greatest according to the changelog is beta5
<joelio> there's no PPA for 16.04 either
<joelio> ahh, found the next ppa
<joelio> yea, still the same issue
<joelio> I guess this is a dead channel then too?
<joelio> guess MaaS not the product for us then.. :)
<lovea> Just upgraded from MAAS 1.8 to 2.0 and my PXE boot is bust. The upgrade has placed my external eth0 (unmanaged) and internal (maas managed) eth1 in the same fabric. I have created a separate fabric for the maas managed subnet but can't seem to move the subnet into it. How do I either move (update) a subnet into a different fabric or move an interface into another fabric?
<lovea> To add to that the upgrade effectively placed both subnets on the same untagged VLAN.
<mup> Bug # opened: 1587539, 1587542, 1587543, 1587544
<mup> Bug #1587548 opened: In p.rpc.clusterservice.Cluster.refresh, no attempt is made to deal with concurrency <tech-debt> <MAAS:New> <https://launchpad.net/bugs/1587548>
<mup> Bug #1587549 opened: In p.rpc.clusterservice.Cluster.refresh, nothing waits for or handles errors from p.refresh.refresh <tech-debt> <MAAS:Triaged> <https://launchpad.net/bugs/1587549>
<mup> Bug #1587605 opened: [2.0b5] Allowed to commission newly enlisted nodes without power type set <MAAS:New> <https://launchpad.net/bugs/1587605>
<sjl> must maas be installed on a bare-metal server?
<kiko> sjl, no, it can run in a VM
<kiko> in fact many people do
<sjl> thanks kiko.
<sjl> anyone get this to work?  'sudo apt-get install maas-image-builder'  I get Unable to locate package error
<kiko> sjl, that package I think may be available from a PPA
<sjl> oh ok, was trying to follow the docs on how to install maas-image-builder:http://maas.ubuntu.com/docs/os-support.html?highlight=builder#maas-image-builder
<kiko> sjl, http://askubuntu.com/questions/744966/how-do-you-install-the-maas-image-builder-on-14-04-lts maybe?
<sjl> kiko, looked promising but fails on the `make` step with "Error: There is a version conflict"
#maas 2016-06-01
<alexlist> Hi. Is there a way to provide a custom kickstart file when deploying CentOS using MAAS?
<kiko> alexlist, yes if you customize using maas-image-builder I believe
<mup> Bug #1587864 opened: [2.0b5] get_interfaces_definition is not thread-safe <MAAS:Triaged> <https://launchpad.net/bugs/1587864>
<mup> Bug #1587896 opened: [2.0b5] p.refresh.get_swap_size misconverting units <MAAS:Triaged> <https://launchpad.net/bugs/1587896>
<mup> Bug #1587936 opened: [2.0, UI] Add fabric, VLAN, Space show's badly place form <MAAS:New for ricgard> <https://launchpad.net/bugs/1587936>
<mup> Bug #1587939 opened: [2.0, UI] 'Commission' a node under the Node Listing Page shows actions not correctly formatted. <MAAS:New for ricgard> <https://launchpad.net/bugs/1587939>
<mup> Bug #1587946 opened: IPMI connection failure causes failed deployment and releasing <MAAS:New> <https://launchpad.net/bugs/1587946>
<sjl> trouble getting dhcp to start.  missing /var/lib/maas/dhcpd.conf, what normally creates this
<mup> Bug #1587946 opened: IPMI connection failure causes failed deployment and releasing <MAAS:New> <https://launchpad.net/bugs/1587946>
<sjl> is there an updated http://maas.io/get-started, the steps don't match what I'm seeing
<kiko> sjl, maas itself starts dhcp up once you've started managing a network segment
<kiko> sjl, what version of MAAS are you using?
<sjl> MAAS Version 2.0.0 (beta3+bzr4941)j
<sjl> MAAS Version 2.0.0 (beta3+bzr4941)
<sjl> simply followed the steps as best I could from http://maas.io/get-started
<roaksoax> slcan you upgrade to beta5 ?
<roaksoax> sjl: ^^
<sjl> sure how do I do that?
<kiko> sjl, apt-get update and dist-upgrade should do it
<mup> Bug #1588000 changed: [2.0, UI] There's no spacing between combo boxes under the Machine Details page <MAAS:New for ricgard> <https://launchpad.net/bugs/1588000>
<mup> Bug #1588000 opened: [2.0, UI] There's no spacing between combo boxes under the Machine Details page <MAAS:New for ricgard> <https://launchpad.net/bugs/1588000>
<sjl> updated to beta5, reboot maas server, but no change. dhcp service still not running.
<sjl> looking at the Nodes->Controller summary->Services shows dhcpd without the green checkmark
<roaksoax> sjl: what does the logs say ?
<roaksoax> sjl: and what you are describing means that DHCP is not turned on
<roaksoax> sjl: did you turn it on Networks > VLAN (select the vlan where you want to enable DHCP, most likely 'untagged' under a specific fabric -> Provide DHCP (in the action menu)
<sjl> I have 5 fabrics ('fabric-0', 'fabric-1', etc.) and fabric-0 is where I added a Subnet 10.17.176.0/22 and you're right, it's 'untagged' VLAN (as it should be in my lab)
<sjl> how do I 'turn it on'?
<roaksoax> sjl: click on the 'untagged' VLAN under fabric-0
<roaksoax> sjl: then, in the action menu on the right, click on 'Provide DHCP'
<sjl> ah..this looks promising..
<sjl> I've 'Provide DHCP', but dhcp service still does not have the 'green checkmark'.  going to reboot server to see if it helps.
<roaksoax> sjl: hold on
<roaksoax> sjl: do you have any logs ?
<roaksoax> sjl: of what happened when you provided DHC{P
<sjl> reboot helped, I see a green checkmark now, going to test
<sjl> thanks roaksoax! my bm was successfully pxe'd by maas.
<roaksoax> sjl: cool
<roaksoax> sjl: that was the missing piece, enabling DHC{
<roaksoax> sjl: that was the missing piece, enabling DHCP
<sjl> yeah, been racking my head over that the past couple days..scouring the docs/internet
<sjl> this is scary, somehow maas was able to discover my bm's iDRAC interface and even added a 'maas' account for itself...
<roaksoax> sjl: that's ipmi autodiscovery
<mup> Bug #1588040 opened: [2.0-bzr5059] "maas <profile> boot-sources read" returns an errant value <MAAS:Triaged> <https://launchpad.net/bugs/1588040>
<sjl> is there a canonical rep who covers Texas that I can reach out to?
<sjl> i've tried the 'contact us' web form and only got the auto response
<mup> Bug #1588093 opened: [2.0-bzr5059] UI reports "Node must be connected to a network" after creating bond <MAAS:Triaged> <https://launchpad.net/bugs/1588093>
#maas 2016-06-02
<alexlist> kiko: That is not what I am looking for... for Ubuntu, you can have custom preseeds. We need something similar for CentOS as well, without going through the pain of creating custom images...
<roaksoax> alexlist: what are you looking for?
<roaksoax> alexlist: you *can* have custom preseeds for centos
<mup> Bug #1588116 opened: [2.0-bzr5059] Adding a rack using dpkg-reconfigure isn't intuitive <MAAS:New> <https://launchpad.net/bugs/1588116>
<mup> Bug #1588125 opened: Nodes commission, but don't see network interfaces with Xenial <cpe-sa> <MAAS:New> <https://launchpad.net/bugs/1588125>
<mup> Bug #1588154 opened: Deploying node fails when it has VLAN configuration <MAAS:New> <https://launchpad.net/bugs/1588154>
<mup> Bug #1588161 opened: Empty boot-source-selections imports all available images <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1588161>
<mup> Bug #1588350 opened: [2.0b5] Multiple power query's in too little time <MAAS:New> <https://launchpad.net/bugs/1588350>
<mup> Bug #1588350 changed: [2.0b5] Multiple power query's in too little time <MAAS:New> <https://launchpad.net/bugs/1588350>
<mup> Bug #1588350 opened: [2.0b5] Multiple power query's in too little time <MAAS:New> <https://launchpad.net/bugs/1588350>
<kiko> alexlist, what roaksoax said -- you can generate custom preseeds for centos; the preseeds are not OS-specific but rather specific to curtin (and cloud-init)
<roaksoax> kiko: /etc/maas/preseeds/curtin_userdata_centos
<roaksoax> alexlist: ^^
<roaksoax> alexlist: so you can basically do what you need via preseeds
<gimmic> in maas, can I set the default ipmi credentials?
<gimmic> somewhere
<gimmic> rather than waiting to assign it to hosts which are discovered
<gimmic> I'd prefer it if Maas defaulted to ipmi 2.0 and used N credentials
<roaksoax> gimmic: you can change the credentials after you commission the machine
<gimmic> so I've got a host deployed, but I can't seem to ssh to it with the set key
<gimmic> actually, status says "deploying"
<kiko> gimmic, need to wait until it's deployed
<gimmic> I can't see any activity on the host or the maas server
<shewless> Hi guys, I changed the "external" facing IP of my maas server and now I'm having trouble. I can connect via ssh but dhcpd isn't running and I dns look-ups to external sites fail. Any hints at config I'd have to change?
<shewless> maas 2.0
<shewless> ah..nevermind I got it figured out. thanks
<mup> Bug #1588466 opened: gpg --batch --verify during maas install causes Unhandled Error <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1588466>
<mup> Bug #1588469 opened: cluster view won't load if a single cluster controller lacks a valid UUID <MAAS:New> <https://launchpad.net/bugs/1588469>
<shewless> hmm.. I spoke to soon. I fixed the dns lookups but dhcpd is not starting
<shewless> I can't seem to find any log files related to maas-dhcpd or dhcp
<shewless> any pointers?
<kiko> shewless, you may need to do a dpkg-reconfigure -- you used to in 1.9. roaksoax, any clue?
<shewless> kiko: thanks - is there harm in my doing that on 2.0.0?
<shewless> kiko: also which package would I reconfigure?
<kiko> shewless, I'd guess maas-rackd
<roaksoax> shewless: have you tried setting upstream DNS server  in MAAS ?
<shewless> roaksoax yes I have
<shewless> roaksoax: it's set to an internal DNS that is still reachable by ping
<roaksoax> shewless: so why is dhcp not running ? have you looked at systemd ?
<shewless> roaksoax: I'm not sure how to do that.
<roaksoax> shewless: journalctl -u maas-dhcpd -> try that ?
<shewless> roaksoax: 1 single entry: Jun 02 14:37:22 maas systemd[1]: Stopped MAAS instance of ISC DHCP server for IPv4.
<shewless> if I restart the service it just prints that
<roaksoax> shewless: so what if you do ps faux | grep dhcpd ?
<shewless> nothin
<shewless> roaksoax: that command shows me tthat dhcpd is not running
<roaksoax> shewless: ok, so is DHCP enabled at all in MAAS ?
<shewless> roaksoax: yes it is. I tried disabling it and re-enabling it but it didn't seem to have an effect.  Related if I do a "service maas-dhcpd status" I see that /var/lib/maas/dhcpd.conf is missing.. that's probably bad right?
<mup> Bug #1588489 opened: package maas-region-controller 2.0.0~beta3+bzr4941-0ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned
<mup> error exit status 1 <amd64> <apport-package> <need-duplicate-check> <third-party-packages> <xenial> <maas (Ubuntu):New> <https://launchpad.net/bugs/1588489>
<shewless> roaksoax: so any idea why the dhcpd.conf file isn't being generated? Is that rackd that would do that?
<kiko> shewless, it's a bug I think -- roaksoax?
<mup> Bug #1532478 changed: Can't add SM15K chassis - can't input management IP address of SM15k <oil> <MAAS:Invalid by newell-jensen> <https://launchpad.net/bugs/1532478>
<kiko> shewless, can you disable and reenable DHCP management on the vlan you are servicing?
<kiko> have you already>
<kiko> ?
<sjl> trying to troubleshoot maas API with Postman.  Anyone able to get this basic GET call to work via Postman? http://<insert-your-maas-server-ip>/MAAS/api/version
<kiko> sjl, I have never tried that -- does Postman do OAuth?
<sjl> yes 1.0 and 2.0
<gimmic> so when deploying, it installs and ends at "cloud-init finished..."
<gimmic> but the box stays in "deploying" state
<gimmic> closer look appears to be "Node installation - 'curtin' failed: configuring disk: sda"
<kiko> yep
<kiko> why did it fail to configure?
<gimmic> where can I find relevant log data?
<kiko> look through the install log
<roaksoax> shewless: have you enabled DHCP ?
<roaksoax> shewless: MAAS will only generate a dhcpd.conf is DHCP is enabled
<roaksoax> shewless: also, pastbin rackd.log
<gimmic> wheres the install log located?
<gimmic> I don't have access to the iron(yet), and don't see anything specific in maas' log
<kiko> gimmic, maas sends it back and it gets attached to the node page IIRC
<gimmic> the node page was where I pulled that from, I don't see any additional detail
<kiko> hm
<kiko> gimmic, are you doing anything custom to sda?
<kiko> and if not, is there anything unusual about it?
<gimmic> generic out-of-box install
<gimmic> r620, single SATA
<gimmic> this will be the second node deployed, the first onedid the same thing until I rebooted it
<gimmic> I'm obviously trying to streamline it a bit and tshoot things
<gimmic> but basic 14.04 LTS image, very generic hardware. I have 30 of these nodes to do
<kiko> gimmic, it's very odd tbh
<kiko> gimmic, so the entire install log has no hints other than the sda failure? can you put it in a pastebin
<gimmic> anywhere I can view the log in actual raw format
<gimmic> rather than the weberface
<gimmic> doesnt seem to be much useful in /var/log/maas
<gimmic> one other interesting question is that on some nodes I want to use a software raid to mirror two disks
<gimmic> but since it uses one off the bat, I can't seem to use maas to configure it to mirror the disk
<kiko> gimmic, you can just configure the software raid in the MAAS UI itself
<kiko> that should be pretty straightforward (and worked as far back as 1.9x)
<gimmic> but not before the node has been commissioned as far as I can tell
<kiko> correct
<kiko> the node needs to commission first
<kiko> oh
<kiko> are you stuck on the commissioning step?
<kiko> commissioning has an option you can select to let you log in and inspect
<kiko> have you checked that?
<kiko> (sorry, I thought you were mid-deployment)
<gimmic> oh, found it. I'm conflagulating two different 'issues' right now.
<gimmic> One set of nodes has two disks, and they will be mirrored /
<gimmic> that I just found under a node that was in ready state
<gimmic> Can I make templates for this somehow?
<kiko> not yet
<kiko> gimmic, you will be able to, but for the moment the solution is to script node config via the API
<gimmic> this is all the detail I can find on the deploy errors:
<gimmic> Node installation - 'curtin' failed: configuring storage	Thu, 02 Jun. 2016 15:23:01
<gimmic> Node installation - 'curtin' failed: configuring disk: sda	Thu, 02 Jun. 2016 15:23:01
<gimmic> Node installation - 'curtin' started: configuring disk: sda	Thu, 02 Jun. 2016 15:23:00
<gimmic> Node installation - 'curtin' started: configuring storage	Thu, 02 Jun. 2016 15:22:59
<shewless> roaksoax: I'll workon getting rackd log. DHCP is enabled as far as the UI tells me.. on my PXE network
<gimmic> kiko: oddly, if I reboot the node it deploys fine the second time
<gimmic> so it is something with partition creation or an error in the install/deploy script
<gimmic> about to test the theory with a third node
<kiko> wwird
<mup> Bug #1519358 changed: Non-fatal error message observed in PXE boot sequence <MAAS:New> <https://launchpad.net/bugs/1519358>
<mup> Bug #1588531 opened: [2.0b5] Deployed regions should be able to transistion back to machines <MAAS:In Progress by ltrager> <https://launchpad.net/bugs/1588531>
<shewless> roaksoax: here is rackd.log: http://paste.ubuntu.com/16930706. Note: for some reason if resolve.conf points to maas' own DNS server I can't resolve external DNS (like google).  DOn't know if that is related
<gimmic> Jun  2 20:47:59 ES18 [CLOUDINIT] util.py[DEBUG]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/cloudinit/stages.py", line 658, in _run_modules#012    cc.run(run_name, mod.handle, func_args, freq=freq)#012  File "/usr/lib/python2.7/dist-packages/cloudinit/cloud.py", line
<gimmic> 63, in run#012    return self._runners.run(name, functor, args, freq, clear_on_fail)#012  File "/usr/lib/python2.7/dist-packages/cloudinit/helpers.py", line 197, in run#012    results = functor(*args)#012  File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.py", line 38, in handle#012    util.runparts(runparts_path)#012  File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 651, in runparts#012    % (len(failed), len(
<gimmic> attempted)))#012RuntimeError: Runparts: 1 failures in 1 attempted commands
<gimmic> small ui bug during deployment: https://i.imgur.com/RgPKPSq.png
<gimmic> system is ON, dropdown option is 'turn system on' which actually turns it off..
<shewless> roaksoax/kiko: I tried to disable DHCP on all networks. I deleted the subnet entirely and then I re-added the subnet and re-enabled DHCP.. dhcpd still won't start because that file is missing
<shewless> what service is responsible for creating dhcpd.conf? I don't see any logs that indicate a problem there
<kiko> shewless, rackd does
<shewless> kiko: okay.. rackd doesn't want to tell me about any errors regarding dhcpd.conf I guess :(
<kiko> shewless, it seems to be confused about the IP address it needs to be listening on?
<shewless> kiko: interesting.. that might be the hint I needed
<jhobbs>  /wg 6
<shewless> kiko: so.. regiond.conf should be the "pxe" IP address right?
<shewless> kiko: It's working now.. I had it set to my external IP address instead somehow
<shewless> anyways.. dhcpd is running.. that's a good first step
<kiko> shewless, you have a single node running both rackd and regiond right?
<shewless> kiko: yup. and I think it's all working now (hopefully) just going to test deployment now
<kiko> shewless, cool
<kiko> shewless, this is an old bug, btw, that changing the MAAS server's IP address causes a mess
<kiko> gimmic, hmm
<gimmic> testing a theory, I changed the default filesystem to use LVM instead of flat
<gimmic> we'll see if deploys still err
<gimmic> i set one to allow ssh and got in to look
<gimmic> everything looks fine aside from that error I posted
<gimmic> seems if I deploy the same node twice, it deploys fine the second time
<gimmic> the raided node deployed fine
<mup> Bug #1523766 changed: Node stops commissioning and powers down before commissioning is complete <MAAS:Won't Fix> <https://launchpad.net/bugs/1523766>
<mup> Bug #1588547 opened: Generated bonding configuration is incorrect. <sts-needs-review> <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1588547>
<mup> Bug #1588547 changed: Generated bonding configuration is incorrect. <sts-needs-review> <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1588547>
<mup> Bug #1588547 opened: Generated bonding configuration is incorrect. <sts-needs-review> <curtin:New> <MAAS:Incomplete by mpontillo> <https://launchpad.net/bugs/1588547>
#maas 2016-06-03
<mup> Bug #1588547 changed: Generated bonding configuration is incorrect. <sts-needs-review> <curtin:Confirmed> <MAAS:Opinion by mpontillo> <https://launchpad.net/bugs/1588547>
<mup_> Bug #1588706 opened: MAAS should not add 'source /etc/network/interfaces.d/*.cfg' to /etc/network/interfaces <MAAS:New> <https://launchpad.net/bugs/1588706>
<mup> Bug #1588706 changed: MAAS should not add 'source /etc/network/interfaces.d/*.cfg' to /etc/network/interfaces <curtin:New> <MAAS:Invalid> <https://launchpad.net/bugs/1588706>
<gimmic> my successful deploy rate of 14.04 LTS is terrible
<gimmic> 10 identical nodes: 3/10 successful deployments
<kiko> gimmic, using MAAS 2.0, right?
<kiko> gimmic, we need to fix that rate and find out what is causing this crappy ratio
<gimmic> can i increase the logging with some sort of debug flag?
<gimmic> I'm about to fire off another 15 nodes at once
<gimmic> they're all identical hardware and in ready state
<kiko> gimmic, we tend to log everything
<kiko> gimmic, remind me, do the nodes commission fine? 20/20?
<gimmic> single disk R620's. Literally the only customization I'm doing is changing the hostname and static assignment of the network interface.
<gimmic> They all commission fine
<kiko> gimmic, always, right? never any commissioning failures?
<gimmic> you know, a more global log viewer would be nice from the weberface
<gimmic> like a general event log
<gimmic> rather than host-specific only
<kiko> gimmic, yeah, I hear you
<kiko> gimmic, we have /something/ like that with remote rsyslog aggregation but I'm not sure it's working in 2.x
<kiko> gimmic, anyway, when deployments fail, is the failure reproducible or rather random?
<gimmic> I fired off all ten last night before I left work and came back to the 3/10. Have not tshot it yet.
<gimmic> will poke at this next batch of 15 since I'm assigning IP/hostnames now.
<kiko> okay
<gimmic> Is there a document anywhere that describes where each subcomponent logs to? I see the logs under /var/log/maas
<kiko> gimmic, maas.io/docs has the architecture -- there are basically two main components, rackd and regiond
<kiko> rackd talks to the nodes (pxe, ipmi, etc)
<kiko> regiond hosts the API and Web server
<mup> Bug #1588846 opened: [2.0b6]  builtins.ValueError: invalid literal for int() with base 10: '' <MAAS:Triaged> <https://launchpad.net/bugs/1588846>
<dimitern> does anyone know how long 'Disk erasing' is expected to take on a 256GB SSD ?
<dimitern> is it a mult-pass secure wipe or something?
<dimitern> roaksoax: ^^
<roaksoax> dimitern: nope, but it iwll take a while
<roaksoax> dimitern: it just wipes the whole disk
<dimitern> roaksoax: yeah, it just finished ~10m for that 256GB SSD; interestingly the other 2 nodes with 120GB SSDs are taking longer (all were started pretty much at once)
<mup> Bug #1588857 opened: [2.0b5] sudo: no tty present and no askpass program specified <MAAS:New> <https://launchpad.net/bugs/1588857>
<mup> Bug #1588868 opened: [2.0b5] While monitoring service 'dhcpd/tgt/dhcpd6/proxy' an error was encountered: Unable to parse the output from systemd for service <MAAS:New> <https://launchpad.net/bugs/1588868>
<mup> Bug #1588875 opened: [2.0-b6] Deploying a trusty (but not xenial) node frequently fails during storage setup of curtin  <MAAS:New> <https://launchpad.net/bugs/1588875>
<gimmic> kiko: http://i.imgur.com/jzb7djY.png
<gimmic> fired this off about 30 minutes ago, all the nodes are hung in deploying state, same logs:
<gimmic> Node installation - 'curtin' failed: configuring storage	Fri, 03 Jun. 2016 10:26:55
<gimmic> Node installation - 'curtin' failed: configuring disk: sda	Fri, 03 Jun. 2016 10:26:55
<gimmic> Node installation - 'curtin' started: configuring disk: sda	Fri, 03 Jun. 2016 10:26:54
<gimmic> Node installation - 'curtin' started: configuring storage	Fri, 03 Jun. 2016 10:26:54
<gimmic> using LVM now
<gimmic> none of them have failed yet, but I suspect there's a timeout somewhere ticking down
<dimitern> gimmic: just filed that bug above for this
<gimmic> dimitern: I'm not sure it's LVM related
<gimmic> initially I was doing flat ext4
<gimmic> and still run into it
<dimitern> gimmic: it is
<dimitern> gimmic: the default layout is flat, but I've created a VG on top of that
<dimitern> in order to emulate having 2 distinct partitions
<gimmic> Yeah, I'm saying I see the same error/behavior without using LV
<gimmic> even installing w/ just flat ext4 curtain hangs in the same way
<gimmic> ..curtin
<dimitern> gimmic: ah, I see .. not good :/
<mup> Bug #1588868 changed: [2.0b5] While monitoring service 'dhcpd/tgt/dhcpd6/proxy' an error was encountered: Unable to parse the output from systemd for service <MAAS:Incomplete> <https://launchpad.net/bugs/1588868>
<kiko> gimmic, can you feed into https://bugs.launchpad.net/maas/+bug/1588875 as well?
<mup> Bug #1588907 opened: [2.0b6] django.db.utils.IntegrityError: insert or update on table "piston3_consumer" violates foreign key constraint "piston3_consumer_user_id_4ac0863fa7e05162_fk_auth_user_id" <MAAS:New> <https://launchpad.net/bugs/1588907>
<kiko> hm
<mup> Bug #1588914 opened: [2.0b6] MAAS writes DHCP multiple times while not much is going on <MAAS:New> <https://launchpad.net/bugs/1588914>
<gimmic> so just fyi
<gimmic> if I use ipmi to power cycle the failed deployment node, release it, and re-deploy
<gimmic> it seems to deploy fine
<gimmic> I suspect it's something with how the partition exit code is
<gimmic> when I redeploy, the partitions are already there
<gimmic> okay, confirmed.. If I just shut the node down and netboot it again
<gimmic> it works fine after re-deploying
<gimmic> so the partition creation is exiting with an error code, but completing
<mup> Bug #1588875 changed: [2.0-b6] Deploying a trusty (but not xenial) node frequently fails during storage setup of curtin  <curtin:Confirmed> <MAAS:Invalid> <https://launchpad.net/bugs/1588875>
<kiko> roaksoax, smoser: is there a way for gimmic to stop the deployment process from rebooting and ssh in?
<gimmic> so my current 'workaround' is let it time out to failed deployment, mark the nodes as broken, mark them as fixed, deploy
<gimmic> that allows me to avoid fiddling with out of band ipmi resets
<gimmic> I have now deployed 30 nodes using that methodology
<smoser> gimmic, 2 ways
<smoser> a.) maas server change /etc/maas/preseeds/curtin_userdata
<smoser> see 'power_state' there
<smoser> comment that out will do
<smoser> b.) ssh in during deplyoment and sudo touch /run/block-curtin-poweroff
<kiko> gimmic, that will let you try and re-run the partitioning command and see what's failing
<kiko> could it be a gpt/efi thing?
<smoser> kiko, you can also just put it comissioning and have it not power itself off
<smoser> and then do whatever you want
<kiko> smoser, well, I think the problem is right now gimmic doesn't even know what curtin is doing
<kiko> so doing it in the deployment phase is going to be easier
<mup> Bug #1588154 changed: [2.0b5] Deploying node fails when it has VLAN configuration <MAAS:Invalid> <https://launchpad.net/bugs/1588154>
#maas 2016-06-04
<mup> Bug #1589042 opened: UX: User preference screen truncates SSH keys poorly <MAAS:Triaged> <https://launchpad.net/bugs/1589042>
<mup> Bug #1589042 changed: UX: User preference screen truncates SSH keys poorly <MAAS:Triaged> <https://launchpad.net/bugs/1589042>
<mup> Bug #1589042 opened: UX: User preference screen truncates SSH keys poorly <MAAS:Triaged> <https://launchpad.net/bugs/1589042>
<SaltyVeryMuch> Hi, I kinda have an issue with the build on the ubuntu 16.04LTS, the DHCP isn't working
<mup> Bug #1589140 opened: No WOL option in latest MAAS version for 16.04 also the Manual settings crashes <MAAS:New> <https://launchpad.net/bugs/1589140>
#maas 2016-06-05
<gromero> hI
<gromero> Does anybody know what maybe the cause of this fishy error:
<gromero> http://storage1.static.itmages.com/i/16/0605/h_1465104265_3548480_49a6c24ad5.png
<gromero> => Extra data: line 1 column 4 (char 3)
<gromero> Hi. Is the âClustersâ tab removed now from last version?
<gromero> *stable version
<gromero> *2.0
<gromero> beta
<D4RKS1D3> someone know how to add new events in maas?
#maas 2017-05-29
<bbn> hi
<Elangovan> Hi
<Elangovan> I am new to MAAS
<Elangovan> hello
<cnf> hello
#maas 2017-05-30
<deej> Hey folks, I'm running into some weirdness with maas 2.2 and VLANs and I'm hoping someone could give me a hand diagnosing it
<deej> Specifically, VLAN interfaces created by MaaS don't appear to work properly for some reason, but if I vconfig rem the vlan from the interface, then add it back, it works
<mup> Bug #1694556 opened: [2.2, UI] RSD POD -  No UI feedback when failing to create a machine <error-surface> <rsd> <MAAS:New> <MAAS 2.2:New> <https://launchpad.net/bugs/1694556>
#maas 2017-05-31
<benlake> Iâm apparantly missing something regarding where to access the API. Is it provided on the same port as the WebUI (Iâd assume so)? /api/version/ and /api/2.0/ both return âNo Such Resource"
<benlake> whelp. Just figured it out, /MAAS/api/version/
<benlake> thatâs a tad unexpected. Had the impression /MAAS/ and /API/ would be respective UI and API roots.
<benlake> The docs do not mention the MAAS_URL is the prefix.
<pmatulis> benlake, please provide the URL to the doc you followed
<benlake> https://docs.ubuntu.com/maas/2.1/en/api
<pmatulis> benlake, ok, interesting. the intended page to get started with the CLI/API is:
<pmatulis> https://docs.ubuntu.com/maas/2.1/en/manage-cli
<benlake> err, I wanted to get started with the API.
<pmatulis> we try to direct people at this top-level page:
<pmatulis> https://docs.ubuntu.com/maas/2.1/en/intro-management
<pmatulis> but i can see how you can end up directly on the API page
<benlake> API Documentation is a giant top level button :D
<pmatulis> right
<benlake> Iâm not using the CLI tool, I was looking for, and found, the API docs :P
<benlake> which are great. Only trip was seeing all the /api roots paths, and not realizing that was in addition to whatever the configured MAAS_URL is
<pmatulis> the API docs are generated from the maas code. i don't have any influence there
<pmatulis> roaksoax, possible to add some clarity? ^^^
<roaksoax> benlake: cli is just a reflection of the API
<roaksoax> benlake: it is auto generated
<roaksoax> benlake: MAAS_URL is a config on /etc/maas/rackd.conf
<benlake> Iâm confused by what youâre attempting to convey
<benlake> why is the CLI relevant to wanting to access the API directly and discoverying how?
<benlake> and yes, I realize where the MAAS_URL is, it was just not apparent it needed to be the prefix to the paths referenced by the API docs.
<benlake> the service provided on 5240 could provide both /api/ and /maas/
<benlake> nothing is wrong. I was just commenting that I tripped over /api needing to be /maas/api
<pmatulis> benlake, in passing, for any general documentation issues you can open them here:
<pmatulis> https://github.com/CanonicalLtd/maas-docs/issues/new
<benlake> yeah, the threshold for making paperwork is tricky.
<benlake> for a minor trip up that could easily be an expectation of mine simply not being the case; I leaned toward saying it here and seeing if itâs worth the issue.
<benlake> thanks for he ref.
<pmatulis> benlake, your comment is valid IMO. and IRC is fine to discuss. i really meant for any future issues
<benlake> if I make interfaces changes to a controller, is there a way to get MAAS up to speed? Are the fabric references tied to interface names internally?
<benlake> are PPAs added via MAAS available through the proxy?
 * benlake is guessing not since Package Repositories says ââ¦subsequently deployedâ
<mup> Bug #1694759 opened: RSD Pod refresh shows ComposedNodeState is "Failed" <MAAS:Confirmed for newell-jensen> <MAAS RSD :Confirmed for newell-jensen> <https://launchpad.net/bugs/1694759>
<mup> Bug #1694761 opened: Invalid Pod Resources error shows in API but not UI <MAAS:Confirmed for blake-rouse> <https://launchpad.net/bugs/1694761>
<mup> Bug #1694761 changed: Invalid Pod Resources error shows in API but not UI <MAAS:Confirmed for blake-rouse> <https://launchpad.net/bugs/1694761>
<benlake> Hmm, are the MAAS NTP settings used in the PXE boot images? Iâm seeing ntp.ubuntu.com used during an attempted Release operation.
<benlake> Iâm also seeing massing ureadahead errors, but I think theyâve been there for a while.
<benlake> s/massing/massive/ - by massive I mean like 29k+ entries in syslog
<mup> Bug #1694761 opened: Invalid Pod Resources error shows in API but not UI <MAAS:Confirmed for blake-rouse> <https://launchpad.net/bugs/1694761>
<mup> Bug #1694767 opened: RSD composition not setting local disk tags <MAAS:Confirmed for newell-jensen> <MAAS RSD :Confirmed for newell-jensen> <https://launchpad.net/bugs/1694767>
<roaksoax> benlake: the ureadahead errors are being looked at, non-maas issue though
<roaksoax> benlake: that's a bug in ureadahead
<benlake> thanks for the confirmation
<roaksoax> benlake: as far as the proxy, it is a cashing proxy , so your ppas would still use the default proxy
<roaksoax> benlake: on NTP, the commissioning environment will configure ntp
<benlake> neat @ proxy
<benlake> Re: ntp, not seeing that happen
<roaksoax> benlake: go to /etc/ and grep for the expected config
<roaksoax> i'd need to check
<roaksoax> benlake: unless it is systemd timesync that's doing those queries
<benlake> yeah, its systemd
<benlake> Iâm seeing these things, which are likely not related at all, because a Release is failing (multiple tries)
<benlake> Iâve deployed/released this node a few times over the last few days, and now itâs hung
<benlake> are there any telltales in the logs that could confirm for me the booted envronment is/will attempt to erase didks?
<benlake> also disks
<roaksoax> benlake: did the machine enter disk erasing mode and started erasing ?
<benlake> yes, then timed out
<roaksoax> benlake: you should see progress in the events log
<roaksoax> benlake: timeout when ?
<benlake> while waiting for disk erase to complete
<benlake> it never timed out before...
<roaksoax> benlake: do you have an example of how it did ?
<benlake> I popped into KVM to see what was up and I caught the tail end of âerasing disks completeâ and then it powered off
<benlake> it just timed out again, so maybe. what do you mean example?
<benlake> âNode changed status - From 'Disk erasing' to 'Failed disk erasing'"
<roaksoax> benlake: do you have a full log ?
<benlake> sure
<roaksoax> benlake: also /var/log/maas/rsyslog/<machine-name>/<date>/messages
<benlake> would you only the latest release attempt and for me to remove ureadahead errors?
 * benlake notices a lot of snapd nonsense attemting to access the internet
<benlake> thatâs annoying.
<roaksoax> benlake: the ureadahead is unrelated though
<benlake> right, I was just going to remove it from the log so you donât have to see it...
<benlake> itâs 29k entries of noise
<benlake> roaksoax: this is allt he runs sans ureadahead http://paste.ubuntu.com/24728711/plain/
<benlake> I just noticed this, which is interesting: cloud-init[3156]: request to http://10.128.1.130:5240/MAAS/metadata//2012-03-01/ failed. sleeping 32.: HTTP Error 409: CONFLICT
<benlake> according to the logs, it says it erased the disks. MAAS state for this node is âfailed disk erasingâ. I guess Iâll mark it as Broken, Fixed, and attempt to Deploy it
<roaksoax> benlake: so, i need a full node event log
<roaksoax> benlake: and the syslog
<roaksoax> to know what may be wrong
<benlake> all I did was grep -v ureadahead, you want all of that?
<roaksoax> benlake: i think wiith what I have wrt to cloud-init is enough
<benlake> the only thing I stripped was ureadahead
<benlake> if that is unrelated, you have everything
<roaksoax> benlake: my thinking is that this could be the culprit: May 31 17:55:52 electron systemd[1]: Failed to start Apply the settings specified in cloud-config.
<roaksoax> May 31 17:55:52 electron systemd[1]: cloud-config.service: Unit entered failed state.
<roaksoax> May 31 17:55:52 electron systemd[1]: cloud-config.service: Failed with result 'exit-code'.
<roaksoax> benlake: do you have a node event log ? You can go to the UI and grab it from there
<roaksoax> specially the "full"
<roaksoax> or you can grab it from the API on the 'events' endpoint
<benlake> lemme see
<benlake> oh, you want the MAAS events, got it
<benlake> ugh, between timesyncd and snapd, this thing sits around timing out for 5 minutes.
<roaksoax> benlake: http://pastebin.ubuntu.com/24728895/
<roaksoax> that's the issue
<benlake> I saw that before pasting and was about to mention it
<benlake> here is the entry in MASS UI, https://screencast.com/t/9ccpHGoc
<benlake> is something wrong with that?
<benlake> I went through the events exercise anyhow: http://paste.ubuntu.com/24728917/
<benlake> hmm, it seems cloud-init might not know to use a proxy
<benlake> is it a project expectation deploying nodes have internet access?
 * benlake is just wondering
<roaksoax> benlake: no. MAAS can work on completely offline environments
<benlake> yeah, I see it can, Iâve been using it that way
<roaksoax> benlake: but my guess of what's happening here is that sudo add-apt-repository ppa:<etc> is failing behind a proxy
<benlake> just digging in the logs due to this PPA thing seems to show a lot of internet access attempts on these ephemeral images
<benlake> roaksoax: aye, that seems to be the case. It not being configured to use one
<benlake> https://askubuntu.com/questions/724224/cant-add-ppa-on-ubuntu-15-10-user-or-team-does-not-exist
<benlake> old, but likely the same issue
<roaksoax> benlake: the other thing you could do, is to add the ppa as a URL
<benlake> I think I tried that, but the UI said it was invalid
<benlake> prevented me from saving it. let me mess with it again
<roaksoax> benlake: do it as 'add repository'
<benlake> gotcha, does this look correct? https://screencast.com/t/lJy1PUpE49i
<pmatulis> benlake, you want to add a PPA right?
<benlake> yes, but see the discussion above
<pmatulis> well, i know i added a PPA before so i'm confused. are you using the default proxy?
<benlake> yes
<benlake> are you sure your node didnât have internet access when being deployed?
<benlake> cloud-init attempts to use add-apt-repository when booted into the ephemeral environment and does not honor any proxy
<pmatulis> oh. right, of course it did
<pmatulis> enabling a PPA requires internet access
<benlake> not sure MAAS devs expect that to be required.
<pmatulis> if maas is supposed to be "offline ready" then i guess that's a bug. but i'm not sure how you can enable a PPA w/o internet
<benlake> if they do, another update to the docs would be a good idea :)
<pmatulis> can you file one?
<benlake> if the env vars htto_proxy and https_proxy are set, itâll use it
<benlake> only if roaksoax confirms
<benlake> *http_proxy
<benlake> roaksoax: adding as non-PPA worked, thanks! Also the URL in my screenshot was completely wrong. I failed on that one.
<roaksoax> :)
<roaksoax> benlake: glad to hear it works now
<pmatulis> benlake, you were able to add a PPA as a non-PPA? 'splain?
<benlake> https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
<benlake> green link, âTechnical details about this PPA"
<benlake> shows the direct repo URL
<benlake> can use that with the add repo option instead of add ppa. basically the PPA helpers just lookup that URL and fetch the repoâs public key for you
<benlake> seems I have another auto IP address selection issue. The NTP server being set for deployed nodes is not the desired/functional IP
<benlake> I like the cascade of NTP servers, region, rack ,etc.
<benlake> but am I going to need to use the âforce external NTP serverâ option here?
<benlake> dang, but that would force the region controller to use itself as an NTP server.
<pmatulis> benlake, thanks (for the PPA stuff)
<pmatulis> benlake, what URL did you end up using? you said you made a mistake originally?
<benlake> bad: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
<benlake> good: http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu
#maas 2017-06-01
<pmatulis> benlake, thank you
<BlackDex> Hello there. Does any of you know of any conectifity issues when using a riverbed appliance inbetween a maas controller and the clients?
<BlackDex> i'm having problems with maas which tells the client during commisioning the auth has failed getting a 401 response
<roaksoax> BlackDex: maas deployed machines need to be able to contact the region controller for metadata service
<roaksoax> BlackDex: the IP they would contact is the one on rackd.conf
<roaksoax> BlackDex: so that's likely what's causing your issues
<roaksoax> using something in between
<BlackDex> roaksoax: well it can conntact the server
<BlackDex> the region controller that is
<BlackDex> but i just get 401 access denied
<BlackDex> even though i have wireshark logs that states sending the oauth tokens etc..
<BlackDex> and i see exact the same that is sent is also received without any modifications
<roaksoax> BlackDex: during commissioning, I believe once it says "i'm done commissioning" then some delayed messages may not make it due to that
<roaksoax> that could be it
<BlackDex> it doesn't even fetch the commissioning script :(
<BlackDex> 401 access denied
<roaksoax> do you have sample logs ?
<BlackDex> one moment, that should be in the rsyslog folder then?
<roaksoax> it should if cloud-init was able to sent it back
<BlackDex> roaksoax: just a moment, ill try to create a new fresh log
<BlackDex> roaksoax: the strange thing is that it worked for a long long time
<BlackDex> and suddenly it stoped
<BlackDex> roaksoax: http://paste.ubuntu.com/24737218/
<BlackDex> i think it maybe is an ntp problem
<BlackDex> it seems the ntp-sync on the region controller wasn't synced for a long time
<BlackDex> also, there are apperently to many hops between the clients and the region controller/external etc.. for the ntp
<BlackDex> so i now have a ntp server which should be reachable for all clients/servers within the network
<BlackDex> lets see what that does
<BlackDex> roaksoax: that doesn't work
<BlackDex> atleast the clock skew adjustment is gone now
<BlackDex> roaksoax: http://paste.ubuntu.com/24737421/
<BlackDex> that is second attempt with the NTP fixed now
<roaksoax> BlackDex: http://pastebin.ubuntu.com/24737462/
<roaksoax> BlackDex: that's the erro
<BlackDex> that causes the 401 errors :S
<BlackDex> oke, lets see what it does when i disable that
<BlackDex> i didn't know that was enabled btw
<BlackDex> good spot!
<BlackDex> lets see
<BlackDex> it think it would be strange if that causes the 401 errors
<BlackDex> but lets see
<roaksoax> BlackDex: the thing is that cloud-init sends a meesage to MAAS and tells MAAS "commissioning has failed", so MAAS goes and says "ok, i'm gonna remove the nonce"
<roaksoax> BlackDex: hence maas can't authenticate
<BlackDex> ah
<BlackDex> darn
<BlackDex> and it works
<benlake> I guess when it rains PPA errors it pours PPA errors :P
<BlackDex> thanks!
<benlake> roaksoax: speaking of NTP, did you see my situation with IP selection?
<BlackDex> trying the deploy now, seems to look good
<BlackDex> strange that this didn't have any impact on the other rack controllers :S
<BlackDex> hmm
<BlackDex> i think they were able to download the gpg key, and this specific site doesn't
<xygnal> roaksoax: how is grub config managed on install?  We need to do some tweaking to the defaults.
<piwi3910> hey people, hope anyone can point me to a solution
<piwi3910> i'm running maas 2.2
<piwi3910> any node i boot, physical or virtual always fails the boot with:
<piwi3910> cloud-init can not apply stage final, no datasource found
<piwi3910> any clues
<piwi3910> fresh install
<benlake> I guess thatâs my cue
<benlake> hello piwi3910, you are me 5 days ago
<piwi3910> hehhe cool, back to the future
<piwi3910> i've installed maas before, never had this issue
<piwi3910> no clue what's going on
<piwi3910> so tell me your magic
<benlake> look at both /etc/maas/rackd.conf and /etc/maas/regiond.conf
<benlake> youâll probably say, âah ha!â when looking at one of those.
<piwi3910> regiond points to my public side of the maas
<piwi3910> rackd to localhost
<benlake> and what subnet is the deployed node being asked to land on?
<benlake> and can said deployed node route to this âpublic sideâ you speak of.
<benlake> s/./?/
<piwi3910> so this is what i noticed, in the dhcp the GW for the pxe network is filled in
<piwi3910> but when the hosts boots, i can only ping it from the maas node
<piwi3910> so the GW is not being taken
<benlake> careful, when you say it boots, you mean when it is booted into the ephemeral image, correct?
<piwi3910> yep
<benlake> I tried troubleshooting network stuff from that image and it behaves very oddly.
<piwi3910> it boots up get's in to ubuntu
<piwi3910> i see the network being brought up
<piwi3910> but i don't see the gw being set
<piwi3910> another machine i tried in the same vlan
<piwi3910> sure the GW works
<piwi3910> it's the image not taking the GW from dhcp
<benlake> does it have a default route?
<piwi3910> or dhcp not providing it
<piwi3910> nothing
<benlake> is that the answer to my default route question?
<piwi3910> well i can do what you propose and run the stuff on the internal pxe network
<piwi3910> so it doesn't go to the public side anymore
<piwi3910> but that would only work for one pxe network
<benlake> backing up, the problem is the deploying/commissioning/enlisting node cannot speak to the rack controller
<piwi3910> kinda fucks up the point of having multiple deploy networks
<benlake> I donât understand what you mean, or your expectation of, âmultiple deploy networks"
<piwi3910> ok:
<piwi3910> from what i can see, for some reason the node doesn't get the gW
<piwi3910> becasue of that it cannot get to the rackserver
<piwi3910> as that one has default it's config on the public side
<piwi3910> so what i can do is edit the file
<piwi3910> and put the pxe side in the rackd config
<benlake> for the subnet you enabled DHCP on, did you confirm a gateway is set?
<piwi3910> yes dhcp is set and gw is defined
<benlake> are you trying to enlist or commission?
<piwi3910> commission
<benlake> so you manually added the hardware?
<piwi3910> yep
<piwi3910> have a few dell server r610 i tried
<piwi3910> and some vm's
<piwi3910> all have the same issue
<piwi3910> i'm gonna try another image
<benlake> and Iâm guessing the motifivation for that is because enlistment didnât work? :)
<benlake> what image are you trying? (Iâm pretty sure it isnât image related)
<piwi3910> now the default 16.04
<benlake> can you screen cap the PXE boot when it acquires an address and poops our a helpful config line?
<benlake> that image is fine, thatâs all Iâve been using.
<piwi3910> ok i'll do a screencap and drop it on dropbox
<benlake> thatâs how I discovered the rack IP when I had this issue.
<benlake> there sure is a lot of code to discover routing information...
<benlake> more precisely, attempt to discover what can route to what.
 * benlake looks at Gavin
<piwi3910> ok very interesting
<piwi3910> just got it fixed on the fault image
<piwi3910> default image
<piwi3910> only thing i did was change the kernel minimum to the hwe kernel
<benlake> err, whatâs a default image?
<piwi3910> now all servers boot fine
<piwi3910> the 16.04
<benlake> oh interesting. guess it was driver related
<piwi3910> on every VM and physical server?
<piwi3910> with different nics and all
<piwi3910> that would be weird
<piwi3910> any way, i'll do the screencap anyway
<benlake> the VMs, yeah, weird. But you only mentioned one bare metal server type
<benlake> and youâve said nothing as to what your VMs are.
<benlake> for all I know they are using sr-iov and thus need more awareness of the underlying NIC.
<benlake> âNTP servers, specified as IP addresses or hostnames delimited by commas and/or spaces, to be used as time references for MAAS itself, the machines MAAS deploys, and devices that make use of MAAS's DHCP services.â
<benlake> Do I understand âMAAS itselfâ to mean the region and rack controllers, correctly?
<roaksoax> benlake: yes
<benlake> alright. then my issue stands. NTP server IP being selected is non-optimal.
<roaksoax> benlake: what version are you running ?
<benlake> 2.1
<benlake> Iâll happily upgrade when it hits GA
<roaksoax> benlake: 2.2 is ga already
<roaksoax> benlake: which 2.1 ?
<roaksoax> 2.1.3 ?
<benlake> well, its PPA GA right? I donât see it in backports for xenial
<roaksoax> benlake: the same version in PPA will hit xenial once the SRU process goes through
<benlake> 2.1.3+bzr5573-0ubuntu1 (16.04.1)
<benlake> right, waiting on SRU I suppose
<roaksoax> benlake: we wont be doing any maintenance on 2.1 anymore
<benlake> Iâm not stuck. I just ansibled the ntp server.
<benlake> understood.
<benlake> If I remember too and see this in 2.2, Iâm sure Iâll whine about it :D
<roaksoax> benlake: cool, if you could file a bug then it would be great
<roaksoax> benlake: provided that in 2.2 we fixed a but wrt
<roaksoax> bug*
<benlake> again, it is dicey as to whether it is an actually flaw or just a awkward use case
<benlake> I saw again with regards to IP selection discussions in general
<benlake> s/saw/say/
<roaksoax> benlake: i do know that there's been some weird things in NTP
<roaksoax> benlake: i can't remember if we backported that to a later 2.1
<benlake> there is a lot of âroute findingâ code that seems to only be used by NTP at first glance
<benlake> so could definitely be isolated weirdness
<xygnal> roaksoax hey?
<roaksoax> benlake: it is indeed, as we try to find all rack controllers in the same vlan to have access to ntp
<roaksoax> xygnal: hey!
<xygnal> roaksoax asked you a question a little while ago
<roaksoax> xygnal: that's done by curtin
<xygnal> roaksoax:  i'll check curtin trunk docs for details about grub
<roaksoax> or the hooks effectively
<roaksoax> xygnal: any particular issues you've seen  ?
<xygnal> roaksoax when does this apply?  We have a client who is applying grub changes in their user_data script
<xygnal> and they are discovering that those settings are being over-written by MAAS
<xygnal> roaksoax hm... the grub section does not even cover kernel options
<xygnal> its the GRUB_CMDLINE_LINUX variable
<xygnal> that is being set
<roaksoax> xygnal: are you modifying this ti inject custom kernel options for the deployed machine?
<roaksoax> to*
<xygnal> xygnal yes, such as console= settings we need
<xygnal> and anything else that may come up
<roaksoax> xygnal: https://docs.ubuntu.com/maas/2.1/en/installconfig-nodes-kernel-boot-options
<roaksoax> xygnal: https://docs.ubuntu.com/maas/2.1/en/manage-cli-advanced#specify-kernel-boot-options-for-a-machine
<xygnal> roaksoax looks like this can only be done via global UI, or per-host CLI? no API?
<roaksoax> xygnal: everything can be done via the api/cli. The UI some stuff is missing indeed
<xygnal> roaksoax I was digging through the API docs and did not see grub options
<xygnal> maybe i should look for kernel =p
<roaksoax> xygnal: the CLI is autogenerated from the API
<roaksoax> at least the current one
<xygnal> kernel_opts?
<xygnal> roaksoax and this is passed for Custom/CentOS as well?
<roaksoax> xygnal: i can't recall of the top of my head, but I think we do
 * roaksoax otp
<xygnal> roaksoax testing that out now.  client also wants to change other GRUB settings,  are those hard-coded?
<xygnal> roaksoax like GRUB_TIMEOUT= and others
<benlake> bah! now what! Jun  1 16:11:27 fair-ewe cloud-init[983]: E: Malformed entry 1 in list file /etc/apt/sources.list.d/linbit-drbd9-stack_4.list (Component)
<benlake> could someone point me to docs regarding the proxy? specifically, Iâd like to flush the cache
<xygnal> roaksoax  hm... seems this setting is not taking.  either globally or per node it appears to be ignored?  When does it apply? client is noticing this during user_data script execution.
<benlake> I donât know why this repo I added is causing commissioning issues (KVM guest). The repo I added has been enabled while deploying 3 bare metal nodes.
<benlake> ^ that completed successfully
<xygnal> roaksoax during user_data execution, that line shows as ="" instead of with our custom tag settings or global settings
<benlake> totes a bug. this is what is ending up in /etc/apt/sources.list.d/linbit-drbd9-stack_4.list
<benlake> deb http://ppa.launchpad.net/linbit/linbit-drbd9-stack/ubuntu  main
<benlake> ie. sans xenial
<benlake> enlist, commission fails. deploy succeeds. with the additional repo enabled.
<benlake> doesnât affect releasing, thatâs interesting.
<roaksoax> xygnal: i'd need to investagte. this could be due how this is generally handled in ubuntu/debian vs how it is handled on centos, but IIRC, we would just copy those extra params and use them for the installed system as well
<xygnal> roaksoax right now I am coding a curtin script to automatically nuke the added lines
<xygnal> but that is not ideal
<xygnal> the grub config has value settings PRIOR to maas modifying it
<xygnal> but mass puts its setting BELOW those
<xygnal> which causes it to ignore the higher ones
<xygnal> right now my script jsut rips those extra lines out during deploy to be sure they do not interfere.  this is not something i could easily do for different clients, so i'd like to bug track this to see if COS is really unsupported/get a bug/feature requets going
<roaksoax> xygnal: You should file a bug and submit a patch :). That'd be awesome!
<xygnal> roaksoax: I dont know python, so patching it would be quite a challenge.   I will help to debug it though, if you can give me some hints on how to verify
<roaksoax> xygnal: i'll need to investigate. Haven't look at that code in ages. But I'd recommend you to file a bug
<roaksoax> so it is tracked at least
<roaksoax> benlake: https://bugs.launchpad.net/maas/+bug/1695083
<roaksoax> benlake: that's what you were hitting earlier today wrt NTP
<benlake> hmm, perhaps. I did have a new fabric pop up, but canât quite pin down timing
<benlake> interesting, so it seems /etc/systemd/timesyncd.conf is never updated. Is that correct?
<mup> Bug #1695083 opened: [2.2] NTP misconfigured after the Rack discovered a new 'lxdbr0' interface <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1695083>
<roaksoax> benlake: we dont update timesyncd.conf we install ntpd
<roaksoax> benlake: effectively is your same issue
<benlake> sure, but the nummer of not touching timesyncd means there is double duty AND I get to see it fail in the logs :P
<benlake> but Iâll ansible that away too I suppose.
#maas 2017-06-02
<ltrager> roaksoax, benlake: There is some preliminary discussion for timesyncd support in cloud-init in https://bugs.launchpad.net/cloud-init/+bug/1686485
<ltrager> rharper: Any progress on that ^?
<mup> Bug #1695229 opened: Badblocks: Value too large for defined data type invalid end block <MAAS:New> <https://launchpad.net/bugs/1695229>
<mup> Bug #1695229 changed: Badblocks: Value too large for defined data type invalid end block <MAAS:New> <https://launchpad.net/bugs/1695229>
<mup> Bug #1695229 opened: Badblocks: Value too large for defined data type invalid end block <MAAS:New> <https://launchpad.net/bugs/1695229>
<cshen> Hi guys, we saw version 2.2 is released. but in our production system, we want to stick to 2.1 version. How can we still get 2.1 from PPA?
<piwi3910> can someone point me to how to install 3rd party drivers before the commission
<piwi3910> I have some fusionIO cards in my servers i want to show up as disks
<piwi3910> install procedure is straightforward just apt-get install a few packages from my repo
<cshen> piwi3910: add repo to apt source?
<roaksoax> piwi3910: /etc/maas/drivers.yaml
<roaksoax> piwi3910: or you could add your own commissioning script
<mup> Bug #1695290 opened: [2.2] If 99-maas-02-capture-lldp times out, the machine gets marked failed commissioning <field> <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1695290>
<piwi3910> thxs guys
<mup> Bug # opened: 1695301, 1695307, 1695309, 1695311, 1695312
<mup> Bug #1695339 opened: [2.2] MAAS racks connect to too many region endpoints <MAAS:Triaged> <MAAS 2.2:Triaged> <https://launchpad.net/bugs/1695339>
<mup> Bug #1695463 opened: [2.2.0+bzr6057-snap] Stacktrace appears when pointing to a non-existent region controller <docteam> <snap> <MAAS:New> <https://launchpad.net/bugs/1695463>
<benlake> roaksoax: âown commissioning scriptâ? Have something you can point at about that. Iâm interested.
<benlake> has anyone bothered to proxy snapd updating?
#maas 2017-06-03
<mup> Bug #1679322 changed: maas-dhcp upgrade to 1.9.5+bzr4599-0ubuntu1~14.04.1 fails to start installed isc-dhcp-server <maas (Ubuntu):Expired> <https://launchpad.net/bugs/1679322>
#maas 2017-06-04
<mup> Bug #1695704 opened: Unclear what '25% connected to region controllers' means <MAAS:New> <https://launchpad.net/bugs/1695704>
<mup> Bug #1695704 changed: Unclear what '25% connected to region controllers' means <MAAS:New> <https://launchpad.net/bugs/1695704>
<mup> Bug #1695704 opened: Unclear what '25% connected to region controllers' means <MAAS:New> <https://launchpad.net/bugs/1695704>
<myvault> Hello
<myvault> Is anyone here?
<benlake> FYI, when attempting to change the hostname of a New machine to a name which is already defined, silently errors. I see the error in syslog, but the UI does nothing.
<benlake> the conflict was between a New machine and an existing Device, if that played a part
#maas 2018-05-29
<ebbex> What's the deal with Pods? I've spec'd a virsh connection to a box, and VMs are created and start up, but they have no NICs. How can I spec which bridge on the KVM host they should attach to? Or should I create a bridge with a special name perhaps?
<jac_cplane> how can I specify a specific kernel version in MAAS 2.3.1 ?   We found a bug in kernel 2.4.0-127 that didn't exist in 0-124 but cannot specific 0-124 in maas anymore    https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1773165
<jac_cplane> looks like Maas no longer allows "PPA" to be release, only supports daily sync
<mup> Bug #1774016 opened: [2.5] Add pool 'Description' text box is too large <resource-pools> <MAAS:Triaged> <https://launchpad.net/bugs/1774016>
<mup> Bug #1774017 opened: MAAS doesn't support multiple vlans with unknown IDs on the same fabric <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774017>
<mup> Bug #1774020 opened: [2.5] Resource pools listing should show 'Empty pool' instead of '0 of 0 Ready' <MAAS:New> <https://launchpad.net/bugs/1774020>
<mup> Bug #1774023 opened: [2.5] Default pool show be highlighted <resource-pools> <MAAS:New> <https://launchpad.net/bugs/1774023>
<mup> Bug #1774024 opened: [2.5] Top-level navigation for 'Pools' should be moved to 'Machines' tab <resource-pools> <MAAS:New> <https://launchpad.net/bugs/1774024>
<mup> Bug #1774025 opened: [2.5] On the machine listing page, remove the 'Pool' tab next to owner. <resource-pools> <MAAS:New> <https://launchpad.net/bugs/1774025>
<mup> Bug #1774026 opened: [2.5] Move the 'Resource pools' filter between 'owner' and 'architectures' <resource-pools> <MAAS:New> <https://launchpad.net/bugs/1774026>
<mup> Bug #1774017 changed: MAAS doesn't support multiple vlans with unknown IDs on the same fabric <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1774017>
<mup> Bug #1774017 opened: MAAS doesn't support multiple vlans with unknown IDs on the same fabric <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1774017>
<mup> Bug #1774017 changed: MAAS doesn't support multiple vlans with unknown IDs on the same fabric <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1774017>
<mup> Bug #1774017 opened: MAAS doesn't support multiple vlans with unknown IDs on the same fabric <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1774017>
<mup> Bug #1774017 changed: MAAS doesn't support multiple vlans with unknown IDs on the same fabric <foundations-engine> <MAAS:Opinion> <https://launchpad.net/bugs/1774017>
<roaksoax> jac_cplane you cannot select specific kernels with MAAS, but you could potentially modify the curtin preseed to select the kernel you'd like to install
<roaksoax> although, i have not tested that
<mup> Bug #1774058 opened: MAAA umount / mount point with bcache <cdo-qa> <cdo-qa-blocker> <cpe-onsite> <MAAS:New> <https://launchpad.net/bugs/1774058>
#maas 2018-05-30
<mup> Bug #1774135 opened: [2.5, snap] dhcpd wasn't started after enabling dhcp on a subnet <MAAS:New> <https://launchpad.net/bugs/1774135>
<mup> Bug #1774135 changed: [2.5, snap] dhcpd wasn't started after enabling dhcp on a subnet <MAAS:New> <https://launchpad.net/bugs/1774135>
<mup> Bug #1774135 opened: [2.5, snap] dhcpd wasn't started after enabling dhcp on a subnet <MAAS:New> <https://launchpad.net/bugs/1774135>
<bjarne> Hi, I saw that 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 was availeble in next repo, but  seems  like 2.4.0~rc2 isn't releaset yet. Which branch is that build based on?
<roaksoax> bjarne: ? 2.4.0~rc2 -> 2.4.0
<roaksoax> bjarne: its the same , its just renamed
<mup> Bug #1774206 opened: MAAS denies recursive DNS queries from subnets it doesn't know about <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774206>
<mup> Bug #1774186 opened: curtin does not recognize / partition created in MAAS <cdo-qa> <cdo-qa-blocker> <cpe-onsite> <foundations-engine> <curtin:Triaged> <MAAS:New> <https://launchpad.net/bugs/1774186>
<xygnal> is 2.4.0 final out?
<xygnal> Ability to increase the number of regiond workers
<xygnal> Following the improved way MAAS daemons are run, further internal changes have been made to allow the number of regiond workers to be increased automatically. This allows MAAS to scale to handle an increased number of internal operations in larger environments.
<xygnal> does this affect more than UI?
<roaksoax> xygnal: yes
<roaksoax> xygnal: /win 3
<roaksoax> err
<roaksoax> xygnal: it allows for more workers communicate with the DB, but would require to increase the amximum number of db connections postgresql accepts
<roaksoax> xygnal: but at the same, your issues with the UI should be considerably improved in 2.4
<roaksoax> xygnal: without increasing the number of workers
<xygnal> yes. impressive release notes sir.
<xygnal> looking forward to a lot
<mup> Bug #1774266 opened: Validate boot images are available for applicable rack controllers only <MAAS:Triaged> <https://launchpad.net/bugs/1774266>
<bdx> concerning 3rd party drivers
<bdx> here is my commissioning script https://paste.ubuntu.com/p/V7RZBRRqtq/
<bdx> I've tried a few different paths, but can't get maas to pick up the mellanox interfaces in commissioning
<bdx> even though the driver install succeeds, and I can ssh into the node following commissioning and see the physical interfaces are there
<bdx> no matter what I do, I can't seem to get maas to pick up the mellanox interface in commissioning
<bdx> any ideas?
<bdx> bug?
<roaksoax> bjarne: whats the name of your script ?
<roaksoax> err
<roaksoax> bdx: ^^ what's the name of your script ?
<bdx> 00-install-xenial-mlnx-drivers
<bdx> the name in the metadata
<roaksoax> bdx: maas should be grabbing the interface information from 'ip addr'
<roaksoax> bdx: so that could mean there's no 'ip addr' info for your interface ?
<bdx> roaksoax: the node isn't powered on anymore, but a few moments ago this is what `ip a` showed https://paste.ubuntu.com/p/6CjG9MKWPq/
<bdx> enp130s0 and enp130s0d1
<bdx> ^ the mellanox
<roaksoax> bdx: check the output in  99-maas-03-network-interfaces
<bdx> roaksoax: it shows the mellanox nic
<bdx> https://imgur.com/a/c3XoTcY
<roaksoax> bdx: and it is not shown on the interfaces tab ?
<bdx> correct
<roaksoax> bdx: and when you commissioning, are you selecting 'Retain network configuration' ?
<bdx> I have been, let me run another commissioning with that checked just for an isolated test
<bdx> jesus
<bdx> they just showed up in the interfaces tab
<bdx> ahh
<bdx> I think I know what changed
<bdx> previously, I wasn't adding the metadata tag to recommission
<roaksoax> ah gotcha
<bdx> as well as checking "retain network"
<bdx> roaksoax: thanks
<bdx> roaksoax: now I have the drivers being installed during commissioning, and the interfaces recognized
<bdx> is my next step to get the drivers into the image at deploy time to build a deb and add them to the drivers.yaml?
<roaksoax> bdx: i would probably recommend you install them the same way you are doing in the ephemeral enviromment
<roaksoax> bdx: via curtin preseeding
<roaksoax> bdx: i have  quesiton though, are these drivers not available in Ubuntu at all ?
<bdx> roaksoax: correct
<roaksoax> so i guess mellanox has not upstreamed these drivers into the linux kernel?
<bdx> also correct
<roaksoax> ok gotcha
<bdx> http://www.mellanox.com/related-docs/prod_software/Mellanox_EN_for_Linux_User_Manual_v4_3.pdf - chapter 2.9 "Installing using apt-get"
<bdx> they suggest building a deb with the included packaging and serving them up from a local file repo
<bdx> I have been trying to build the deb and upload it to launchpad under my own ppa
<bdx> ^ then I could just create an entry in drivers yaml
<bdx> the preseed may be easier for sure
<bdx> because the packaging doesn't build successfully on launchpad
<roaksoax> right
<bdx> I've been trying to fix/hack it
<roaksoax> yeah the preseed may be straightforward since you already have such script
<roaksoax> you could just run that script in a late_command
<roaksoax> inside the installed target
<bdx> totally
<bdx> I've done this before back in 1.7 or something ..... are the preseeds under SNAP_COMMON now I take it?
<bdx> I can setup a new apt installed maas
<bdx> I want to deploy all my new things under maas 2.4 anyway
<roaksoax> probably not, since we are still working on getting the snap done for 2.4, we haven't made changes wrt to the snap in 2.3
<bdx> ok
<bdx> I may want to use maas from apt is what you are saying
<bdx> got it
<roaksoax> indeed
<bdx> ok
<roaksoax> bdx: you could then migrate to a 2.4 snap, once ready, pointing it to your current DB
<bdx> roaksoax: totally
<bdx> well, I have the two entirely decoupled
<bdx> so that will probably be even easier
<bdx> roaksoax: how do I migrate the db from 2.3.x -> 2.4 ?
<mup> Bug #1771885 opened: bionic: static maas missing search domain in systemd-resolve configuration <bionic> <network> <cloud-init:New> <juju:Fix Committed by ecjones> <juju 2.3:Fix Released by ecjones> <MAAS:New> <https://launchpad.net/bugs/1771885>
<roaksoax> bdx: i would suggest you backup your db
<roaksoax> bdx: once that's done, isntall maas from packages
<roaksoax> sudo service maas-regiond stop
<roaksoax> bdx: then configure /etc/maas/regiond.conf to point to where the 2.3 db is
<roaksoax> bdx: and then run: sudo maas-region dbupgrade
<roaksoax> bdx: then: sudo service maas-regiond restart
<roaksoax> bdx: you /may/ need to rm -rf /var/lib/maas/secret and let maas re-create a new one
<bdx> ok, awesome
<bdx> roaksoax: my maas is currently deployed to a xenial box, and my postgresql is on another xenial box
<bdx> roaksoax: is 2.4 compatible with xenial, or should I deploy a bionic host to install maas from apt on?
<roaksoax> bdx: you'd need bionic
<bdx> thats what I thought
<roaksoax> bdx: i'll check what the status of the snap is tomorrow though, since we were dependent on core18 changes
<roaksoax> s/core18/base18
<bdx> great, thanks for all of this
<roaksoax> np!
#maas 2018-05-31
<KingJ> Trying to create a bond through the web UI, but the bond mode dropdown is blank. Has anyone seen this before? Screenshot: https://imgur.com/a/0zPXc4S
<mup> Bug #1774422 opened: [2.5,2.4] Table to create a bridge is not formatted correctly <MAAS:New for deadlight> <MAAS 2.4:New for deadlight> <https://launchpad.net/bugs/1774422>
<mup> Bug #1774424 opened: [2.5, 2.4] Table to create a bond is not formatted correctly  <MAAS:Triaged by deadlight> <MAAS 2.4:New for deadlight> <https://launchpad.net/bugs/1774424>
<KingJ> Heh, either I have exceptionally good timing for asking about this or it was checked and confirmed.
<KingJ> I've no idea how, but I have once managed to trigger the UI to display it properly, but it was down to chance.
<roaksoax> KingJ: let me test
<KingJ> (apologies, context was unclear - the bug that was just opened is my exact issue - wasn't trying to imply that anyone here was sugglish in replying!)
<roaksoax> KingJ: ah gotcha, yeah I can see the bond modes
<KingJ> Hmm. I'm not sure why, but on a different machine I can see the modes - although the table headers are truncated as per https://launchpadlibrarian.net/372591728/2.5-create-bond-bug.png
<mup> Bug #1752687 changed: Quanta D52B-1U unable to PXE-boot in EFI mode <hwcert-server> <MAAS:Incomplete> <grub2-signed (Ubuntu):New> <https://launchpad.net/bugs/1752687>
<mup> Bug #1752687 opened: Quanta D52B-1U unable to PXE-boot in EFI mode <hwcert-server> <MAAS:Incomplete> <grub2-signed (Ubuntu):New> <https://launchpad.net/bugs/1752687>
<mup> Bug #1752687 changed: Quanta D52B-1U unable to PXE-boot in EFI mode <hwcert-server> <MAAS:Incomplete> <grub2-signed (Ubuntu):New> <https://launchpad.net/bugs/1752687>
<Supo> I have been trying to deploy my first machines with Maas with no success. I'm able to commission and test the hardware but when it comes to deploy, it acquires it but releases it in 20 seconds.
<Supo> Has anyone seen this before? am i missing something?
<roaksoax> Supo: have never seen that. SO you say when you click 'Deploy' it goes back to 'Ready' ?
<Supo> gown to acquired first and 20 seconds later goes to ready
<roaksoax> Supo: tail -f /var/log/maas/*.log and try again and see if something comes up
<roaksoax> Supo: i've never seen that happening before
<roaksoax> Supo: also, what version are you using ?
<Supo> repeats this line 2018-05-31 16:20:31 maasserver.preseed: [warn] WARNING: '/etc/maas/preseeds/curtin_userdata' contains deprecated preseed variables. Please remove: main_archive_directory, ports_archive_directory, http_proxy
<roaksoax> Supo: yeah not a reason to fail actually, so you are good
<Supo> 2.4 beta2
<Supo> and this one 2018-05-31 16:20:53 maasserver.websockets.protocol: [critical] Error on request (385) machine.action: This transaction has already been attempted multiple times; giving up.
<roaksoax> Supo: try upgrading to 2.4.0 (final) first
<roaksoax> Supo: uhmmm yeah please upgrade to 2.4.0 (final) which was released yesterday
<roaksoax> Supo: and lets attempt
<Supo> ok will do
<Supo> beta2 still shows as the latest
<mup> Bug #1774058 changed: [2.4] Creating bcache partitions fails in some situations <cdo-qa> <cdo-qa-blocker> <cpe-onsite> <MAAS:Invalid> <MAAS 2.3:In Progress by blake-rouse> <MAAS 2.4:In Progress by blake-rouse> <https://launchpad.net/bugs/1774058>
<xygnal> roaksoax: what version of cloud init needed for static ip on centos?
<mup> Bug #1774058 opened: [2.4] Creating bcache partitions fails in some situations <cdo-qa> <cdo-qa-blocker> <cpe-onsite> <MAAS:Invalid> <MAAS 2.3:In Progress by blake-rouse> <MAAS 2.4:In Progress by blake-rouse> <https://launchpad.net/bugs/1774058>
<mup> Bug #1774058 changed: [2.4] Creating bcache partitions fails in some situations <cdo-qa> <cdo-qa-blocker> <cpe-onsite> <MAAS:Invalid> <MAAS 2.3:In Progress by blake-rouse> <MAAS 2.4:In Progress by blake-rouse> <https://launchpad.net/bugs/1774058>
<supo> after upgrading to 2.4 still getting the following error when i try to juju bootstrap: Conflict error. Try your request again, as it will most likely succeed.
<roaksoax> supo: ah, you are deploying with Juju, not directly with MAAS ?
<bdx> roaksoax: is there an api call I can make to change the primary rack controller?
<bdx> I think I almost have a successful upgrade
<bdx> but I'm hitting this https://www.dropbox.com/s/hp6jfrr5iapwhvk/Screen%20Shot%202018-05-31%20at%2012.55.44%20PM.png?dl=0
<bdx> I stopped all the services on my 2.3 region/rack and for some reason I con no longer ssh into it, I may have lost the node for the time being
<bdx> but my database migration worked
<bdx> and 2.4 seems to be happy
<bdx> other then the fact that it cant assume primary
<roaksoax> maas $PROFILE vlan update $FABRIC_ID $VLAN_TAG dhcp_on=True \
<roaksoax>     primary_rack=$PRIMARY_RACK_CONTROLLER
<bdx> awesome, thanks
<bdx> roaksoax: when I run that command
<bdx> $ maas admin vlan update fabric-1 200 dhcp_on=True primary_rack=rxyyr6
<bdx> {"primary_rack": ["Select a valid choice. rxyyr6 is not one of the available choices."]}
<bdx> should I need to add it as a secondary ?
<roaksoax> bdx: not really
<roaksoax> bdx: try secondary_rack=''
<roaksoax> and see what happens
<roaksoax> ?
<bdx> https://paste.ubuntu.com/p/VK2x89XbBj/
<bdx> it doesn't look like the empty string had any affect
<roaksoax> bdx: i meant: primary_rack=rxyyr6 secondary_rack=''
<bdx> ok
<mup> Bug #1774495 opened: [2.5] Clicking on Zero of 15 ready for the default pools filters to undefined <resource-pools> <MAAS:Triaged> <https://launchpad.net/bugs/1774495>
<bdx> $ maas admin vlan update 1 200 dhcp_on=True primary_rack=rxyyr6 secondary_rack=''
<bdx> {"primary_rack": ["Select a valid choice. rxyyr6 is not one of the available choices."]}
<bdx> $ maas admin rack-controllers read | pastebinit
<bdx> http://paste.ubuntu.com/p/y9H6jN2hnV/
<bdx> shows it to indeed be a rack controller
<roaksoax> bdx: is the rack controller on the vlan ?
<bdx> yes
<roaksoax> bdx: look at the interface set, the 'vlan' is null
<bdx> ubuntu@maas-region-rack-01:~$ ping -I vlan.200 192.168.200.1
<bdx> PING 192.168.200.1 (192.168.200.1) from 192.168.200.11 vlan.200: 56(84) bytes of data.
<bdx> 64 bytes from 192.168.200.1: icmp_seq=1 ttl=64 time=0.207 ms
<bdx> 64 bytes from 192.168.200.1: icmp_seq=2 ttl=64 time=0.070 ms
<bdx> ooh
<bdx> roaksoax: how can I change that ?
<roaksoax> bdx: check the interfaces of the rack controller and make sure they are correct, if not correct that
<mup> Bug #1774186 changed: curtin does not recognize / partition created in MAAS <cdo-qa> <cdo-qa-blocker> <cpe-onsite> <foundations-engine> <curtin:Triaged> <https://launchpad.net/bugs/1774186>
<bdx> roaksoax: it looks like I added the vlan200 interface after I had installed maas
<bdx> when I removed it, and reinstalled with the vlan200 interface pre-existing, I was able to get past the error
<bdx> but now another
<bdx> roaksoax: https://imgur.com/a/yitXnRk
<bdx> any idea on what I might do here?
<bdx> can I force the removal somehow?
<bdx> it errors similarly from the cli
<bdx> $ maas admin region-controller delete hhwcfn
<bdx> {"pool": ["Can't assign to a resource pool."]}
<mup> Bug #1774529 opened: Cannot delete some instances of model 'Domain' because they are referenced through a protected foreign key <MAAS:New> <https://launchpad.net/bugs/1774529>
<roaksoax> bdx: you try to remove a region-controller and it tells you you can't assign to a resource pool ?
<bdx> thats correct
<bdx> at least my env is stable now to some degree
<bdx> I need to rid that stale controller, but all the things seem to be working on the new 2.4 controller
<bdx> minus the sync status
<bdx> I'm thinking that will clear when I rid the old controller from the db
<bdx> roaksoax: would you like me to compile a tar of logs for you for this?
<bdx> https://paste.ubuntu.com/p/DBMXsdpYrr/
<bdx> here's the full regiond.log http://paste.ubuntu.com/p/wNCy8c4n2c/
<roaksoax> bdx: ok, so let me udnerstand this right. the controller you are trying to delete no longer exists
<roaksoax> bdx: the machine is gone one
<roaksoax> gone gone
<roaksoax> bdx: this is a region controller
<roaksoax> bdx: but when you try to delete it (i assume both api/ui) it throws you that error ?
<bdx> roaksoax: exactly
<bdx> its a region+rack
<roaksoax> bdx: could ytou please file a bug and attach that log e.g. "Trying to delete a region/rack that no longer exists, results in {'pool': ["Can't assign to a resource pool."]}'
<roaksoax> bdx: i wont be able to attempt to reproduce today
<roaksoax> but will look into it tomorrow
<roaksoax> but someone on my team will probably pick it up
<bdx> sure thing
<bdx> np
<bdx> great, thanks again
<mup> Bug #1774538 opened: Trying to delete a region/rack that no longer exists, results in {'pool': ["Can't assign to a resource pool."]} <MAAS:New> <https://launchpad.net/bugs/1774538>
#maas 2018-06-01
<KingJ> I'm not sure if this is an issue with MAAS or netplan, but i'm having a lot of difficult getting a 9000 byte MTU on a 802.1ad bond. In the generated netplan config, I can see that the MTU is set to 9000 for the bond member interfaces, but when the bond comes up everything switches to 1500.
<KingJ> Indeed, checking dmesg I can see a lot of "changing MTU from 9000 to 1500" messages.
<roaksoax> KingJ: is that for a deployed system ?
<KingJ> roaksoax: Yeah
<roaksoax> KingJ: what's the resulting netplan ?
<KingJ> One sec
<KingJ> roaksoax: Netplan config: https://paste.ubuntu.com/p/mxp6Dkxfqg/
<xygnal> roaksoax: why is pgs10 needed for 2.4?
<roaksoax> KingJ: seems you have eth0 with mtu 1500 and the rest with mtu 9000, which is your expected config, right ?
<roaksoax> KingJ: but those w/ mtu9000 are really being changed to mtu 1500 ?
<KingJ> roaksoax: Yep, eth0 is supposed to be at 1500 (can't support higher, and is just a management interface). enp* are meant to be at 9000 and bonded.
<KingJ> Let me grab the dmesg and ip link sh output
<roaksoax> KingJ: ok, so probably due to the bond not declaring MTU ?
<KingJ> dmesg: https://paste.ubuntu.com/p/PQrJcKbXtv/
<KingJ> ip link sh: https://paste.ubuntu.com/p/GRP3SGWrSt/
<KingJ> roaksoax: Possibly? There was no option on the UI to set the MTU on the bond.
<roaksoax> xygnal: i dont /think/ we have a strict requirement on pg10, but rather, pg10 is the default on bionic
<xygnal> yes figured.  we run our db separate so verifying we can keep it.
<roaksoax> KingJ: ok, so IIRC, if the parent interfaces of a bond have the higher mtu, then the bond should inherit that
<KingJ> I did also try assigning a VLAN with a 9000 MTU to the bond interface itself, although that doesn't appear to have made it in to the netplan config (possibly because I only assigned the VLAN and left subnets/addressing unassigned?)
<roaksoax> KingJ: and it seems in this case is not
<roaksoax> KingJ: try this: maas <user> machine get-curtin-config <system-id>
<roaksoax> KingJ: and deploy a machine in xenial and see whether the bond ends up with the mtu defined ?
<KingJ> roaksoax: Here's the curtin config (just the networking section) https://paste.ubuntu.com/p/5Tdb6xwh69/
<KingJ> I'll reclaim the machine now and re-provision on xenial
<roaksoax> KingJ: https://pastebin.ubuntu.com/p/jdC68hPRJR/
<roaksoax> KingJ: seems to be that the mtu is defined
<roaksoax> KingJ: so that looks like a cloud-init issue to me
<KingJ> Yeah, odd. I'll admit I also have tried running interface update the6f3 153 mtu=9000 however that was done after deployment - so I wouldn't expected that to have an effect?
<roaksoax> KingJ: no, so lets wait to see what e/n/i results in xenial
<roaksoax> KingJ: to compare
<KingJ> roaksoax: Gotcha. I'll report back once deployed on Xenial. Should be within the next half hour.
<KingJ> (Thanks as always for your help here!)
<roaksoax> mp!
<xygnal> roaksoax: what cloudinit version is officially needed for static ip support?
<xygnal> roaksoax: and look at this... https://bugs.launchpad.net/cloud-init/+bug/1712680
<roaksoax> xygnal: static ip addressing is in 17.x something
<roaksoax> xygnal: the bug is fixed in cloud-init 18.2+ i think, which the centos iamges are not yet using
<xygnal> they are using an older version with a patch
<xygnal> and the patch appears to contain it
<xygnal> as it works with your centos7 image
<xygnal> however, seems that bug report implies its broken in other ways
<xygnal> ty
<KingJ> roaksoax: Deployed on 16.04, still seeing a 1500 MTU. dmesg still has messages about the MTU being changed from 9000 to 1500. Checking /etc/network/interfaces.d/50-cloud-init.cfg I can see that all interfaces, including the bond, have a 9000 MTU configured (paste incoming)
<KingJ> https://paste.ubuntu.com/p/mr2JCwTzfX/
<KingJ> Ah wait
<KingJ> My bad... it's actually all at 9000 and dmesg was about changing from 1500 to 9000
<roaksoax> xygnal: it shouldn't really, if you use maas provided config
<roaksoax> KingJ: ah so both bionic/xenial doing the same thing ?
<KingJ> roaksoax: No sorry. Bionic has dmesg messages about MTU changing from 9000 to 1500, xenial has messages about 1500 to 9000. Xenial's ip link sh output confirms 9000 MTU, pings with large packets work without fragmenting. So Xenial works, Bionic doesn't.
<xygnal> roaks: i dont change its config. i just deploy in maas with autoassign and post deploy iatrou_nspection shows a local static config file.
<roaksoax> KingJ: please file a bug and attach all relevant info (maas preseed config, the cloud-init netplan config on bionic, the e/n/i on xenial, and the kernel messages)
<KingJ> e/n/i ?
<roaksoax> KingJ: /etc/network/interfaces.d/50-**
<KingJ> Ahhhh, yes.
<KingJ> I'll get that filed now, thanks.
<roaksoax> xygnal: sorry, i dont follow ?
<xygnal> there was a typo in there sorry.  If I deploy the stock centos7 image from MAAS in 2.4, it writes a static IP config locally on the CentOS box
<xygnal> using auto-assign
<xygnal> you seemed to think it should not, but i'm telling you, it does
<xygnal> just FYI
<roaksoax> xygnal: ah no, i think there's some confusion there then, because it should deploy correctly a static config file
<roaksoax> xygnal: my bug was about modifying the config manually post deployment, cloud-init would re-write it on reboot
<xygnal> roaksoax: ah you were addressing the bug specifically. gotcha.
<xygnal> i only pointed that out as i noticed that is the version that you deploy
<roaksoax> gotcha
<xygnal> (or CentOS deploys, as it were)
<mup> Bug #1774665 opened: MAAS creates GPT header and partition on one disk when creating a 2 disk raid-1 set <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774665>
<mup> Bug #1774666 opened: Bond interfaces stuck at 1500 MTU on Bionic <mtu> <netplan> <MAAS:New> <https://launchpad.net/bugs/1774666>
<roaksoax> KingJ: one more thing, can you provide the version of cloud-init ? dpkg -l | grep cloud-init
<chquark> Hi guys,
<chquark> iá¸¿ trying to follow this guide https://docs.openstack.org/charm-deployment-guide/latest/install-maas.html
<KingJ> roaksoax: Done :) Let me know if there's anything else I need to add
<chquark> using ubuntu 18 LTS, but have this error https://imgur.com/a/1lSwt4z
<roaksoax> KingJ: i think that's all, i opened bugs against cloud-init as i think that's wehre the error is
<roaksoax> chquark: go to the 'Logs' tab, what does it show ?
<roaksoax> chquark: for the installation log
<KingJ> roaksoax: Yeah, that does look to be the case. Thanks for your help here.
<roaksoax> np!
<chquark> Installation has failed and no output was given.
<chquark> ^^ that is all it says
<chquark> https://pastebin.com/yXiVDWid
<chquark> https://pastebin.com/cHZM0Evz Â´/var/log/maas/rsyslog/juju1/2018-06-01Â´
<roaksoax> chquark: in the UI you have a 'logs' tab
<roaksoax> chquark: with a red
<roaksoax> icon
<roaksoax> it will show you the logs
<roaksoax> chquark: for installation
<roaksoax> chquark: does it show ?
<roaksoax> chquark: looking at the rsyslog, this is the error: https://pastebin.ubuntu.com/p/xDMxpkYSV5/
<chquark> so it basically needs access to the internet
<roaksoax> chquark: well yes and no, 10.0.2.15:8000 -> who owns that IP ?
<roaksoax> chquark: is that MAAS itself ?
<chquark> its my NAT ip.. going to internet.
<chquark> MAAS is 192.168.100.1
<roaksoax> chquark: so the deployment needs to access the ubuntu archive, if you have a local archive you can use that
<roaksoax> chquark: so the deployment needs to access the ubuntu archive, if you have a local archive you can use that instead
<roaksoax> chquark: but that error is telling me that maas is trying to contact the internal proxy on that IP address
<roaksoax> chquark: that would mean maas itself is running on a machine that has that IP ?
<chquark> yes MAAS server has that IP, it is NATed to get internet access
<roaksoax> chquark: and are machines on 192.168.100.0/24 ?
<chquark> MAAS has 3 interfaces , 1) 192.168.100.1 , 2) 10.0.2.15, 3) 192.168.56.204
<roaksoax> chquark: ok what's in /etc/maas/rackd.conf ? and whats the network that machines are PXE booting from (i'm guessing its 192.168.100.1? )
<chquark> Juju 1) 192.168.100.0/24 -- DHCP from MAAS
<roaksoax> chquark: ok, so check rackd.conf
<chquark> [sudo] password for user:
<chquark> cluster_uuid: ec08ffa0-1d19-4591-bd3d-a31381de5ba7
<chquark> maas_url: http://localhost:5240/MAAS
<chquark> user@user:/var/log/maas/rsyslog/juju1/2018-06-01$
<roaksoax> chquark: do this, use 192.168.100.1 instead of 'localhost'
<roaksoax> chquark: in rackd.conf
<roaksoax> sudo service maas-rackd restart
<roaksoax> chquark: and try to deploy again
<chquark> ok
<chquark> roaksoax: I think it is working :)
<roaksoax> chquark: awesome!
<mup> Bug #1774665 changed: MAAS creates GPT header and partition on one disk when creating a 2 disk raid-1 set <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774665>
<mup> Bug #1774665 opened: MAAS creates GPT header and partition on one disk when creating a 2 disk raid-1 set <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774665>
<roaksoax> KingJ: fwiw, what is being asked is for you to set the MTU on the bond interface itself and see if that correctly sets the MTU
<KingJ> roaksoax: Gotcha. Just tried adding it to the bond section and re-running netplan apply but to no avail - also tried a systemd-networkd restart. Just going for a full reboot now before reporting back on the bug ticket.
#maas 2018-06-02
<supo_> Hi everyone, have a question about deploying on a brand new Mass 2.4 install.
<supo_> I can not seem to get it to work
<supo_> here is what i see in the logs
<supo_> 2018-06-02 00:30:43 maasserver.websockets.protocol: [critical] Error on request (22) machine.action: This transaction has already been attempted multiple times; giving up.         Traceback (most recent call last):           File "/usr/lib/python3.6/threading.py", line 864, in run             self._target(*self._args, **self._kwargs)           File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 850, in worke
<supo_> i ran into this error on the 2.4beta2 also
#maas 2018-06-03
<mup> Bug #1760703 changed: 2.4b1: unknown storage in controller page <MAAS:Expired> <https://launchpad.net/bugs/1760703>
#maas 2020-05-25
<mup> Bug #1880495 opened: Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist <MAAS:New> <https://launchpad.net/bugs/1880495>
<mup> Bug #1880208 changed: MAAS 2.7.1 - "No Such Resource" after fresh install  <MAAS:Invalid> <MAAS 2.7:Fix Released by adam-collard> <https://launchpad.net/bugs/1880208>
#maas 2020-05-26
<mup> Bug #1865859 changed: HTTP Error 415 with iDRAC 9 using Redfish power type <power> <MAAS:Expired> <https://launchpad.net/bugs/1865859>
<mup> Bug #1880495 opened: Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Xenial):New> <https://launchpad.net/bugs/1880495>
<mup> Bug #1880495 changed: Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Xenial):New> <https://launchpad.net/bugs/1880495>
<mup> Bug #1880495 opened: Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Xenial):New> <https://launchpad.net/bugs/1880495>
<mup> Bug #1880697 opened: maas cannot provide all official maas images <hwcert-server> <MAAS:New> <https://launchpad.net/bugs/1880697>
<mup> Bug #1880697 changed: maas cannot provide all official maas images <hwcert-server> <MAAS:New> <https://launchpad.net/bugs/1880697>
<mup> Bug #1880697 opened: maas cannot provide all official maas images <hwcert-server> <ui> <MAAS:New> <https://launchpad.net/bugs/1880697>
<mup> Bug #1880697 changed: maas cannot provide all official maas images <hwcert-server> <ui> <MAAS:New> <https://launchpad.net/bugs/1880697>
<mup> Bug #1880697 opened: maas cannot provide all official maas images <hwcert-server> <ui> <MAAS:New> <https://launchpad.net/bugs/1880697>
<mup> Bug #1880697 changed: maas cannot provide all official maas images <hwcert-server> <ui> <MAAS:New> <https://launchpad.net/bugs/1880697>
<mup> Bug #1880746 opened: snap post-refresh crashes on dbupgrade <MAAS:New> <https://launchpad.net/bugs/1880746>
<mup> Bug #1880746 changed: snap post-refresh crashes on dbupgrade <MAAS:New> <https://launchpad.net/bugs/1880746>
<mup> Bug #1880746 opened: snap post-refresh crashes on dbupgrade <MAAS:New> <https://launchpad.net/bugs/1880746>
<mup> Bug #1880755 opened: [snap] snap upgrade requires 2x disk space <MAAS:New> <https://launchpad.net/bugs/1880755>
<mup> Bug #1880755 changed: [snap] snap upgrade requires 2x disk space <MAAS:New> <https://launchpad.net/bugs/1880755>
<mup> Bug #1880755 opened: [snap] snap upgrade requires 2x disk space <MAAS:New> <https://launchpad.net/bugs/1880755>
<mup> Bug #1880765 opened: [ui, feature] PXE field should not be interactive <MAAS:New> <https://launchpad.net/bugs/1880765>
<mup> Bug #1880765 changed: [ui, feature] PXE field should not be interactive <MAAS:New> <https://launchpad.net/bugs/1880765>
<mup> Bug #1880765 opened: [ui, feature] PXE field should not be interactive <MAAS:New> <https://launchpad.net/bugs/1880765>
#maas 2020-05-27
<mup> Bug #1869264 changed: Upgrade from MAAS 2.6.0 to 2.6.2 in HA environment results in maas DB user's password reset, no MAAS nodes able to connect to DB <MAAS:Expired> <https://launchpad.net/bugs/1869264>
<mup_> Bug #1880966 opened: machines read (API) call is unreasonably slow <performance> <MAAS:New> <https://launchpad.net/bugs/1880966>
<mup> Bug #1880966 opened: machines read (API) call is unreasonably slow <performance> <MAAS:New> <https://launchpad.net/bugs/1880966>
<mup_> Bug #1880966 changed: machines read (API) call is unreasonably slow <performance> <MAAS:New> <https://launchpad.net/bugs/1880966>
<mup> Bug #1880966 changed: machines read (API) call is unreasonably slow <performance> <MAAS:New> <https://launchpad.net/bugs/1880966>
<mup_> Bug #1880966 opened: machines read (API) call is unreasonably slow <performance> <MAAS:New> <https://launchpad.net/bugs/1880966>
<mup> Bug #1880966 opened: machines read (API) call is unreasonably slow <performance> <MAAS:New> <https://launchpad.net/bugs/1880966>
#maas 2020-05-28
<mup> Bug #1747548 changed: [2.3] 500 error from maas metadata while running hardware tests <MAAS:Expired> <https://launchpad.net/bugs/1747548>
<mup> Bug #1881116 opened: AttributeError when commissioning ppc64le machine <MAAS:Triaged> <https://launchpad.net/bugs/1881116>
<mup> Bug #1881117 opened: Failed to gather resource information from arm64 node <MAAS:Triaged> <https://launchpad.net/bugs/1881117>
<mup> Bug #1881116 changed: AttributeError when commissioning ppc64le machine <MAAS:Triaged> <https://launchpad.net/bugs/1881116>
<mup> Bug #1881117 changed: Failed to gather resource information from arm64 node <MAAS:Triaged> <https://launchpad.net/bugs/1881117>
<mup> Bug #1881116 opened: AttributeError when commissioning ppc64le machine <MAAS:Triaged> <https://launchpad.net/bugs/1881116>
<mup> Bug #1881117 opened: Failed to gather resource information from arm64 node <MAAS:Triaged> <https://launchpad.net/bugs/1881117>
<mup> Bug #1881133 opened: DNS Servers not set as expected <MAAS:New> <https://launchpad.net/bugs/1881133>
<mup> Bug #1881136 opened: MAAS API does not allow changing an fstype or mount_point on LVM volume-groups <MAAS:New> <https://launchpad.net/bugs/1881136>
<mup> Bug #1881143 opened: The Take action menu has deteriorated with the last update. <MAAS:New> <https://launchpad.net/bugs/1881143>
<mup> Bug #1881201 opened: singlespa has not been updated for Debian packages <ui> <MAAS:New> <https://launchpad.net/bugs/1881201>
<mup> Bug #1881203 opened: SnapStore and Snap Blocks Install more Apps  <MAAS:New> <https://launchpad.net/bugs/1881203>
<mup> Bug #1873502 changed: maas-cli deb installs full maas snap? <maas (Ubuntu):Invalid> <https://launchpad.net/bugs/1873502>
#maas 2020-05-29
<mup> Bug #1881247 opened: Machine listing link from resource pools page doesn't work <ui> <MAAS:New> <https://launchpad.net/bugs/1881247>
<mup> Bug #1881247 changed: Machine listing link from resource pools page doesn't work <ui> <MAAS:New> <https://launchpad.net/bugs/1881247>
<mup> Bug #1881247 opened: Machine listing link from resource pools page doesn't work <ui> <MAAS:New> <https://launchpad.net/bugs/1881247>
<mup> Bug #1881247 changed: Machine listing link from resource pools page doesn't work <ui> <MAAS:New> <https://launchpad.net/bugs/1881247>
<mup> Bug #1881247 opened: Machine listing link from resource pools page doesn't work <ui> <MAAS:New> <https://launchpad.net/bugs/1881247>
<mup> Bug #1881262 opened: Tooltip in header for AZs doesn't show <ui> <MAAS:New> <https://launchpad.net/bugs/1881262>
<mup> Bug #1881262 changed: Tooltip in header for AZs doesn't show <ui> <MAAS:New> <https://launchpad.net/bugs/1881262>
<mup> Bug #1881262 opened: Tooltip in header for AZs doesn't show <ui> <MAAS:New> <https://launchpad.net/bugs/1881262>
<mup> Bug #1881275 opened: Machine listing got messed up <ui> <MAAS:New> <https://launchpad.net/bugs/1881275>
<mup> Bug #1881275 changed: Machine listing got messed up <ui> <MAAS:New> <https://launchpad.net/bugs/1881275>
<mup> Bug #1881275 opened: Machine listing got messed up <ui> <MAAS:New> <https://launchpad.net/bugs/1881275>
<mup> Bug #1881275 changed: Machine listing got messed up <ui> <MAAS:New> <https://launchpad.net/bugs/1881275>
<mup> Bug #1881275 opened: Machine listing got messed up <ui> <MAAS:New> <https://launchpad.net/bugs/1881275>
<mup> Bug #1881360 opened: Interfaces that have failed tests have confusing status <ui> <MAAS:New> <https://launchpad.net/bugs/1881360>
<mup> Bug #1881361 opened: Websocket doesn't show interface test failure when one iface passes and another fails <MAAS:In Progress by ltrager> <MAAS 2.7:In Progress by ltrager> <https://launchpad.net/bugs/1881361>
#maas 2020-05-31
<mup> Bug #1774529 changed: [2.4] Cannot delete some instances of model 'Domain' because they are referenced through a protected foreign key <MAAS:Expired> <https://launchpad.net/bugs/1774529>
<mup> Bug #1831891 changed: GUI - cannot set bond_xmit_hash_policy different from "layer2" <MAAS:Expired> <https://launchpad.net/bugs/1831891>
<mup> Bug #1869800 changed: Hardware test result links transposed for nvme drives <MAAS:Expired> <https://launchpad.net/bugs/1869800>
