#maas 2012-11-05
<jam> hi mgz
<mgz> hey jam!
<dimitern> jam, mgz: hey guys :)
<jam>  /wave
<mgz> bug 1068086 is what I'd like to know what actually needs fixing
<ubot5> Launchpad bug 1068086 in juju "juju fails to deploy with MAAS by default" [Undecided,Confirmed] https://launchpad.net/bugs/1068086
<jtv> Hi jam, hi mgz :)
<jam> hi jtv
<jam> good to see someone else has made it safely home
<jtv> Most of Red & friends did, actually.
<jtv> Only 2 MIA.
 * jtv is looking on the bright side today
<jam> jtv: I saw that thumper only made it back to NZ about 4 hours ago, so i'm imagining that bigjools is similarly occupied.
<jtv> He may be there, but he's not here.  :)
<jtv> I believe I heard he arrived at 01:00 local, today.
<jam> rvba: hey raphael, we're poking around the kernel parameters stuff this week. We were wondering if there is an obvious DB table to put "global config"
<jam> (command line parameters that by default should apply to every node)
<rvba> jam: well, the Config table is certainly an option.
<jam> sounds reasonable
<rvba> Also, if you make this a Config item, it's easy to expose it on the Settings page if you need to.
<jtv> Hi matsubara -- did you hear that our jenkins instance has gone read-only again?
<jtv> Oh, looks like it was fixed a few hours back.
<jtv> No, it wasn't.  Grrr.  :(
<Lele_> Hi guys, anyone can lend me a hand on a maas setup ?
<Lele_> we already have everything installed
<Lele_> but when the server boots from pxe
<Lele_> and timeouts the 20 secs on the boot menu that has all the options (local, maas-i386, comissioning, etc)
<Lele_> doest boot anything to commision itself as DECLARED
<Lele_> in the dashboard
<Lele_> im missing anything ?
<matsubara> did you run maas-import-pxe-files?
<matsubara> what error do you get in the node trying to enlist?
<Lele_> yep matsubara
<Lele_> i have all the profiles
<Lele_> if i run cobbler profile list
<Lele_> and all the images are stored on the images directory
<matsubara> hmm maas is not using cobbler anymore
<matsubara> what version of maas are you running?
<Lele_> i installed the 12.04 version
<Lele_> from the main repo
<Lele_> should i run the 12.10 ?
<matsubara> ah, there's an updated version without cobbler. you probably want to use that one
<Lele_> can i deploy 12.04 from that version ?
<matsubara> Lele_, you can probably use the precise package in this PPA: https://code.launchpad.net/~maas-maintainers/+archive/dailybuilds
<Lele_> awesome matsubara
<Lele_> ill get you back if i continue with issues
<Lele_> thanks for the support
<bigjools> morning
#maas 2012-11-06
<jam> morning all
<jam> glad you made it back safe bigjools
<bigjools> morning jam
<bigjools> me too ;)
<jtv> Hi jam, hi bigjools
<mgz> did anyone catch smoser yesterday?
<mgz> he posted a why-what? comment in bug 1044503 which is a bit suprising as I thought we'd asked him about it last week
<ubot5> Launchpad bug 1044503 in maas (Ubuntu Raring) "kernel command line is not easily customizable" [High,Triaged] https://launchpad.net/bugs/1044503
<mgz> jam, dimitern: have put up a very simple migration branch, will commit some smarts around model and fallback now
<Daviey> mgz: i did, i did!
<mgz> Daviey: and the upshot was? :)
<mgz> I'm still not really sure how this is going to be used, beyond maybe some simple customisations for specific hardware (like the highbank console thing currently hardcoded in the arch method)
<Daviey> mgz: No upshot.. i just happend to catch him.
<Daviey> now i think about it, it's not that helpful.
<mgz> well, if you catch him again, don't let him go!
<Daviey> because he bit your finger so?
<mgz> hm, jenkins is unhappy, "Failed to create a temp file on /home/ubuntu/launchpad-jobs/workspace/maas-merger-quantal"
<jam> mgz: it has been unhappy fora  few days, it is either a perms issue, or a out-of-disk space, IMO
<jam> rvba: ^^ you have direct access to jenkins, right?
<jam> mgz: commented on https://code.launchpad.net/~gz/maas/1.2_add_db_kernel_params/+merge/133044
<mgz> thanks jam
<rvba> jam: yes, the started yesterday, the whole filesystem of the vm went read only.  matsubara is aware of the problem and worked on it yesterday; apparently he did not manage to fix the problem yet.  I'm waiting for him or Larry to come online to see what they think.
<rvba> s/yes, the/yes, that problem/
<matsubara> rvba, hey, I'm around now. I waiting on Larry as well since I couldn't get thta read-only problem yesterday
<rvba> matsubara: all right.  Please keep us posted when you'll have news, this problem is really blocking us now.
<matsubara> rvba, will do. sorry about that
<rvba> ta
<roaksoax> rvba: hey man! how's everything? how was your flight back
<rvba> roaksoax: Hey Andres, everything all right.  How are you?
<roaksoax> rvba: long :)
<roaksoax> rvba: so anyways, where are we standing towards an SRU
<rvba> Indeed.
<roaksoax> rvba: are there things ready for me to start preparing for SRU?
<rvba> roaksoax: you're talking about the precise SRU right?
<gwd-laptop> I've got a question: Does MaaS require IPMI to function, or can it work with just DHCP / PXE / &c?
<roaksoax> rvba: quantal
<roaksoax> rvba: i need to catch up with the security team on the precise SRU
<rvba> roaksoax: ok
<roaksoax> rvba: but for 12.10 we has half of the bugs in the list already fixed, so I think we need to start SRU'ing by chunks
<roaksoax> rvba: i'm also gonna be trying to reproduce the upgrade bug again and see if it shows up
<rvba> roaksoax: we've committed most of the fix which will go in the SRU already https://launchpad.net/maas/+milestone/12.10-stabilization
<rvba> fixes* even
<rvba> roaksoax: 2 bugs are assigned to you in that list.  Not sure what's the status for these twoâ¦?
<roaksoax> rvba: well 1 is pretty hard to fix as there's no real way to find out whether it is a VM or real HW. and the second one is mostly IPMI issue itself, but will handle it by waiting more time before optaining an IP address, and dealing with the 0.0..0.0 as a failure to obtain IPMI address
<mgz> huh. we haven't backported our other changes to 1.2 yet so basing on 1.2 was maybe not helpful.
 * mgz adjusts the test
<mgz> jam, dimitern: how goes?
<dimitern> mgz: I saw your change and I'm adapting my tests and code to match what you added
<dimitern> I'll be ready later today with my changes, but still need a ui review
<gwd-laptop> Ping? Anyone have an answer to my question?
<mgz> gwd-laptop: what ever it was, it's not in my scrollback
<gwd-laptop> mgz: Thanks -- the question was: "Does MaaS require IPMI to function, or can it work with just DHCP / PXE / &c?"
<roaksoax> gwd-laptop: no it doesn't require IPMI to function, but then you would need to manually turn on machines
<gwd-laptop> mgz: I think that's probably OK -- in this case they'll actually be VMs. :-)
 * mgz takes all the credit
<gwd-laptop> Oops, sorry, didn't look closely. :-)
<roaksoax> gwd-laptop: yeah, there's an option to use virsh as a power method
<gwd-laptop> I work for Citrix on the Xen open-source team.  I want to write a charm to install Xen / XCP on MaaS, so that they can be tested with the OpenStack testing that the Server team does.  But we don't use MaaS for our test infrastructure at Citrix, so I was thinking of trying to use a virtualized environment.
<mgz> that sounds like a good plan. not sure what docs we have currently on testing like that.
<gwd-laptop> roaksoax: Would doing it manually work as well?  I'm a lot more familiar with setting up XenServer than with setting up libvirt -- although, it's probably not bad to get a bit more practice with the latter...
<smoser> mgz, the screen shots didn't make a lot of sense to me.
<roaksoax> gwd-laptop: hold on, you want to deploy Xen based Vm's with MAAS?
<roaksoax> gwd-laptop: or you simply want ot deploy Xen/XCP on machines using a charm?
<gwd-laptop> roaksoax: What I actually want is a charm to install Ubuntu + Xen in a MaaS environment.
<gwd-laptop> roaksoax: But I don't actually have a real MaaS environment, so I'm thinking of trying to set up a virtualized environment, just to test the charm.
<gwd-laptop> roaksoax: If it's not really feasible, I can probably get a phsyical MaaS system set up here, but it's just more work.
<roaksoax> gwd-laptop: right, so yeah a virtualized environment should be enough. The power management in maas is simply to start/stop machines/vm's when juju deploying something
<gwd-laptop> roaksoax: So if I'm willing to babysit the installation and flip the virtual switch myself, I should be able to test the charm.  Great -- I'll give that a try.
<mgz> smoser: sorry, screenshots? customising the kernel params is pretty much just and admin level config thing to avoid some hardcoding of stuff
<smoser> screenshots on the merge proposal
<dimitern> smoser: that ones I added?
<smoser> and i was never clear on what "tags" meant. i just wante dto see some example.
<roaksoax> gwd-laptop: yeah, power management is not really required :)
<gwd-laptop> roaksoax: Cool -- thanks for your help!
<roaksoax> gwd-laptop: anytime!
<mgz> dimitern: you seem to have proposed a 1.2 based branch against lp:maas rather than lp:maas/1.2 maybe?
<dimitern> mgz: it's agains 1.2, the MP is there now just for ui review
<dimitern> so it's on trunk (by mistake)
<smoser> dimitern, yeah, i suppose.
<smoser> i jus wanted an example description of how tags will be rendered.
<mgz> smoser, okay, so the concrete stuff I can see this doing is changing stuff like the compose_arch_opts function in provisioning.kernel_opts into an admin level command like 'add-tag --kernel-params="console=ttAMA0" --definition="contains(/node/product, 'Highbank')"'
<dimitern> smoser: so the tags are things you can attach to nodes, and the kernel opts can be part of the tag info, an in addition there are the global params in settings
<mgz> so, rather than needing a provisioning server code change to alter the params for a specific set of hardware, you define something that matches against the lshw output, then the final stage boot uses the params attached to that tag
<roaksoax> rvba: alright, so we are SRU'ing quantal MAAS into precise
<mgz> roaksoax suggested this would be enough for what you guys want to do, hopefully that's the case?
<smoser> hm..
<roaksoax> well yes, I suggested based on the fact that this was decided to be resolved using tags
<smoser> mgz, so --definition= says 'attach tag named kernel-params with value "console=...." to all machines that are Highbank'
<rvba> roaksoax: yeah, but that is going to require some extra work because of the dependencies (Django 1.4, etc.)
<roaksoax> rvba: yeah, i guess we could simply SRU the minimal change as well?
<smoser> and 'kernel-param' is some special tag that would even be identified as such in the maas-cli ?
<roaksoax> rvba: that'[s the only blocker really
<mgz> definition just specifies which nodes have that tag, based on their reported lshw output
<mgz> by attaching the kernel params to tags, we hope to make the boot after that admin customisable based on the hardware
<rvba> roaksoax: not sure it's the only blocker, I need to check the new dependencies that we use in the quantal version.
<roaksoax> rvba: ok cool that would be helpful
<roaksoax> mgz: right, so i think that regardless of the definition, it's still necessary to be able to assign kernel params to any given system
<roaksoax> mgz: and might be required to assigned based on the name of the system
<mgz> but what I'm still not clear on is which boots matter
<smoser> well, any boot in which you want to do something matters
<smoser> :)
<mgz> with the ability to manually add a tag to any name, you can give one (or more) boxes completely custom params
<smoser> so what happens if multiple tags named 'kernel-params' match a node?
<mgz> what are the names for the three steps again? enlistment, commissioning, something else?
<smoser> and are admin specified appended after MAAS required (ie, iscsi_target=....)
<mgz> smoser: the easiest thing for us is to just take the params from one matching tag, in a deterministic fashion
<jtv> mgz: you mean deployment?
<smoser> that is the easiest thing, yes.
<mgz> basically, tag-level customisation can't kick in till after we have the lshw output, which happens on the comissioning step, right?
<roaksoax> right
<smoser> so if i did like you suggested above with but with a different value with different criteria
<smoser> what would happen?
<smoser> which match would "win"
<mgz> the globally configured ones could maybe be used before that, but there still needs to be a way for the nodes to talk to the maasserver to get that out
<smoser> is that determinable?
<mgz> smoser: one idea is to just take the first tag lexicographically, but we've not coded it yet, so what ever you like :)
<mgz> the UI dimitern is working on aims to make this clear
<smoser> that word is way to big for me.
<mgz> by displaying the params used and where they came from
<mgz> smoser: A comes before Z and so on
<smoser> i'm confused.
<smoser> which is why i just wanted some example that showed what would happen
<nveitch> i think thats not a great idea
<dimitern> smoser: you have tags, named "big-gpu", "large-mem", "0-special-tag", etc., so when sorted the 3rd one will be applied first, even if more than one of them match a single node
<nveitch> chronologically would be better
<mgz> the problem with chronologically is that would either need tracking seperately or would be unstable in the face of minor tweaks to tag definitions
<smoser> right.
<dimitern> smoser: and the other ones matching will be ignored, if none matches - use the global params from settings
<smoser> so tag has name and value
<smoser> right?
<smoser> this is confusing me
<dimitern> nveitch: ordering them chronologically poses problem with how to control the order and fine-tune it
<mgz> they have a name, and a definition, which specifies what they match
<dimitern> smoser: name, description (comment) and optionally, kernel_params
<smoser> because it seems like your implying that tag_name="0-special-tag" and value="console=ttyS0"
<mgz> (and a comment, which is just fluff, and shortly also an optional list of kernel params)
<smoser> but then i dont know where the string 'kernel_params' would fit.
<smoser> oh
<smoser> that is odd
<dimitern> smoser: the kernel_params field of the tag is the complete boot command line for the kernel, as I understand it - all params as a string
<smoser> so you're suggesting that names have, importantly, a name and a 'kernel_params'
<mgz> no, tag (name='0-special-tag', definition='true', kernel_params='console=ttyS0)
<smoser> dimitern, well, it can't have the entire set of parameters, because that is not knowable outside of maas.
<smoser> that is wierd.
<dimitern> tag names are used just for sorting
<smoser> dont you think this is weird?
<smoser> we're promoting "kernel_params" to some sort of super item that is even then part of a tag?
<dimitern> smoser: maybe I don't get it well enough to tell why it's weird?
<mgz> maas will still supply some internally (needs to for various things that take arguments), but will allow you one extra set of params to stick on the end
<smoser> if you told me "you can apply tags to nodes or groups of nodes"
<mgz> smoser: yes. but I don't think it really matters.
<smoser> i would think that a "tag" is either a key:value pair, or (less usefully) a 'name'
<smoser> but i would probably not expect that it is a key:value:kernel_params triplet
<dimitern> smoser: a tag is a few extra properties, which can be attached to a node, one of them can be the kernel params
<mgz> it complicates the tag is name+definition thing, but sticking tag associated kernel params elsewhere wouldn't really change much
<dimitern> smoser: it's not just key/value
<mgz> I would prefer if the apis were kept clean of kernel stuff, but jam disagreed on the basis that this is the simplest thing
<mgz> it would be perfectly possible to stick the actual data whereever, the main question is if getting that stuff based on one tag would be okay for what you'll want to do with it
<mgz> the main issue with fancier things seems to be that kernel params are pretty opaque to us, we can't easilly tell what overrides what, what combines with what, so merging multiple params is non-trivial
<smoser> right.
<smoser> so i think its not at all unreasonable to just throw up your hands. and not attempt to merge or join in any way, but just let some definition "win"
<smoser> but with tags, that seems somewhat arbitrary.
<smoser> so i suspect thats why the lexical ordering based on 'name'.
<mgz> right, and then we document that you name your tags specially for kernel params with 001- prefixes if you want clarity
<smoser> but if we're playing the funny name game ...
<smoser> then why not
<smoser> 001-kernel-params="console=ttyS0"
<mgz> because how do you then associate 001-kernel-params with a particular set of machines?
<smoser> however you do that anyway.
<smoser> i had assumed the --definition thing was not specific
<smoser> to kernel params
<mgz> or you mean pickling the params into the tag name?
<smoser> if i have both key and value for a "tag" (which 'im still confused by)
<smoser> then no pickling needed.
<mgz> tags are just names, with some fancy logic attached
<mgz> the fancy logic is optional
<mgz> rather than needing to manually look at each machine and say whether a tag applies to it or not (node triage...)
<mgz> you can give a definition that we then use to automatically apply the tag if appropriatte to any new machines as they show up
<smoser> ok. that at least maks sense.
<mgz> the kernel params are again just a bit of extra logic bundled with the tag, to say when booting a machine that matches this tag, add in these params
<smoser> then yeah, my solution would require pickling name/value
<smoser> which i dont think is beautiful
<smoser> but i dont think the extra special attribute is either.
<smoser> but then my suggestion was that you could just have a name of "001-kernel-params=console=ttyS0"
<smoser> which would probably be improved to move kernel-params to the beginning.
<smoser> but either way.
<smoser> http://pad.daviey.com/maas-kernel-params
<smoser> i put 3 points there that i'd like to at least have considered.
<smoser> and know how i would address them
<smoser> i think it'll work, but i just iddn't wan tto create somethign that wouldn't solve anyone's actual problems
<mgz> thanks, extra details are useful
<mgz> the -- is interesting... I wonder if we can leave that just to the user supplied params?
<mgz> or will some of the ones that maas needs to add in itself have to be mixed before and after?
<mgz> ...or can we assume user supplied ones should always be carried over to the installed system?
<smoser> mgz, well, the -- is annoyhing.
<smoser> and i ran into this personall a bit ago.
<smoser> the BOOTIF= is the most annoying bit
<smoser> as it gets copied over :)
<smoser> as IPAPPEDN is "append"
<rvba> jam: jtv mgz: matsubara fixed the Jenkins problem (apparently, rebooting the vm many times fixed it).
<jtv> rvba: \o/  thanks!
<rvba> Diogo will investigate further but in the meantime so that it does not happen again but we can land our branches.
<jtv> And thanks to matsubara for the work...  what fixes it for me is a manual âfsck -f -y /dev/<partition>â from a boot shell.
<mgz> go matsubara
<jtv> rvba: you landed my lint branch already?
<rvba> jtv: yes, I used that branch as a test :)
<jtv> I was so looking forward to landing that one.  :)
<roaksoax> rvba: jtv so when do you guys have a catchup call
<jtv> roaksoax: normally around 08:00 UTC
<rvba> roaksoax: 08:30 UTC
<jtv> (I'm being vague because I believe it was just being changed due to DST)
<jtv> Ah yes, the half hour too.
<roaksoax> uh, that's early for me
<roaksoax> :)
<roaksoax> rvba: ok so we need to have a catch up call to talk about the upgrade path
<roaksoax> and so on
<rvba> roaksoax: ok, we can do that tomorrow.
<roaksoax> alright cool
<jtv> roaksoax: thanks for the review...  I'm out too.  :)
<mgz> matsubara: tried merging again, but it failed once more alas...
<matsubara> mgz, which merge proposal?
<mgz> jenkins job 252, merge for lp:~gz/maas/1.2_fix_duplicated_039_migration
<matsubara> job #253 passed
<mgz> was that against trunk or against 1.2?
<matsubara> 1.2
<matsubara> there are separate jobs for 1.2 and trunk
<mgz> shall I try again and hope?
<matsubara> I just did
<matsubara> let's see
<matsubara> mgz, the eagle has landed
<mgz> woho!
<mgz> clearly jenkins likes you better :)
#maas 2012-11-07
<jam> so mgz, what pieces can I pick up so we can finish up the kernel params stuff, since we got Jenkins running again
<mgz> so, what I still don't know how we're going to do is get the kernel options to the provisioningserver
<mgz> also, did you see the notes from scott in last nights log?
<jam> mgz: I did. I'm pretty confident about the specific setup, the most interesting part was the '--' issue.
<jam> I think for a first pass, we can pick a spot to put it
<jam> and if it is important, we can detect '--' in the parameters being set, and split it to that side of a '--' in the commandline.
<mgz> that makes sense
<jam> mgz: though maas itself doesn't seem to be setting '--', wh
<jam> which means that in the immediate term, we don't have to do anything.
<jam> if they need to pass "opts more opts -- extra_opts after doubledash"
<mgz> it perhaps should, but we can probably punt on it then.
<jam> it would just work
<dimitern> jam, mgz: do we have the call this morning?
<jam> dimitern: not explicitly, though I'm happy to chat with you guys. Shall we just meet up on mumble?
<dimitern> sure, I have a headset now :) noise cancelling
<jam> mgz: ^^
<mgz> er, let me install that, I did the g+ stuff
<mgz> er... that won't be happening any time soon
<mgz> can we do hangout instead?
<jam> mgz: sure, though what happened to you being able to use mumble? (out of curiosity)
<mgz> not using the special laptop, due to the screen borkage
<mgz> and this borrowed one lacks mumble... or the qt libraries
<jam> mgz: sure, I was wondering more about "that won't be happenign any time soon"
<mgz> all will become clear :)
<jam> dimitern: I seem to have 3 google identities for you, should I use the one with you picture associated with it?
<jam> mgz: dimitern: https://plus.google.com/hangouts/_/216f679bd782c69f54ec4960241fd8cedef4340f?authuser=2&hl=en#
<dimitern> jam: well, I have the @canonical one - that's it
<jam> dimitern: when Martin joined, you suddenly disconnected
<mgz> sorry, mistouchpadded the close button
<jam> allenap: poke about the tftp server, I'm trying to figure out if it ever talks to the maas server
<jam> Specifically, I realize now why mgz was worried about getting the provisioning server code to get information from the maas server.
<jam> Because tftp is where the kernel commandline is getting set
<jam> and AFAICT it never talks to the MAAS_URL
<jam> so I can import 'auth' and use that stuff, but I want to make sure that is a sane thing to do.
<jam> (I don't quite understand the layout given that tftp sits right next to the other worker tasks, maybe they run in the same process?)
<jam> rvba: maybe you have an idea about how the tftp service is set up?
<rbasak> Are we still supposed to be using quantal for development, or have we switched to raring?
<rbasak> I ask because I just tried on quantal and maas-import-squashfs appears broken (in a way that the daily ppa probably fixes)
<jam> rbasak: the lp:maas/1.2 branch is meant to be the stable release series for Quantal, I'm not sure about the ppas
<rbasak> jam: what about trunk?
<jam> rbasak: I don't really know. I would think that it isn't worth breaking Q compatibility yet, but I don't know if there is a reason to diverge.
<rbasak> OK
<rbasak> I just had to add multiverse for python-selenium, which wasn't a requirement previously, but fair enough
<rbasak> Next import-squashfs failed :-(
<dimitern> jam, mgz: how is it going guys?
<jam> d
<jam> dimitern: I belive mgz is finishing up his lunch. I've gotten the apis exposed, which should unblock your work.
<jam> Though I have now hit the tftp wall, and while I can force my way through, I was hoping for feedback from red squad to make sure that was ok to do.
<dimitern> jam: I see also your initial MP I depended on is approved now, how about landing it?
<jam> dimitern: I marked it as such, jenkins should pick it up and land it shortly
<dimitern> jam: ok, sweet - i'll get it from there so I can finish my tests
<rvba> jam: I'm not really familiar with the tftp stuff.  But AFAIK, it's a separated twisted process.  It creates 'readers', one reader reads from the local filesystem to serve the images, the other uses the API to generated PXE files on the fly (the url for this is read from pserv.yaml).
<jam> rvba: right, I roughly see the code that does it. The problem is it doesn't seem to use any MAAS logic to actually generate the pxe files.
<jam> (it doesn't talk to the maasserver for anything, it just grabs the request parameters and uses it to fill out a template file)
<jam> at least that I can see.
<rvba> jam: it is supposed to talk to the API to generated the PXE file.
<jam> rvba: k, it might be, I see a 'generator_url' in there.
<jam> ok, found it, I think we can just poke data onto the KernelParameters, which should make things reasonable for us. Yay!
<jam> rvba: thanks!
<rvba> np
<jam> untangling how it was all talking to eachother was a bit tricky
<rvba> Indeed, that twisted code isâ¦ twisted.
<jam> rvba: stuff like having a 'context' object that you can 'get()' data out of
<jam> rather than passing parameters
<jam> and just sort of magical 'get()' calls.
<rvba> Yeah, that makes following what's going on harder.
<jam> something seems fishy about the 1.2 series config. 'make run' is failing to startup again, which seems to be an '/etc/bin/rndc.key' failure-to-read
<mgz> hm, have not tried make run in 1.2 yet
<jam> mgz: I get the feeling people generally don't :).
<jam> mgz: https://code.launchpad.net/~jameinel/maas/1.2-pxeconfig-includes-kernel-opts/+merge/133255 is closing the loop to the tftp server
<jam> mgz: and now I'm 1 hr past EOD, and my son is right in getting me to stop.
<mgz> the son is right
<mgz> play time! see you tomorrow :)
 * mgz gets on with some branches
<roaksoax> robbiew: hwody! is trunk fully operational?
<roaksoax> err
<roaksoax> sorry :)
<roaksoax> rvba: ^^
<robbiew> heh
<rvba> robbiew: hi, we have to fix bug 1075597 but that's underway.
<ubot5> Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/1075597
<rvba> err
<rvba> roaksoax: ^
<rvba> Sorry again Robbie :)
<roaksoax> rvba: alright, I'd like to upload to raring so I can start preparing the SRU for quantal
<roaksoax> rvba: but now I'll look into the precise -> quantal upgrade to see whether i can reproduce the issue or not
<rvba> All right.
<rvba> jtv is working on a fix for bug 1075597.
<ubot5> Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/1075597
<roaksoax> ok cool
<roaksoax> rvba: what were the consequences of upgrade failure?
<roaksoax> rvba: not being able to deploy?
<rvba> roaksoax: if you're talking about 1072772, the main symptom was that the DNS config was not being written.  So yes, unable to deploy anything.
<roaksoax> rvba: ok cool i'm upgrading now so we'll see what's the outcome
<roaksoax> rvba: what else does that affect (without having to enable DNS/DHCP?)
<rvba> roaksoax: no task can be fired so: no power_on/power_off task can be fired, the reporting of the boot images cannot happen.
<roaksoax> rvba: ack!
<roaksoax> rvba: alright, so I just upgraded and the issue didn't appear
<roaksoax> rvba: message about images dissappear correctly
<roaksoax> rvba: testing power management now
<roaksoax> if it works, then DNS should also
<roaksoax> rvba: alright, what seems to be the problem now is that tftpd-hpa does not stop
<roaksoax> rvba: hence maas-pserv doesn't start
<rvba> roaksoax: that looks like another problem.
<roaksoax> rvba: python-maas-provisioningserver should probably conflict with tftpd-hpa... but that's weird because maas already does
<roaksoax> err maas-cluster-controller
<roaksoax> rvba: ok so I upgraded again
<roaksoax> rvba: seems to be working just fine
<roaksoax> rvba: but the logfile for cluster-celery wasn't created
<rvba> roaksoax: and the celery process for the cluster is running?
<roaksoax> rvba: nope
<rvba> roaksoax: any error in the upstart log?
<roaksoax> rvba: there's no log
<rvba> Weird.
<roaksoax> rvba: you can access the instance
<rvba> Yep, I'm in.
<roaksoax> rvba: so nodes enlist and commission
<rvba> Nov  7 17:17:47 server-0fd520a0-454d-43b7-8e86-a7d5a4661e8e kernel: [ 1507.296247] init: maas-cluster-celery main process (1229) terminated with status 1
<roaksoax> yeah there's no error log though
<roaksoax> rvba: what's the logfile that it should have? celery.log?
<roaksoax> or celery-cluster.log
<rvba> /var/log/maas/celery.log
<rvba> roaksoax: I wonder how nodes could be enlisted if pserv is not running.
<roaksoax> rvba: i mnanually restarted pserv
<rvba> ok
<roaksoax> rvba: so I stopped tftpd-hpa and started pserv
<roaksoax> rvba: ok so the maas-pserv issue is definitely becuase tftpd-hpa is running
<roaksoax> rvba: but maas-cluster-celery is the issue
<rvba> roaksoax: I tried to restart maas-cluster-celery and it crashed. maas-cluster-celery main process (8707) terminated with status 1
<roaksoax> rvba: ok i think i got it
<roaksoax> rvba: i think it is the way how it is sourcing MAAS_URL
<roaksoax> or the file that containst it
<roaksoax> rvba: ah no
<rvba> roaksoax: I ran what the upstart job runs and I got this
<rvba> https://pastebin.canonical.com/77847/
<roaksoax> rvba: yep, exaclt what i get
<rvba> /usr/share/maas/celeryconfig_cluster.py is not on the path.
<roaksoax> rvba: yeah I think i saw a bug fixing that
<roaksoax> as it was sourcing something else instead
<rvba> This must have been caused by a very recent change because it was working fine last week.
<roaksoax> indeed
<roaksoax> rvba: IIRC, we looked at it last week and it worked
<roaksoax> rvba: i mean, we found the root cause
<roaksoax> rvba: could this be it? "Another thing that came up is that maas-provision complains about not finding celeryconfig.py, a configuration module that we only use on the region controller.  In a separate branch I'll make the import script select cluster_celeryconfig.py instead."
<rvba> No, I don't think this is related.
<roaksoax> rvba: i'm pretty sure you debug this couple weeks ago, I came to you and you found the problem
<rvba> roaksoax: yeah, this rings a bell indeedâ¦ but I can't find what the problem is this time :(
<roaksoax> rvba: wasn't it something in /usr/sbin/maas-provision?
<rvba> roaksoax: I don't really remember tbh.  Did you manually changed that file?  Looks like it's changed since I checked.
<roaksoax> rvba: yeah but set it back to default
<roaksoax> rvba: it is not that file, i can't remember what was it
<rvba> roaksoax: I think I found the pb.
<roaksoax> rvba: what is it?
<rvba> roaksoax: /etc/maas/maas_cluster.conf contains the wrong URL.
<rvba> MAAS_URL=http://10.55.60.86/
<rvba> No, I don't think this is related.I changed it to:
<rvba> MAAS_URL=http://10.55.60.86/MAAS/
<rvba> Then I restarted the service.
<roaksoax> hmmm
<rvba> If you look in /var/log/apache2/access.log you will see that it was desperately trying to register.
<roaksoax> yeah
<roaksoax> let me test this locally real quick
<roaksoax> rvba: yeah that seems to be the solution
<rvba> roaksoax: We really need to improve the error logging here :)
<roaksoax> indeed :)
<rvba> roaksoax: filed bug 1076080.
<ubot5> Launchpad bug 1076080 in MAAS "No error message is printed anywhere when the cluster controller fails to register itself." [Low,Triaged] https://launchpad.net/bugs/1076080
<roaksoax> cool, I filed bug 1076075
<ubot5> Launchpad bug 1076075 in maas (Ubuntu) "maas_cluster.conf ends up with the wrong URL after upgrade." [Critical,Confirmed] https://launchpad.net/bugs/1076075
<rvba> Cool.
<Lele_> hi guys, im running maas from the ppa dailybuiilds on precise
<Lele_> when im trying to install maas-dhcp
<Lele_> i get
<Lele_> Setting up maas-dhcp (0.1+bzr971+dfsg-0+998+75~ppa0~precise1) ...
<Lele_> Usage: /usr/bin/django-admin config_master_dhcp [options]
<Lele_> Initialize master DHCP settings.
<Lele_> a
<Lele_> usr/bin/django-admin: error: no such option: --interface
<Lele_> dpkg: error processing maas-dhcp (--configure):
<Lele_> subprocess installed post-installation script returned error exit status 2
<Lele_> Errors were encountered while processing:
<Lele_> maas-dhcp
<Lele_> and indeed --interface its not a valid django-admin option
<Lele_> is this a BUG ?
<Lele_> a
<Lele_> anyone guys ?
<Lele__> i submited a bug for the dailybuild https://bugs.launchpad.net/maas/+bug/1076092
<ubot5> Ubuntu bug 1076092 in MAAS "maas-dhcp fails to install on Precise - dailybuild ppa" [Undecided,New]
<Lele__> hope someone can take a look at it
<bigjools> morning
<bigjools> roaksoax: hi, want to talk about SRU?
<roaksoax> bigjools: yes
<roaksoax> but im coming back from lunch
<bigjools> roaksoax: np, I am free for an hour
<roaksoax> sonif you can give a few minuted will be grear
<bigjools> are you drunk? :)
<roaksoax> lol i wish it was friday night for me to do so
<roaksoax> but im from the ohone
<roaksoax> phone
<melmoth> on 12.04, is there an easy way to tell maas to use a specific preseed file for _some_ nodes ?
<melmoth> the idea would be to use lvm for computes nodes that have 2 drives.
<melmoth> right now, maas install computes nodes without touching the 2 drive. So i ended up with 14 drive not being used
<melmoth> (actually, 7 drive not being used)
<bigjools> melmoth: no, that feature will be in 13.04 though
<melmoth> thanks.
<roaksoax> melmoth: u would have to ply witth cobbler
<roaksoax> bigjools: ready when you are
<bigjools> roaksoax: don't encourage people to play with cobbler
<bigjools> since we're getting rid of it :)
<bigjools> roaksoax: ok so is there anything else special needed for the SRU?
<roaksoax> bigjools: I'm not encouraging :) i'm just saying... maybe costumer needs it
<roaksoax> bigjools: ok so, yes.
<bigjools> so far we have: Backport the Django 1.4 feature we're using and any other new dependencies (need to check those)
<bigjools> and test like hell :)
<roaksoax> yes
<roaksoax> bigjools: so, the plan is basically, finish with those 12.10 bugs
<bigjools> and the SRU team agree to this already?
<roaksoax> bigjools: and implement the features the qa requrested, plus those missing such as kernel parameters
<bigjools> I will make a 12.04 PPA build toda to get the ball rolling
<roaksoax> and SRU that to both, quantal, and 12.04
<roaksoax> bigjools: i already have precise in PPA
<roaksoax> its in experimental, same version we have in quantal
<bigjools> roaksoax: the 1.2 branch?
<bigjools> ah
<bigjools> nice
<bigjools> what did you backport?
<roaksoax> bigjools: nothing yet, I simply needed that to test upgrade failures
<roaksoax> but so far is the django thing
<roaksoax> the problem
<bigjools> ok, it won't work then :)
<roaksoax> and dependencies in universe
<roaksoax> bigjools: now, I did want to discuss this
<bigjools> universe in 12.04 and main in 13.10 I presume
<roaksoax> bigjools: when are you guys looking to finish the SRU-able features?
<bigjools> so the "features" were 13.04 only really
<roaksoax> bigjools: right, but kernel parameters are needed to be SRUable
<bigjools> however given the QA team wanted it in 12.04 I think we should backport those separately
<roaksoax> bigjools: uhmmmm maybe
<bigjools> so the initial backport should be the bugs in the stabilization milestone
<roaksoax> bigjools: well the thing is that we discussed this with smoser and jamespage
<bigjools> yeah
<bigjools> so I want to minimise risk
<roaksoax> bigjools: and we agreed that the best approach was to SRU the QA team features as well
<bigjools> if we change too much at once it's not minimised
<roaksoax> bigjools: our mplan is to have precise SRU'd by 12.04.2
<bigjools> ok
<smoser> bigjools, its not minimised :)
<bigjools> that's January, plenty of time
<smoser> that ship has sailed
<roaksoax> yeah
<bigjools> smoser: there's relative levels of minimising :)
<roaksoax> bigjools: however, note that we will have to make incremental SRU's, so I'm gonna start with quantal
<bigjools> so my suggestion would be to SRU the quantal stabilization stuff first, then backport that to precise, then SRU the new features to both
<roaksoax> bigjools: my plan is that, sru stabilization, and then sru to precise
<roaksoax> bigjools: not backport, SRU
<bigjools> roaksoax: perfect
<bigjools> yes, I meant SRU
<bigjools> but it involves backporting dependencies of course
<roaksoax> bigjools: we can't introduce new dependencies
<roaksoax> bigjools: so they will have to get shipped with the maas source
<bigjools> gaaahhhhhh
<bigjools> really?
<smoser> what dependencies are we talking about here?
<roaksoax> bigjools: i have that covered since yui3, and python-tx-tfpt are
<roaksoax> smoser: yui3, python-tx-tftp
<smoser> as in those are not available in 12.04 ?
<bigjools> correct
<smoser> is it just a matter of pulling versions in quantal to 12.04 ?
<roaksoax> smoser: both yui3 and python-tx-tftp are two new sources not in 12.04
<bigjools> this is also a matter for the security team
<bigjools> and who will win? :)
<roaksoax> bigjools: us... we have no other option than doing it :)
<smoser> roaksoax, given the source of this request, i suggest that new packages to 12.04 is at least possibly negotiable.
<roaksoax> smoser: i wasn't aware, nor never heard of being able to introduce new sources into the archive for an older release
<smoser> and from security team perspective, if you put yourself in their shoes, and you were going to have to suppor this one way or another...
<bigjools> smoser: +1
<smoser> which would you rather hvae?  "real packages" or "all shoved inside maas"
<bigjools> exactly - security team will need to track stuff shoved into maas
<roaksoax> smoser: that's out of the question, definitely real packages, but in terms of policies, I don't think that's covered, is it?
<smoser> https://launchpad.net/ubuntu/+source/walinuxagent
<smoser> roaksoax, note, that package is not present in 12.04 "release" pocket.
<roaksoax> smoser: right, alright, if it is possible
<smoser> also note, that i'm not terribly happy with using it as an example, but i'm not aware of others.
<roaksoax> then we should do that
<smoser> ie, i'm not happy with that crappy package . but it is an example of something that went from "not packaged" to "main" in a SRU
<roaksoax> smoser: right,
<roaksoax> smoser: i don't really have issues in doing so, i just wasn't aware it was even possible
<roaksoax> bigjools: ok so, the plan would be to finish 12.10 stabilization, SRU to quantal, then start looking at precise, sru that
<roaksoax> and then SRU raring
<roaksoax> as a second mayor SRU
<roaksoax> smoser: agreed?
<smoser> roaksoax, its not *really* possible
<smoser> :)
<roaksoax> smoser: right :)
<bigjools> roaksoax: yes
<smoser> roaksoax, yeah, i think that is a reasonable path. the only change i woudl suggest is
<smoser> that you do not wait until "finish 12.10 stabilization" before "start looking at precise"
<roaksoax> smoser: oh definitely not, I already have precise packaged
<smoser> ie, lets get a ppa functional with what is in quantal now.
<smoser> right.
<roaksoax> smoser: just need to pull in django fix
<smoser> bigjools, are you just awake because you're stuck on europe time?
<roaksoax> and now split the other dependencies into their own packages
<bigjools> smoser: pretty much
<bigjools> smoser: body clock is screwed to hell
<bigjools> smoser: was up at 4am
<bigjools> sun comes up at 5
<smoser> roaksoax, we should consult security team to see what they suggest on the new dependencies.
<roaksoax> smoser: will do
<bigjools> then my twins wake up ... not much hope of sleep after that :)
<roaksoax> smoser: the only real problem would be yui3, which is a new source that really replaces yui
<smoser> and point out to them that a.) we really, really want SRU of 12.10 maas to 12.04, and b.) walinux did it, so maybe we can too
<roaksoax> so they won't be co-installable IIRC
<smoser> roaksoax, hm.. yeah, i dont know. probably can be sorted out.
<roaksoax> alright, i'll talk to them and then sort things out
<roaksoax> them = security team
<bigjools> roaksoax: great, I'll catch up with you next week about that then (I am off work Friday)
<roaksoax> bigjools: same here
#maas 2012-11-08
<jam> bigjools: ping if you are still around
<bigjools> jam: I am here
<jam> bigjools: /wave.
<bigjools> surprisingly awake too :)
<jam> We've been landing our changes into the 1.2 branch
<jam> but I'm realizing they should also get merged into trunk, right?
<bigjools> I saw
<jam> I tried doing the obvious merge, but there are a handful of conflicts.
<bigjools> ideally they need to start in trunk because I wanted to do a bug-only SRU into quantal :(
<jam> bigjools: I can revert the changes, but the bug was on the 12.10-stabilization
<jam> which I thought meant it needed to be in the 1.2 branch.
<bigjools> It does.  I had intended to SRU features later but I realise that the bugtask was a bit misleading :(
<jam> bigjools: so, if you want I can revert what I've landed so far, retarget them onto trunk.
<jam> There is still the fact that 1.2 doesn't merge cleanly into trunk, should it?
<jam> or are they intended to diverge?
<jam> (So in the *immediate* term, I can put together my teams changes as a single branch, which you can land once you have done the bugfix SRU, and are ready to do a feature SRU)
<jam> though I imagine any branch will bitrot if not landed soon
<jam> bigjools: is it worth doing that effort?
<bigjools> they are diverged
<bigjools> (sorry had to deal with a large insect)
<bigjools> you need to cherry pick revisions
<jam> so, they *have* diverged, but it would be possible to converge them. s there something you don't want in trunk that is in 1.2?
<bigjools> hmmmm let me think more about the feature branch
<bigjools> there's stuff in trunk we don't want in 1.2
<jam> bigjools: right, I'm talking about the other direction
<jam> I was thinking to land our code in 1.2 then merge up into trunk
<bigjools> there *might* be stuff in 1.2 - jtv has been landing there first and forward porting
<jam> but it isn't possible today because of divergence on both side.
<jam> mostly simple textual conflicts that are fairly easy to resolve
<bigjools> you should be able to CP stuff though?
<jam> bigjools: we can, but it seems a waste to do so unless there is stuff you really don't want in trunk
<bigjools> if you can work out which revisions are missing we can decide
<bigjools> but there should be nothing that's not in trunk
<bigjools> unless there was a separate fix landed just for a bug in 1.2
<jam> I'm willing to do the conflict cleanup. The main problem with specific revisions is that cherrypicking makes it hard to see where things specifically come from.
<bigjools> ok thanks
<jam> bigjools: so one diff between 1.2 and trunk that I think you worked on. There is a pserv task "import_pxe_files" in 1.2, and "import_boot_images" in trunk.
<jam> I'm not sure which name you want to use
<jam> .
<bigjools> jam: rvba renamed the former to the latter; he needs to backport it
<jam> k
<jam> I'll make sure to use boot_images after the merge.
<bigjools> rightyo
<bigjools> jam: I think you need to revert the 1.2 features and land them on trunk
<jam> bigjools: i'm willing to do so, do you know an ETA when 1.2 would be opened again.
<jam> I guess at this point, we would essentially just hand it off to your team for that, though.
<jam>  /wave mgz, dimitern
<dimitern> jam: /wave :)
<jam> I'm sitting in mumble if you guys would care to join.
<dimitern> ok
<jam> w7z: poke
<mgz> hey jam
<mgz> ...I wonder if I can find an extra network cable
<jam> mgz: I've got about 10 of them here.
<jam> If you feel like dropping by :)
<jam> mgz: care to join us on mumble?
<jam> bigjools: https://code.launchpad.net/~jameinel/maas/1.2-remove-kernel-opts/+merge/133424
<jam> and now https://code.launchpad.net/~jameinel/maas/1.2-remove-kernel-opts/+merge/133425 to properly target 1.2
<bigjools> jam: if you make a branch targeted to 1.2, we can land it after the first SRU
<jam> bigjools: sure
<jam> the patch is essentially revert this revision :)
<bigjools> ok :)
 * bigjools heading off shortly, tired.com
<jam> mgz: https://code.launchpad.net/~jameinel/maas/1.2-remove-kernel-opts/+merge/133425
<jam> mgz, dimitern: https://code.launchpad.net/~jameinel/maas/merge-1.2/+merge/133418
<jam> mgz: DatabaseError: column maasserver_tag.kernel_opts does not exist
<jam> LINE 1: ...er_tag"."definition", "maasserver_tag"."comment", "maasserve...
<jam> evilnickveitch: just to let you know, I'm landing a 1.2 merge  => trunk which includes your doc fixes
<evilnickveitch> jam, thanks for the headsup
<jam> mgz, dimitern: https://code.launchpad.net/~jameinel/maas/land-kernel-opts-in-trunk/+merge/133434
<w7z> failure is https://jenkins.qa.ubuntu.com/job/maas-merger-quantal/264/console
<jam> mgz w7z: Did you see bug #1073090 ?
<ubot5> Launchpad bug 1073090 in MAAS "Using MAAS and --constraints "maas-name=$SPECIFIC_HOST" juju will deploy when that host is in commissioning or some other non-ready state" [Critical,Triaged] https://launchpad.net/bugs/1073090
<jam> I didn't think this was specific to something we've changed, but I could be wrong.
<w7z> looking
<jam> w7z: from what I can see, we already have: 'available_nodes.filter(status=NODE_STATUS.READY)'
<w7z> hm, we probably inherit the same dodgy logic for other constraints though
<jam> so get_available_ shouldn't be allowing it if the status isn't actually ready.
<jam> though maybe juju keeps poking at us for a specific node, and the status goes to READY before some action actually finishes?
<jam> w7z: is it a possible data race between multiple juju requests to acquire a node, and the backend having a specific node ready?
<w7z> hm, right, only accepting READY seems fine
<w7z> I wonder if a symptom is being misdiagnosed and it's actually some other issue with using maas-name
<w7z> when does READY get changed to something else...
<jam> w7z: when it is "damn well read
<jam> ready"
<jam> :)
<w7z> in acquire, which is before the api returns
<jam> w7z: I think you mean in RELEASE
<w7z> so, no obvious chance for maas to mislead juju here
<jam> but note, any HTTP level request is inside a transaction that doesn't commit until it returns.
<w7z> right, READY->ALLOCATED in acquire, ...->READY in release
<w7z> I think the bug description is just wrong
<jam> w7z: right, and the API level is "get_available(), node.acquire()"
<jam> so we shouldn't return a node without it already being marked acquired.
<w7z> the issue is that if you try to allocate more than one machine with the same maas-name constraint, juju will get no possible options for the second, and be stuck in pending
<w7z> the fix being, don't use that constraint then
<dimitern> jam, mgz: my branch is ready for review https://code.launchpad.net/~dimitern/maas/1.2-node-view-shows-kernel-params/+merge/133449
<jam> w7z: it sounds more like if you do a deploy before the machine has finished being ready, juju will just stop trying
<w7z> I don't think he's actually confirmed that, though release does seem optimistic
<jam> becoming ready
<w7z> sending reboot then immediately putting in READY may not be good enough
<w7z> I don't think that's what he's observing though
<jam> w7z: from what I can pull out of the bug comments, the issue is that you do a 'juju deploy' and it doesn't do anything, because there are no nodes matching the constraint that are READY
<jam> then juju doesn't wake up and realize once the machine is actually ready.
<w7z> I think what he's seeing is just the known issue with trying to deploy with a set of constraints that returns no acquirable nodes
<jam> w7z: is there a specific bug on it?
<jam> I agree that it doesn't seem specific to maas-name, just easier to trigger because maas-name can have at most 1 machine available.
<jam> dimitern: I'm curious about the CSS in your patch.
<w7z> there's a helpful mailing list thread where melmoth was trying to use maas-name... annoyingly it's not the public list >_<
<dimitern> jam: the css just sets the line input to be longer on the settings page
<jam> dimitern: sure, it just seems odd to have it be tied to '.size12'
<jam> .size12 doesn't seem related to 'input'
<dimitern> jam: well, it's what was used for other form fields, so I thought I should reuse it
<dimitern> jam: the problem is, that you cannot set easily (at least I could not find an easy way) the css class of a form field in the template, that's why I changed the parent and made the child input take 100% width, because it was constrained otherwise
<w7z> jam: so, I suspect if the bug reporter checks /var/log/maas/maas.log he will also see "NodesNotAvailable: No matching node is available"
<w7z> likewise the juju provisioning agent log will have something along those lines
<jam> w7z: right, so where is the failure that is preventing  the *user* from understanding that things aren't going to work?
<jam> Is juju deploy failing to give the message immediately
<w7z> that juju doesn't forward errors from the provisioning agent to the client
<jam> is it not supposed to fail right away
<jam> but is failing to retry later?
<w7z> if this happened on bootstrap, the client you're running is talking directly to maas, and you'd see the error
<w7z> but once the master node is up, all the client does is say "juju please make the state look like this" and the provision agent does all the actions against maas
<jam> w7z: sure, and what happens if it can't do so immediately?
<w7z> this means any errors after bootstrap go in the provisioning agent log on the master, and the client never sees them
<w7z> so, `juju deploy something --constraints=maas-name=bogus` will always succeed... and give you an ever pending machine
<w7z> whereas `juju bootstrap --constraints=maas-name=bogus` will fail with the nice error the maas api returns
<jam> w7z: as would `juju deploy something --constraints=maas-tag=no-available-machines
<jam> where the tag exists, but all machines are busy rightnow.
<jam> It may be that juju isn't meant to handle it, as it assumes providers that have capacity...
<w7z> right. anything that doesn't return a machine, or any transient error (though things do get retried in some circumstances that I'm still not completely clear on)
<Daviey> evilnickveitch: Hey, can you join #ubuntu-server pls?
<w7z> so, basically bug 984640
<ubot5> Launchpad bug 984640 in juju "Unsatisfied constraints are not reported back to the user" [Medium,Confirmed] https://launchpad.net/bugs/984640
<jam> w7z: so the only bit that makes me think bug #1073090 may not be bug #984640 is that the 'retry forever' doesn't seem to be working.
<ubot5> Launchpad bug 984640 in juju "duplicate for #1073090 Unsatisfied constraints are not reported back to the user" [Medium,Confirmed] https://launchpad.net/bugs/984640
<ubot5> Launchpad bug 984640 in juju "Unsatisfied constraints are not reported back to the user" [Medium,Confirmed] https://launchpad.net/bugs/984640
<jam> Specifically, after the node *is* ready, it doesn't actually get activated.
<w7z> right, juju gets the "there are no acquirable nodes" response and takes that as immutable truth
<jam> w7z: I thought juju was supposed to retry forever, if it isn't retrying again... sigh
<jam> It feels like they said, no no, you shouldn't care that it isn't ever working because we'll do it again, except we don't do it again.
<jam> You don't need any actual information about what is going wrong...
<jam> anyway, bath time for the boy, have a good weekend mgz, maybe I'll see you in dota2?
<jam> I *think I sent you a friend request, so maybe I'll catch you around there.
<dimitern> jam: ping
<jam> dimitern: pong
<dimitern> jam: so I see the branch is now merged - now what?
<dimitern> should I go back to go?
<jam> 1: work with mgz tomorrow, 2: if you see roaksoak or smoser online, chat with them to see if we can get them to test the kernel opts once it hits the daily ppa.
<jam> 3: work on Go if you run out of things.
<dimitern> jam: ok 10x
<rbasak> roaksoax: around? In maas-import-squashfs, you have "set -o nounset" but etc/maas/import_squashfs doesn't set RELEASES (commented out) so maas-import-pxe-files fails when running out of trunk. What's your intention here? Do you want RELEASES to be set to an empty string in etc/maas/import_squashf?
<roaksoax> rbasak ill have a look in a bit ( taking care of something else right now)
<rbasak> OK thanks
<roaksoax> rbasak: can you pastebin the failure?
<roaksoax> rbasak: and yes i want it to not be set to a string in etc/maas/import_squashfs
<rbasak> roaksoax: http://paste.ubuntu.com/1342601/
<roaksoax> same as what happens with maas-import-ephemerals
<w7z> roaksoax: we've landed all the bits we think you need for customising kernel parameters
<roaksoax> w7z: cool, thanks! how can I customize it?
<rbasak> roaksoax: I think we need a different test on line 47 then
<w7z> can you find a way of trying it out on trunk, and telling us if it does what you need?
<w7z> and also what documentation you want :)
<roaksoax> rbasak: so my intention there is that if in etc/maas/import_pxe_files quantal has not been set, then maas-import-squashfs should not import anything
<roaksoax> w7z: documentation on hjow to add/modify kernel parameters for each particular host or group of hosts
<roaksoax> s/host/nodes
<roaksoax> rbasak: http://pastebin.ubuntu.com/1342612/
<w7z> basically, when creating or modifying a tag, you can supply an optional kernel_opts param (see the for the cli to double check),
<w7z> which will then be appended to the kernel opts used by the provisioning server on boot
<roaksoax> w7z: an example would be helpful :)
<roaksoax> w7z: and you need to let evilnickveitch know too to add that to the documentation on maas.ubuntu.com eventually
<evilnickveitch> w7z, yes, that would definitely help!
<w7z> it's essentially as per the auto generated docs
<w7z> but, specifically
<w7z> something like `maas-cli tag create --name=00-default-console --definition=true --kernel_opts=console=ttyS0`
<w7z> will make all nodes (the defintion says effectively match everything) boot with the extra kernel param "console=ttyS0" after all the other ones maas uses
<w7z> or for a similar thing
<roaksoax> w7z: cool, thanks
<w7z> the current highbank special case could be redone as a tag like `maas-cli tag create --name=01-highbank-console --definition="contains(/node/product,'Highbank')" --kernel_opts=console=tty`
<w7z> or, you can set stuff manually just on one machine
<rbasak> roaksoax: this scripts fails on a fresh updated quantal machine (with multiverse enabled for python-selenium): http://paste.ubuntu.com/1342651/
<w7z> `maas-cli tag create --name=99-special-snowflake --kernel_opts=whatever_magic_param_here` followed by `maas-cli tag update-nodes 99-special-snowflake --add SYSTEM_ID`
<w7z> plus or minus typos
<w7z> the naming scheme is something work mentioning in the docs, if multiple tags match, the last in lexi-order will be used, so number prefixing or a similar obvious scheme should be encouraged
<Lele_> Hi guys, in 12.10 maas doesnt have the snippets and the maas_proxy file
<Lele_> were can i setup http proxy for the image installation
<Lele_> oh i found it :) enlist_userdata
<roaksoax> Lele_: enlist_userdata is for enlistment, not for installation
<Lele_> oh ! i see cause didnt work
<Lele_> where i can configure proxy por installation ?
<Lele_> roaksoax :)
<rbasak> roaksoax: I've filed bug 1076409
<ubot5> Launchpad bug 1076409 in MAAS "maas-import-pxe-files fails when run from trunk" [Undecided,New] https://launchpad.net/bugs/1076409
<roaksoax> rbasak: ack
<jtv> Code review needed!  https://code.launchpad.net/~jtv/maas/import-from-configured-mirror/+merge/133505
<rvba> jtv: I'll take it.
<jtv> Thanks.
<rvba> jtv: I've got a few branches up for review my self if you fancy doing some reviewing ;)
<jtv> I can get to that pretty soon!
<rvba> Cool, ta.
<jkordish> so the maas on 12.10 (non clobber version - I'm actually pulling from the daily-builds ppa) - where are the preseed files? can't seem to find it in the docs anywhere :/
<roaksoax> jkordish: /usr/share/maas/preseeds
<jkordish> roaksoax: wow. that simple.
<roaksoax> :)
<jkordish> roaksoax: feel like an idoit for missing it. I was doing a dpkg -L on all the maas packages and didn't see it :/
<roaksoax> :)
<roaksoax> jkordish: the maas-region-controller package installs them
<jkordish> thanks roaksoax
<roaksoax> jkordish: you're welcome :)
<roaksoax> _
#maas 2012-11-09
<jam> mgz, dimitern: /wave, just figured I'd stop by before heading out again, and see how things are going
<mgz> hey
<dimitern> jam: hey :)
<dimitern> jam: I'm working on the openstack client for go right now
<jam> dimitern: sounds reasonable, certainly worth getting familiar with how everything hooks together.
<jam> You might try looking at some of the code mgz and I landed in goose trunk
<jam> see if you can pull any of the Auth code into your client work.
<jam> mgz, dimitern: anyway, I'm off to go do errands, but have a great weekend.
<mgz> have a good weekend!
<lifeless> goose ?
<mgz> lifeless: go openstack... had to make a silly name
<lifeless> ah
<lifeless> you guys should post to the openstack lists about that :)
<lifeless> I'm positive they'd love to hear :)
<mgz> maybe when we actually start working on it, also monty trolls enough already
<mgz> it'll just be the small amount of client code juju needs to talk to openstack that's currently in the juju.providers.openstack.client module
<mgz> unfortunately python-novaclient is neither sufficient, nor stable or reusable enough anyway
<mgz> (I have got some code that uses it directly, but you have to be very careful about which version of the client you run against which openstack deployments due to api changes which is more than a little annoying)
<mgz> I should recheck if HP have updated their volume stuff yet
<dimitern> jam: ok, same to you :)
<jtv> matsubara, smoser: I see that the Quantal version doesn't import Quantal boot images, only Precise ones...  That seems to have been fixed in trunk, but how does the import script succeed on trunk given that maas-import-squashfs 404's on the Quantal image!?
<matsubara> jtv, you mean the integration tests?
<matsubara> I added a workaround to not download the squashfs images
<jtv> But that blows up in production, right?
<matsubara> don't know what you mean by production but yes, maas-import-pxe-files is broken and there's a bug filed for it: bug 1074167
<ubot5> Launchpad bug 1074167 in MAAS "maas-import-pxe-files fails to download squashfs ephemeral images." [High,Triaged] https://launchpad.net/bugs/1074167
<matsubara> jtv, ^
<jtv> AFAICT support for Quantal nodes is blocked on that.
<jtv> Thanks for the bug number â I'll annotate the other bug I'm looking into.
<matsubara> jtv, according to rvba, not downloading the squashfs only makes things slower but shouldn't break anythign
<jtv> Right.  It's an optimization, and I think it's slated to be replaced by some other approach.
<jtv> Now, I've been assuming that the quantal version was supposed to support quantal nodes.  Do you know if that's the case?  Or is lack of quantal support a known limitation?
<matsubara> jtv, last I tried quantal did work for nodes so not sure what you're referring to
<jtv> But the import scripts don't import Quantal boot images.
<jtv> In the Quantal version, that is.  They do in trunk.
<jtv> So two different problems:
<jtv> 1. The Quantal version doesn't import Quantal boot images.
<jtv> 2. Importing Quantal boot images fails at the maas-import-squashfs stage.
<matsubara> jtv, I see what you mean
#maas 2012-11-10
<AskUbuntu> Where are the instructions for MAAS when one does not own the network? | http://askubuntu.com/q/215204
#maas 2013-11-04
<Epic_> hi, I have a strange problem, I have a couple of maas nodes setup as virtual machines in virsh, so I setup the power controls for the node in maas, but when I try to start (or do anything) with the node that would toggle power, the entie maas server interface hangs/unresponsive for about 60+ seconds and the power is never controlled.  I see no errors in the logs, and the virsh test works fine fromt he command line, am I missing something
<bigjools> morning
#maas 2013-11-05
<freeflying> http://paste.ubuntu.com/6362769/
<freeflying> what could this be
<freeflying> node trying to get boot image from maas
<bigjools> freeflying: ummmm
<bigjools> is the node still defined in maas?
<freeflying> bigjools, no, this is a fresh installation of maas
<bigjools> is anything actually broken?
<freeflying> bigjools, cluster controller on a different server
<freeflying> bigjools, not I can see so far besides this
<bigjools> sorry I don't know what you mean
<bigjools> my point is, this could be normal.  Ignoring the log file, is anything not working as it should?
<freeflying> bigjools, the node can't get the boot image then
<freeflying> bigjools, from the client console, what we can see after fail to boot is: Cannot locate configuration file
<bigjools> well this is before that stage, it's getting pxeconfig
<bigjools> right
<freeflying> http://paste.ubuntu.com/6362441/
<bigjools> so is the MAC correct?
<bigjools> check all the detauls from the request in your server
<bigjools> details*
<jtv> http error?  I guess this'd be the pxeconfig code requesting node metadata from the API...
<bigjools> jtv: no, it's asking for pxeconfig
<freeflying> bigjools, yes from dhcp's dump, its the same mac addr
<bigjools> oh sorry that's what you said, basically
 * jtv unconfuses
<bigjools> freeflying: is the MAC defined against any node in the database?
<bigjools> something is causing a 404....
<bigjools> and I suspect you edited something after the node enlisted?
<freeflying> bigjools,  i think no
<bigjools> freeflying: what are you doing here, enlisting, commissioning or installing?
<freeflying> bigjools, we don't have anything enlisted so far, we just had maas configured, and try to enlist this one
<freeflying> bigjools, trying to enlisting
<bigjools> freeflying: is the cluster accepted ?
<freeflying> bigjools, yes, i had it configured in web UI too
<bigjools> did you enlist any other nodes yet?
<freeflying> not yet
<bigjools> so there's zero nodes in maas right now?
<freeflying> bigjools, yes
<bigjools> ok
<bigjools> when the node enlists it should try to grab quite a few different pxeconfigs before finding one that exists
<bigjools> it should end up at "default"
<bigjools> since we don't know it yet
<bigjools> so this output in the pserv log is normal as far as I know.  Can you send the rest of the log please?
<bigjools> jtv: actually I think my problem is easy now - the func in maasserver.dns is the single call point I want
<bigjools> I can inject more kwargs in there
<jtv> Great.
<bigjools> it just confused me because it was the same name, of course.
<bigjools> freeflying: is your MAAS_URL correct?
<freeflying> bigjools, yes
<bigjools> freeflying: cam you show me?
<bigjools> can*
<bigjools> because it looks wrong in your log :)
<freeflying> MAAS_URL="http://10.215.128.3/MAAS"
<freeflying> this is from maas_cluster.conf
<freeflying> in pserv.conf, its http:///MAAS/api/1.0/pxeconfig/
<freeflying> i tried to use the exact ip addr in pserv.conf, but didn't work out
<jtv> What about "localhost"?
<freeflying> jtv, to me?
<jtv> Yes â I don't see it making a big difference, but not having a hostname in that URL does look suspicious.
<freeflying> jtv, but the region controller is not on this machine
<jtv> Ah
<jtv> Right
<jtv> But then there definitely has to be a hostname part in that URL.
<jtv> I guess the reason for your 404 log output is that the request is going to the wrong server...
<freeflying> jtv, i tried with the API address too, didn't work out
<jtv> Sounds as if you had a different problem originally, and created a second problem on top of it.
<jtv> Do you know anything more than that it didn't work out?
<freeflying> jtv, no
<freeflying> jtv, weird, this time it works
<freeflying> I mean node pxe boot up from maas
<jtv> \o/
<jtv> Right
<bigjools> freeflying: did you fix pserv.conf?
<freeflying> bigjools, re-configure it to use exact api url
<bigjools> ok
<freeflying> bigjools, jtv thanks for help
<jtv> np
<bigjools> jtv: do you know if we can catch undefined symbols in tempita?
<bigjools> short of resorting to :py ...
<bigjools> otherwise I am going to have to update all the damn callsites in tests :'(
<freeflying> if a node stays commissioning status after doing commissioning, what could the reason be
<jtv> Don't know, sorry.  Would be nice if there were a dedicated exception class, but...
<freeflying> the server has been powered off
<jtv> bigjools: make it default to 8.8.8.8?  ;-)
<bigjools> freeflying: it means it failed to contact the server, did you see anything on the console?
<freeflying> after a while since we click commissioning from webui
<bigjools> freeflying: and what else have you changed in other configs?
<freeflying> bigjools, the other part I did was configure maas-dhcp server, add some other subnet into the file
<freeflying> bigjools, configured import-pxe-files to only download precise's
<jtv> I think commissioning will use Precise...  One thing that may have happened though is a failure while downloading the ephemeral image.  It's huge.
<jtv> Any chance of a look at the node's console?
<freeflying> jtv, on it, windows xp crashed :)
<freeflying> and the kvm works under widows only
<jtv> Losing your host will do it...
<freeflying> :)
 * jtv grabs a break
#maas 2013-11-06
<bigjools> hey roaksoax, got a minute?
<roaksoax> bigjools: hey!
<roaksoax> sure! what's up?
<bigjools> roaksoax: hey dude
<bigjools> I am trying to make maas configure the forwarders option for bind
<bigjools> it's tricky, because I can't duplicate the options{} block
<bigjools> at least I think that's the case - just wanted to check with you on your opinions of what I can do here?
<roaksoax> bigjools: uhmm, so i do not thinkig we would need to duplicate the options would we?
<roaksoax> bigjools: we only need to add forwarders if the user wants to add them
<bigjools> roaksoax: all I need to do is add forwarders - but options is in a file that we don't have templated
<bigjools> named.conf.options
<bigjools> I assumed I could add something only in named.conf.maas.... but bind9 complains there's two options blocks
<roaksoax> bigjools: i see what you mean
<bigjools> so I'm looking for suggestions :)
<roaksoax> bigjools: let me look at it a bit deeper
<roaksoax> bigjools: otherwise, we would need to ask lamont
<bigjools> hang on, door
<roaksoax> bigjools: yeah, I think we'd need to ask lamont
<roaksoax> bigjools: ok I think we can do this:
<roaksoax> zone "pixar.com" { type forward; forwarders { 138.72.10.20; 138.72.30.28; };
<roaksoax> };
<bigjools> sorry, back
<roaksoax> bigjools: http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_05.htm
<bigjools> roaksoax: I looked at that, it's not what we want I think
<bigjools> it only forwards for requests on the same zone doesn't it?
<bigjools> unknown requests I mean
<bigjools> the point of setting the global one is so that nodes can resolve non-node addresses
<roaksoax> yeah
<roaksoax> it seems so
<bigjools> the other option is to template the options file
<roaksoax> bigjools: yeah... i don't think that'd be ideal
<bigjools> quite
<roaksoax> bigjools: i think we should ask lamont
<bigjools> or - try and rewrite it on the fly
<bigjools> is lamont there?
<roaksoax> bigjools: in ODS? no
<bigjools> ok.  how'd it go BTW?
<roaksoax> bigjools: the demo? didn't finish
<roaksoax> not enough time
<bigjools> damn
<roaksoax> bigjools: so he just didn't show the running cloud
<roaksoax> but things were working
<bigjools> roaksoax: was that from the bundle deployment?
<freeflying> when apply a tag to a node, can I use hostname of the node?
<roaksoax> bigjools: yup
<bigjools> freeflying: http://maas.ubuntu.com/docs/tags.html#manually-assigning-tags
<bigjools> basically, no, you need to use system id
<freeflying> bigjools, ok, thanks :)
<bigjools> roaksoax: will you chase lamont or do you want me to do it?
<roaksoax> bigjools: I think it would be better if you could since i'm still oin hongkong and i'm on and off
<bigjools> roaksoax: no probs
<bigjools> cheers
<roaksoax> bigjools: thanks!
<shimey> Does MaaS have the ability to discover/configure IPMI addresses automatically/dynamically?
<bigjools> shimey: sort of.  It inspects for BMCs on enlisting machines and reprograms one of the user slots with a known password.
<bigjools> that gets sent back to maas when it finishes enlisting so then maas can use ipmi
<shimey> that's cool, but you still have to obtain the IPMI IP address...
<shimey> If it can set the BMC password, should it be able to assign an IP?
<bigjools> it obtains all its details, no more action required.
<bigjools> BMCs get IPs over DHCP generally, and ipmitool can inquire about all that
<shimey> Ahh, so people just DHCP the address
<bigjools> yes, but we don't depend on that to get all the details
<shimey> Wouldn't it make sense for MaaS to DHCP lease the IPMI address?
<bigjools> shimey: it does.
<bigjools> well, it doesn't *have* to, that's the point
<bigjools> but it can
<shimey> Oh, I guess it would...if my IPMI where on the same network as my MaaS
<bigjools> it inspects the BMC details by running a script on the machine itself
<bigjools> yes - many people have separate BMC networks
<shimey> Right I've actually have three separate networks. Public, IPMI and a MaaS backend...  I guess If I had controller of the IPMI network I could start DHCP on the cluster controller for that interface.
<shimey> my MaaS backend I setup a private vlan but the IPMI I don't have real control over.
<shimey> (more of a personal issue)
<shimey> Another question however, I have nodes get stuck for various reasons and have to use IPMI to reboot them, should MaaS have the ability to shutdown/reboot a node via IMPI via the GUI/ReST?
<bigjools> yeah, that's a possibility
<bigjools> for now it assumes (naively) that all the power commands work and the machine boots perfectly every time
<shimey> Ya it seemed like there wasn't any hardware watchdog support for images booting.  And if just using WOL, I think it would be very easy to loose machines.
<bigjools> WoL is going away
<shimey> Ya, I don't think it would suffice.
<bigjools> we plan on adding watchdogs, but it's a little tricky
<shimey> Do you know if there is going to be any support for other linux distros?
<bigjools> not in the immediate future, but you never know
<shimey> PCH watchdog?
<shimey> Maybe a software watchdog would suffice...
<shimey> One last question, any suggestions on getting juju and MaaS working together.  Are there certain versions that work together?
<bigjools> the latest versions in saucy should work
<bigjools> 1.16.2 is the minimum juju-core
<bigjools> also if you want to use precise, add the cloud-archive
<bigjools> I have to factory reset my router now, back in a while
<shimey> I was having trouble getting the ephemeral images to mount in saucy
<shimey> thanks bigjools,  I'll see about 1.16.2 of juju-core and try the could-archive
<shimey> good night
<bigjools> bye
<Azendale> So, I have a weird problem with MaaS. If too many (like 5) machines start within 20 seconds or so of each other, it seems like it will overload MaaS, and then past that, the dhcp server will not respond over the network, even well after any load has gone away. Rebooting the MaaS server will fix it
<Azendale> Running ps aux shows the DHCP process, and a packet capture on the MaaS server with the DHCP on it will show all the requests making it in
<Azendale> any suggestions on where/how to figure out what is going on?
<freeflying> maas-cli maas tag update-nodes my_tag remove=system_id, seems doesn't remove a node from a tag, anything I missed?
<jtv> gmb: were you saying you might extract run_command / run_remote_command / test_install_maas_package?
<gmb> jtv: Yes.
<jtv> gmb: great!  It ties in with my work.  Could I ask you to push your branch at EOD even if it isn't done, so I can build on it tomorrow?
<jtv> Tag-team dev...
<gmb> jtv: Certainly!
<jtv> Thank you.
<jtv> And with that, I bid you good day.
<phenoche> is it possible to create software raids with maas and juju?
<Azendale> phenoche: you mean as in the various services having a software raid configuration of their hard disks?
<phenoche> yeah i have a box with 2 disks that is enlisted and i would like to run say the wp charm on that but have it make a software raid with the disks
<phenoche> Azendale, so yes : )
<Azendale> phenoche: this is going to be more of a juju thing to deal with. But a charm can be made to set up any configuration you can configure an ubuntu install in. The wordpress charm probably doesn't have the code to do that though (I could be wrong, look at the charm options). If you wanted to make your own version of the wordpress charm and get into what the code in the charm does, you could make a modified version of the charm, and then specify what machine it get
<phenoche> Azendale, great thanks thats exact what i was wondering i'll check in on this on the juju side of things
<Azendale> phenoche: No problem. I think you can get the wordpress code with something like bzr branch lp:charms/wordpress  And you can deploy with something like juju deploy --repository=<path_to_branch_on_your_disk> local:wordpress
<phenoche> so I have been looking a little closer at creating a raid with maas and i think you might be able to do it by altering the preseed file. does that sound like I'm on the right path
<bigjools> good morning
#maas 2013-11-07
<shimey> help
<bigjools> shimey: hello
<gmb> Morning jtv. Sorry I didn't email you to tell you where I'd put my (very vestigial) branch. Did you find it, and was it useful?
<gmb> (I got totally sidetracked yesterday, so I'd only just started iterating on it)
<jtv> gmb: I did not find it, therefore no, not very useful.  :)
<gmb> jtv: Damn, and it was my only maas-test branch, too. Oh well. Apologies again. Somehow I suspect you managed fine :)
<jtv> gmb: But this may put you in the position of the most understanding potential reviewer for https://code.launchpad.net/~jtv/maas-test/create-admin/+merge/194284 :)
<jtv> gmb: not the lint one!?
<jtv> Ah.
<gmb> Ah, well played sir :)
<gmb> jtv: Branch, not MP.
<gmb> And so no
<jtv> *Now* I see extract-functions.  Why didn't I see that?  I thought I looked about 5 hours ago.
<gmb> It wasn't my only one
<gmb> jtv, I don't think you'll have suffered for it; I only just got started teasing things apart.
<jtv> gmb: careful with the "test base class" stuff â you may hit a nerve with this team.  :)
<jtv> We have tons and tons of testcase classes, base and leaf and mixed, in the main tree and it's been getting to us.
<gmb> jtv: Yeah, I know. That was my second iteration or so... I just wanted to pull it out of the test case for starters and then I could start massaging it into a nice utility.
<gmb> jtv: Anyway, I'll review your branch now.
<jtv> Thanks for both then.  :-)
<jtv> rbasak: morning, and Q about uvtool... does it set up a fixed username/password or something?
<rbasak> jtv: the username is "ubuntu" (cloud-init default in our cloud images).
<rbasak> jtv: the password is unset by default. uvtool defaults to adding your ~/.ssh/id_rsa.pub if you have one and aren't overriding cloud-init userdata.
<rbasak> jtv: or use --password to set a password for the ubuntu user
<jtv> rbasak: oh, and I forget â how do I actually reach the VM, networking-wise?
<jtv> I tried <vmname>.local but no luck.
<jtv> Heyyy, that just started working.  Maybe it just needed a while!
<jtv> Thanks for the pointers.
<jtv> No, not pointers.  The info?
<rbasak> jtv: .local seems unreliable and I might drop it. It takes ages for the instance to install avahi-daemon.
<jtv> Oh, will the uvt-wait thingy actually wait for the hostname to show up?
<bigjools> what does this do over vagrant, out of interest?
<rbasak> jtv: the latest trunk adds "wait" and "ip" subcommands. It actually just peeks at /var/lib/libvirt/dnsmasq/default.leases
<rbasak> jtv: (so requires that you are using the default libvirt bridge/dnsmasq)
<rbasak> bigjools: it does far less than vagrant, intentionally. It's just a wrapper over simplestreams, libvirt and cloud-init. So you get to use unmodified cloud images directly, with easy cloud-config definition or direct use of your own overrides.
<bigjools> nice
<bigjools> cloud images you say ...
<bigjools> could these be the same cloud images that maas uses for ephemerals?
<bigjools> rvba might see where I am going with this ...
<rbasak> Remind me how ephemeral images differ again?
<bigjools> I have no idea if they even do
<rbasak> I'm sure we do; otherwise we wouldn't generate them.
<rbasak> I just can't remember how.
<bigjools> we mount them over iscsi as /
<rvba> rbasak: right now maas-test uses the name published over avahi, but as we discussed yesterday, we will ditch that and use the ip (parsed from default.lease).  The only thing we need is for wait to wait until cloud-init has run.
<bigjools> rvba: are you using cloud-init to install maas?
<jtv> I suppose we could always just loop until either the hostname resolves or we lose patience...
<jtv> bigjools: not yet.
<rvba> bigjools: not currently, but we could.
<jtv> Right now it's apt-get (good enough for me tbh)
<bigjools> in other words is cloud-init an overhead we don't need?
<rvba> bigjools: we need it for something else
<bigjools> I would be looking to pare down the VM image as much as possible
<rvba> To configure the bridged interface.
<bigjools> how big is it BTW?
<bigjools> I'm a bit concerned about the amount of downloading to run this stuff
<rbasak> http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral/view/head:/maas-cloudimg2ephemeral is the cloud image to ephemeral image conversion script
<rvba> I'm not going to disagree with that.
<bigjools> rvba: can we use that script to cut some downloading out?
<rbasak> uvtool can use an external image, provided it's in qcow format. It just needs it to have cloud-init installed. I'm not sure if ephemeral images will work, though.
<bigjools> well if we just download cloud images and then run this tool ...
<rbasak> I'm a little lost. What's your goal here?
<rvba> bigjools: maybe, this is worth looking into.
<bigjools> it's a shame it doesn't explain what it's actually doing
<bigjools> rbasak: so we have to download images for uvtool and then again download ephemeral images for maas inside the vm.  We should be able to avoid that here
<bigjools> they are largely identical
<bigjools> apart from whatever this script does
<rbasak> I see. That makes sense.
<bigjools> I made sense \o/
<rbasak> One possible difference is that ephemeral images need to ignore other disks that are on the machine. uvtool uses cloud-init to supply metadata without a metadata server by adding a second disk with the metadata and having cloud-init find it. I wonder if that's compatible.
<bigjools> I don't see how it can even come into play
<rbasak> uvt-kvm will take a --backing-image-file qcow2 format file directly if you have one
<rbasak> Then it doesn't need to have downloaded anything previously; you can take care of it.
<rvba> rbasak: btw, I managed to dhcp a node on the bridge interface all right.
<jtv> Oh, you have the bridging working?
<rbasak> \o/
<bigjools> \o/
<rvba> Yep, as a proof-of-concept for now: i.e. no code, just manual stuff.
<rvba> But now it can be transformed into code.
<rbasak> rvba: if I drop avahi-daemon installation, and you drive it from outside the guest, then do you still need the wait subcommand extended? I'm not exactly sure how to do that part right now.
<rvba> rbasak: that would be idealâ¦ but I think we can work around it.
<rbasak> rvba: btw, you can wait for /var/lib/cloud/instance/boot-finished to appear. It just doesn't work in the general case. I've talked to smoser about something in /run so things don't get confused by reboots
<rvba> rbasak: instead of using uvtool's cloud-init config and inject our stuff in the run-command, we could simple configure the entire cloud-init's config.  With this level of control we would be fine.
<rbasak> rvba: sure, you can do that.
<rvba> rbasak: hum, even if we do that, we would still a way to know when cloud-init has finished running.
<rbasak> rvba: as a workaround, I think "while ! -f /var/lib/cloud/instance/boot-finished; do sleep 1; done" should do it.
<rbasak> rvba: it just doens't work in the general case, because if you booted the backing image previously, then that file already exists.
<rvba> rbasak: thanks, that should work in case so we can use that until something more solid is in place.
<rvba> in our* case
<rvba> rbasak: I also want to try using a "interface type='direct'" to see if we can avoid setting up the bridge.
<rvba> rbasak: it seems to work just as well.
<roaksoax> rvba: ping
<roaksoax> err
<roaksoax> rbasak: ping
<rbasak> roaksoax: pong
<roaksoax> rbasak: pm :)
<rvba> rbasak: any reasons why we should use a bridged connection instead of a "direct" connection?  A "direct" connection seems a bit simpler to set up.
<rbasak> rvba: for the connection to the independent NIC that connects to the MAAS node under test? "direct" would be fine in that case, if it works. I've never tried it.
<rvba> rbasak: yes, for the connection to the node.  I just tried it, it seems to work fine.
<rvba> We will start with this then.
<jtv1> rvba: it's about a TODO (I think you probably wrote it) to check for an ssh key.
<jtv1> AFAICS we can't connect to the VM without one, unless perhaps we start doing nasty stuff with stdin/stdout.
<rvba> Right.
<jtv1> And uvtool only uses the default key, which is something I'd love to change but for now seems to be something we just have to deal with.
<rvba> Well, there is a way around this.
<jtv> Oh?
<rvba> i.e. we could generate our key and use thatâ¦
<jtv> That's the thing â the path is hard-coded in uvtool.
<rbasak> If you generate your own cloud-config and supply uvtool with that, you can supply whiever key you want
<jtv> $HOME/.ssh/id_rsa
<rvba> That's exactly what I was thinking about.
<jtv> Ah!
<rvba> And we want to do that anyway.
<rbasak> Or I could make the path overridable I suppose. I just need sensible logic that works reasonably in the general case.
<rvba> Because we will want to add ppas and stuff.
<rvba> So owning cloud-init's config seems like the logical thing to do.
<jtv> rbasak: command-line option?  For our usage it'd be fine to have a disposable key.
<rbasak> rvba: or you could do that stuff (apart from the key) from a script. The advantage of that is that it would be synchronous, and slightly easier to debug.
<jtv> Easier in some ways, harder in others...
<rbasak> jtv: a command line option would be fine
<jtv> But anyway, for now userdata sounds like a good option.
<rvba> rbasak: well, technically we could even use the run-cmd-once to set up another ssh key.
<rbasak> I'm a fan of making everything synchronous where possible
<rbasak> create, wait, grab the ip then run a script that calls ssh as needed
<rbasak> Then you can do things synchronously across the host and the guest
<jtv> (Out of interest, what is the advantage of setting up PPAs and installing packages through cloud-init, given that we need to ssh into the VM anyway?)
<rbasak> jtv: right
<rvba> Well, in my view, it's just that cloud-init is already designed to do most of the preparation things we will want to do:
<jtv> (I wasn't making a statement, only asking)
<rvba> set up ssh keys, install ppas, install packages.
<rvba> Sure, we can do most of this (not the ssh key installation) manually using ssh.
<rbasak> rvba: I think that's handy when you don't have an easy synchronous means to do the same thing. Eg. if you want to start an EC2 instance and don't want to hang around to ssh into it, but just want it deployed.
<rbasak> rvba: OTOH, using ssh gives you stdin/out/err capability, for example. You aren't running scripts in the dark. So I think cloud-init's functionality makes sense when there is no ssh option, but ssh makes sense in the general case when there is an option to use it.
<jtv> I think basic laziness is the most powerful argument here.  :)
<jtv> I'll selfishly focus on the ssh key.
<rbasak> Debugging and development velocity for me. Unix-land is focused on sending errors to stderr when something fails. With ssh, everything is hooked up so you see it.
<rvba> rbasak: that's a good point.  So, could you add an option to uvtool to support using a user-defined ssh key?
<rbasak> (and the user, too)
<jtv> rbasak, is it a matter of passing something like userdata=<data> to uvt-kvm?
<rbasak> rvba: sure. That's relatively easy.
<rvba> jtv: yes
<jtv> No base64 or anything like that?
<rbasak> No base64. Although you do need a temporary file.
<rvba> jtv: If rbasak can do, then all maas-test has to do is create and use that key, how does that sound?
<jtv> Well... is uvtool in the cloud PPA?
<jtv> In other words, is it safe to depend on a future version of uvtool?
<rbasak> You mean the cloud archive? Yes, but we can't really put new functionality into Precise before Trusty's release, if that's what you're asking.
<rbasak> (into Precise via the cloud archive, that is)
<jtv> Then ISTM it's not quite enough to add an option to uvtool and make maas-test use it.
<rbasak> It looks like "--user-data -" would work, so you can send it in stdin. I think. Not tested.
<rvba> We can create a file, that's really not a problem.
<jtv> And I can take a cue from create_default_user_data I suppose.  It sends yaml.
<rbasak> Which release does maas-test need to be able to run in?
<jtv> Good question.  Arguably, 14.04 would do because we run a VM anyway.
<rvba> Well, uvtool needs to be installed on the host machine.
<rvba> bigjools: ^ "Which release does maas-test need to be able to run in"
<jtv> Yes, but what I mean is: if you want to test with MAAS on Precise, you can tell maas-test to fire up a Precise VM.
<rbasak> Right. You might just be required to have a 14.04 laptop to run the test on though, for example. Right?
<jtv> The problem with writing our own key right now is that its filename has to be $HOME/.ssh/id_rsa[.pub].
<rbasak> Though uvtool is in the cloud-tools pocket, and that will be updated on release of Trusty (for Precise).
<jtv> If there's already a key there and it has a passphrase, it gets harder.
<rvba> rbasak: right, but still, I'd like bigjools to comment on that.
<rbasak> OK
<rvba> jtv: we should put maas-test's key somewhere else
<jtv> Duh :)
<rvba> and write our own cloud-init config with that key.
<jtv> Yes...
<rvba> I don't see where the problem isâ¦ ?
<jtv> I think we were thinking different things when you said we can write a file...
<rbasak> I can add an option to override the default path. Provided that it's OK for uvtool with that option to only be available in Trusty and Precise (with cloud-tools after Trusty's release), then you should be OK using it.
<jtv> Anyway, yes, I'm looking at the code that generates the default user data.
<rvba> rbasak: I think that option is useful to have.  But we need to figure out if we can rely on newer version of uvtool.
<rvba> jtv: at the time we were talking about the user-data file.
<jtv> Either that, or do the userdata thing.
<jtv> rvba: I think I interpreted it as an alternative to what we were discussing.
<rvba> jtv: ah ok
<jtv> Tone of voice does not carry well on IRC... Let's try talking louder.  :)
<rbasak> I THINK WE AGREE
<rbasak> :)
<jtv> YAY!
<rvba> OK!
<jtv> R U ON AOL 2 LOL
<jtv> rbasak: I'll just imitate the yaml dump from create_default_user_data then...
<rbasak> jtv: sure. Do you still want an option to override the default key path?
<bigjools> rbasak: last LTS is all that counts, plus HWE kernels as necessary
<rvba> jtv: as a reference, here is a real-world user-data created by uvtool: http://paste.ubuntu.com/6375343/
<jtv> Thanks rvba.
<rvba> jtv: line 7-13 is the base64-ized version of the script I gave to uvt-kvm using --run-cmd-once.
<jtv> bigjools: so we don't need maas-test on Precise?
<rvba> jtv: and you can probably get rid of the avahi-daemon stuff, I'll put together a branch today to use the IP rather than the name (because avahi takes ages to publish that name).
<rvba> Use the IP rather than the name when sshing into the machine that is.
<jtv> rvba: why use the IP?  And how do we obtain it?
<jtv> I can look it up, but this sounds as if we already get it somewhere.
<rvba> jtv: no, we use the name published by avahi.
<rvba> jtv: like I said, it's slow, I want to use the IP.
<rvba> utv-kvm gives us the IP
<rbasak> "uvt-kvm ip" in trunk. It reads libvirt's dnsmasq leases file
<rvba> (it parses the log file)
<rvba> the lease file
<rbasak> (but requires that you are using libvirt's dnsmasq for DHCP)
<rvba> And we do that.
<jtv> utvtool in 13.10 doesn't support that, does it?
<rbasak> Right. I'm just paranoid that someone will apply it to a different use case and it won't work.
<rbasak> jtv: no
<rvba> arg
<jtv> So then we might as well go with a "here's my ssh keypair" option.
<jtv> And just not do any user-data unless we really want to.
<rbasak> Do you need it to? You can use ppa:uvtool-dev/trunk for 13.10 this cycle, and after 14.04 release you won't care, right?
 * rbasak unaways himself, as he's actually here.
<jtv> rbasak: I suppose we could, but either way, if we require the PPA anywhere at all we might as well go with the ssh-key option.
<jtv> At least it seems to me the cleanest, smallest, least fragile solution.
<rvba> If we have to stick to what's released now, we could: stick with using avahi for name resolution, customize user-data to pass our own key.
<jtv> But if we then also want to avoid using the hostname in the user-data...
<rbasak> jtv: ie. avoid using avahi
<jtv> ...it'd still be nice to have the "uvt-kvm ip" command.
<rbasak> rvba: why would we need to stick to what's released now?
<rvba> rbasak: I don't know.  I'm just unsure what the requirements are when it comes to using ppas and stuff.
<rvba> rbasak: I'll ask Julian to clarify that.
<rbasak> OK
<rvba> jtv: rbasak But I'd like to use "uvt-kvm ip" as much as you do.
<rbasak> :)
<rvba> rbasak: okay, we will be using the ppa for saucy.
<jtv> rbasak: need any help getting that option for the ssh key in?
<rvba> rbasak: I filed bug 1248895. I phrased it like "Add option toâ¦" because you seem to be filing bug this way yourself.
<ubot5> bug 1248895 in uvtool "Add option to choose which ssh key gets added onto the machine." [Undecided,New] https://launchpad.net/bugs/1248895
<jtv> rvba: all that's missing is the word "should"  :-P
<rvba> heh
<bigjools> snork
<bigjools> I am EODing
<bigjools> cheerio folks
<jtv> nn bigjools
<rvba> nn bigjools
<jtv> I'll be following.
<bigjools> 12h work day for me ...
<jtv> gmb: don't forget to land that lint branch
<jtv> bigjools: don't do that!
<gmb> jtv: Good point, ta.
<bigjools> making up for yesterday!
 * bigjools departs
<jtv> gmb: I now realize why I didn't find your branch â maas vs maas-test
<jtv> Be careful come MP time...
<gmb> Ooh, MemryError. Haven't seen one of those for a while.
<gmb> *Memory
<gmb> rvba: Have at it: https://code.launchpad.net/~gmb/maas-test/extract-functions/+merge/194351
<rvba> gmb: cool, I'll have a look in a minute.
<rvba> gmb: reviewing your branch now.  Here is a tiny branch to review if you have time: https://code.launchpad.net/~rvb/maas-test/vm-network/+merge/194353
<rvba> gmb: I get test failures when I run 'make test' in your branch: http://paste.ubuntu.com/6376687/
<rvba> gmb: err, http://paste.ubuntu.com/6376689/
<gmb> rvba: Oh! I didn't see those before, but I haven't make test since I merged trunk. Hang on.
<rvba> gmb: also, this is what I get when I run the script: http://paste.ubuntu.com/6376753/
<gmb> rvba: Yes, jtv's changes clobbered some of mine and caused problems. Fixing it now.
<gmb> rvba: Should be fixed now.
<rvba> gmb: approved with a couple of remarks.
<gmb> rvba: Thanks
<gmb> rvba: Fixed + asked a question re: check_call.
<rvba> gmb: you're right, there is a bit of duplication: one one side we use addDetail and inside run_command itself we (optionally) check the retcode (but note that the Exception contains all the info)
<gmb> rvba: Right. So, I guess I'm asking: is there a preference for test_install_maas_package?
<rvba> gmb: I think I put the check_call stuff in run_command so that even the commands run at the fixture level can be checked.
<gmb> rvba: When you say "run_command()" you mean utils.run_command(), right?
<rvba> Right.
<rvba> gmb: I'm trying to think what's better for test_install_maas_packageâ¦
<rvba> gmb: on second thoughts, maybe it's better to keep the check at the test level.
<rvba> If we do that everywhere in the test, it will give us more control.
<gmb> rvba: I agree.
<rvba> Cool, let's do that then.
<gmb> rvba: So for clarity: do you mean that we stick with using asserts to check that a process has behaved properly (and therefore treat a failure as a failure rather than an error)?
<freeflying> after a node get a new ip via dhcp, how long does maas take to update its dns record
<rvba> gmb: yesâ¦ do you agree that it's best?
<gmb> rvba: Yes, I do. After all, we're leveraging the testing infrastructure for a reason.
<rvba> Right.
<gmb> Cool.
<rvba> freeflying: it can take up to 1 minute.
<gmb> I'll get this landed then.
<freeflying> rvba, if it doesn't, what shall I look into to troubleshoot?
<rvba> freeflying: check that there is a record for your node in the dhcp lease file (/var/lib/maas/dhcp/dhcpd.leases)
<freeflying> rvba, the lease file has been changed
<rvba> freeflying: can you check that the cluster's celery log contains traces of the task named 'upload_dhcp_leases' being executed?
<rvba> freeflying: also, please check that the region's celery log contains traces of tasks related to DNSâ¦ when new dhcp lease should have triggered a DNS zone rewrite on the region.
<freeflying> rvba, [2013-11-08 00:43:28,566: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[7c0e286e-41c4-49c8-a58c-28ac1b62de6c] succeeded in 3.58162903786s: None
<rvba> That looks all right.
<freeflying> [2013-11-07 19:30:11,536: INFO/MainProcess] Task provisioningserver.tasks.rndc_command[7ba2f719-6988-43f6-b70e-64f2670db0dc] succeeded in 0.0125498771667s: None
<freeflying> seems this is the problem
<freeflying> last update was couple of hours back
<rvba> Yeah, that's not right.
<rvba> Is your cluster configured to handle DHCP *and* DNS?
<freeflying> rbasak, cluster only handle dhcp
<rvba> freeflying: Well, that's your problem then.
<freeflying> rvba, any solution
<rvba> freeflying: change the cluster's setting :)
<freeflying> rvba, like what?
<rvba> freeflying: you can do that in the UI, or via the API.
<freeflying> rvba, what do I need change
<rvba> freeflying: where did you see that cluster was only handly dhcp?
<rvba> freeflying: have a look at http://maas.ubuntu.com/docs/cluster-configuration.html
<freeflying> rvba, the region controller and cluster are on different server
<rvba> Right, so the DNS server is on the region and the DHCP server is on the other server, but still, you need to configure the cluster controller to say that it should relay the DNS info to region.
<freeflying> rvba, so I only need cluster to manage dns? and then region controller will manage dns?
<rvba> freeflying: yeah, you need to set the cluster to "Manage DHCP and DNS" and then the DNS server on the region will have all the DNS info.
<freeflying> rvba,  that is what I have it set via web UI
<rvba> freeflying: can you check that your dns server is running on the region machine?
<freeflying> rvba, yes, it is
<Azendale> I have a problem where if I boot too many machines in my MaaS cluster, it makes the DHCP server in MaaS stop responding until I reboot the MaaS server (which is both a region controller and the cluster controller). Anyone have any ideas where to start debugging?
<freeflying> Azendale, how many node you booted simultaneously
<Azendale> freeflying: 3 or 4 probably will cause it, 5, 6 or 7 will do it for sure
<Azendale> freeflying: These are all KVM machines connected (with virtio drivers) to a bridge on the host machine
<freeflying> Azendale, can you check /var/log/maas/pserv.log
<freeflying> Azendale, uhmm
<freeflying> Azendale, I had similar symptom before, but it were 30 vms
<Azendale> freeflying: ok, I'll look at that log file, give me a few minutes
<bigjools> Azendale: I'm interested in your log file
<Azendale> bigjools: Ok, I can get you a copy. Should a paste bin it somewhere or upload it somewhere?
<bigjools> Azendale: pastebin for now and then we can detemine if it's a bug
<Azendale> ok, sounds good
<bigjools> our QA lab does multiple simultaneous boots OK so this is curious
<Azendale> it seems like usually it will boot (but booting can take a while with all the concurrent image loads). But then at some point after that, either when the installer is trying to get dhcp or on the reboot after the install, the DHCP fails and the machine gets stuck. Usually, I will restart the maas server, and then power off the nodes and start one or two at a time
<Azendale> bigjools: It's going to take a while for stuff to install, and I'll have to run to class in a few minutes, but I'll get back to this in a few hours and get you a copy of the log
<bigjools> Azendale: ok thanks
<Azendale> bigjools: np, thank you for the help!
<bigjools> is DHCP failing or the tftp?
<Azendale> bigjools: it's the dhcp I believe. I've seen cases where the installer can't get an address, and gives up. I've also seen the PXE boot in the VM give up on an interface because it can't get an IP and go to the next interface, ad infinitum. (I may have seen the tftp get stuck before, but I'm not sure on that. I'll keep notes)
<Azendale> ok, got to go for now, but I'll be back
<bigjools> Azendale: ok thanks - that sounds like isc-dhcpd then. see you later
#maas 2013-11-08
<freeflying> http://paste.ubuntu.com/6379554/
<freeflying> maas stop to update dns, a lot of error like this can be found from maas.log
<bigjools> what api call is causing that?
<freeflying> very likely update_dns
<freeflying> when celery-region try to update dns maybe
<bigjools> can you show me the celery-region log please
<bigjools> also the celery-cluster log
<freeflying> http://paste.ubuntu.com/6379577/
<bigjools> rndc is failing - it was previously working.  What changed inbetween>
<bigjools> ?
<bigjools> I suspect the cluster is not authorised as well, it does periodic lease updates on the api (but I can't tell for sure that's what's failing as you chopped the timestamp on the maasserver log)
<freeflying> i found it fails to reolves some node's name, then checked, found ip has changed, but dns not updated, so restart named, it can't be stopped
<freeflying> then did kill it, and restart it again
<bigjools> try this:
<bigjools> maas set_up_dns
<bigjools> it'll re-write everything DNS-related
<bigjools> also show me your cluster celery log please
<freeflying> No handlers could be found for log "metadataserver"
<freeflying> http://paste.ubuntu.com/6379619/
<bigjools> ok so there's failures  earlier on for import_boot_images and restart_dhcp_server,  are those ok now?
<bigjools> also are you still seeing those api failures?  If so can you look at the apache log file and for the corresponding timestamp of a failure look up what the request was
<freeflying> bigjools, they're fine now
<freeflying> bigjools, but no idea maas.log grows that bigger
<bigjools> sorry that doesn't make any sense, are you saying maas.log is growing?
<bigjools> I need to know if you are still seeing those authorization errors for an api call
<freeflying> bigjools, I mean the import_boot image seems fine now, we did it 3 day ago
<freeflying> bigjools, there is no error from maas.log since 10:36
<freeflying> now is 10:50 on the server
<bigjools> and is everything working?
<freeflying> don't have new node deployed, so not sure, is it normal that if no new ip in lease file, then celery-region won'r have anything to do
<freeflying> bigjools, from celery-region.log, there is no action for a while
<bigjools> correct, the celery-region just writes out changes to DNS as required
<freeflying> bigjools, ic, thanks
<bigjools> as I said above you can force a re-write with the "maas set_up_dns" command.
<freeflying> yea
<Azendale> bigjools: here's my /var/log/maas/pserv.log file (with just the stuff from today, otherwise it was 1.5 million lines!) http://paste.ubuntu.com/6379827/
<bigjools> Azendale: your pserv log looks fine.  Can you snip the appropriate bits for dhcpd out of syslog?
<Azendale> bigjools: sure
<Azendale> bigjools: This time, the VM's booted, but the login prompt has something like "192-168-178-68" as the the host name
<Azendale> bigjools: I can ssh to the individual node, but of course have no idea of the credentials to login
<bigjools> Azendale: how did you allocate and start it?
<Azendale> bigjools: juju deploy
<bigjools> then juju puts your ssh key on it, assuming it installed ok
<Azendale> bigjools: log as requested http://paste.ubuntu.com/6380436/ (syslog filtered through grep "dhcpd")
<bigjools> Azendale: there's nothing that leaps out to explain why you had problems.
<bigjools> are things very reliable when booting one at a time?
<Azendale> bigjools: it seems to work just fine when I bring them up staggered a bit
<Azendale> bigjools: I do see in the log "Dynamic and static leases present for 192.168.178.86"
<bigjools> that's normal
<bigjools> and it's definitely the dhcp stage that's timing out, not tftp?
<Azendale> bigjools: I've tried to get this to work so many times, it's hard to be sure for 100% of the time
<Azendale> bigjools: But I know for sure I have seen cases where the PXE boot never gets an address. That didn't seem to happen in this case because I can ssh to the node
<bigjools> I am going to guess that you saturated your network with udp packets
<Azendale> bigjools: I'm not sure what would cause the machine to come up with no host name (but still with an IP)?
<bigjools> it's pulled its host name from DNS I expect, which means cloud-init failed
<Azendale> bigjools: what all does cloud init depend on to succeed?
<bigjools> there's a ton of stuff that can go wrong
<bigjools> hard to see without looking at its log
<bigjools> and if you can't ssh in then  you can't get it
<bigjools> we're working soon on making that easier, but for now you have to use a special ephemeral that hard-codes a backdoor user + password
<Azendale> bigjools: is the log stored on the harddisk? I might be able to power off the machine and then retrieve it, it's a .qcow2 disk image
<bigjools> yes /var/log/cloud-init/
<Azendale> bigjools: ok, I'll see if I can get a copy
<bigjools> ok
<bigjools> I'm leaving soon so I'll catch you Monday, it's Friday here :)
<Azendale> bigjools: ok. If you think it would help, I could try running through a few cycles of boot various number of machines, and collect data if that would help. If so, just give me an idea of what information it would be helpful for me to gather
<bigjools> not sure tbh, if it's a networking problem there's not much I can do
<bigjools> are you on 100M, GigE?
<Azendale> bigjools: it's virtio tap interfaces to a linux bridge on the host
<Azendale> bigjools: http://paste.ubuntu.com/6380463/ cloud init log
<bigjools> ah ok
<bigjools> well that log looks like it booted without maas asking it to boot
<Azendale> bigjools: hm, weird. I have MaaS set up with the virsh option for power management. The machine would have been triggered to turn on (via virsh) by maas when I ran juju deploy. Wouldn't MaaS expect the machine to reboot automatically after installing?
<bigjools> hmm ok
<bigjools> yes it reboots and then installs juju
<Azendale> bigjools: so, if I'm keeping you, just let me know
<bigjools> I am going now actually
<Azendale> bigjools: ok, I guess I'll talk to you Monday
<bigjools> yep - let's continue this
<bigjools> bye
<jtv> rbasak, do let me know if you want help adding the ssh-key-file option to uvtool!
<rbasak> jtv: it's in and build in the PPA
<rbasak> built
<rbasak> jtv, rvba: though --ssh-public-key might more more consistent than --ssh-public-key-file.
 * rbasak doesn't know
<jtv> rbasak: great news, thanks!  The "-file" does make it nicely clear that this is not a key ID.
<gnuoy> Hi, I want to configure bonding on a server under maas control the port I'll be pxe booting via is part of the bond. Any advice on how to go about this ? I can add a script to setup the bonding with the pressed config but it'll be to late then I think.
<gnuoy> ...as the switch will be configured to bond the interfaces right from the start
<gnuoy> Do I need to doctor the initrd image ? or is there a whole world of pain waiting around the corner ?
<jtv> Grrr why can't I do self.useFixture(TempDir()) in KVMFixture!?
<jtv> AttributeError: 'KVMFixture' object has no attribute '_cleanups'
<jtv> allenap: it sounds like the sort of thing you'd know all about...  ^
<jtv> (And __init__ and setUp do their upcalls, so it's not that)
<jtv> Arrrrgh I have it.  The fixture is never set up.
<allenap> jtv: Was the call in __init__?
<allenap> That dosnae work; needs to be in setUp().
<jtv> Yeah, it was just a test that skipped setUp().  Sorry!
<rbasak> allenap: got time at some point to talk about the node subarch field thing?
<allenap> rbasak: Sure. I'm also reading your release combos email right now, so we could talk about that. I need to refresh my mind about the subarch thing. 1145 okay?
<rbasak> allenap: Sure, thanks!
<allenap> rbasak: Cool.
<gnuoy> I've have a Global Kernel Parameter set but one of my nodes its not showing it on the display screen and it doesn't seem to be being passed when the node boots. Any ideas how I can fix that ?
<gnuoy> all nodes are in the "ready" state
<gnuoy> I've tried removing the kernel option and adding it back in again
<gnuoy> maas-cli seems to returns the correct value
<gnuoy> $ maas-cli maas maas get-config name=kernel_opts
<gnuoy> "console=ttyS1"
<rbasak> gnuoy: I think (but am not sure) that the kernel parameters there are for enlistment, commissioning and firing off the installer. For the deployment itself, I think you need to preseed. I'm not sure about what you need to do for fast-path, though.
<gnuoy> rbasak, preseed is fine, but its very odd that the gui isn't showing the kernel params for just one node
<rbasak> gnuoy: ah. I didn't realise you meant that it was missing from the gui. I'm not sure about that, sorry.
<gnuoy> rbasak, ok, thanks for the preseed suggestion.
<rbasak> gnuoy: no problem. Note that I might be inaccurate. My knowledge is a little dated.
<gnuoy> rbasak, ok, noted. :-)
<rvba> gmb: could you please have a look at https://code.launchpad.net/~rvb/maas-test/vm-network4/+merge/194522 if you have time today?  I'd like to get that landed before my long weekend if possible so that you guys will have it on Monday.
<gmb> rvba: Sure, I'll take a look now.
<rvba> gmb: well, okay, I also want Jeroen to deal with all the conflicts this will create with his branch ;)
<gmb> lol
<rvba> Ta.
<gmb> rvba: Approved with a couple of comments.
<rvba> gmb: Thanks!
<gmb> rvba, allenap: I'm almost done for the day; I thought I'd get to jtv's branch but didn't; it's here should either of you want to round out your friday nicely... https://code.launchpad.net/~jtv/maas-test/ssh-key/+merge/194488
<allenap> gmb: Okeydoke. Have a good weekend dude.
<gnuoy> If anyone was following along at home with regards to my earlier problem with installing with bonded nics, in the end initial registration and commissioning worked fine, with the nics on the switch in an active channel-group but the install went dark almost immediately after the initrd was loaded. I got around this by shutting down the second nics interface on the switch.
<mthaddon> gnuoy: I'd like to make sure that's documented somewhere - allenap, how would we get something like that into the maas docs?
<gnuoy> mthaddon, ok, I'll follow that up
<mthaddon> thx
<mthaddon> save someone else your pain :)
<allenap> mthaddon, gnuoy: The docs at http://maas.ubuntu.com/docs are generated from rst files in lp:maas/docs. Add something there - any of us will be more than happy to help - then propose a merge.
<allenap> (That's the docs/ directory in lp:maas.)
<gnuoy> allenap, lovely, thanks
<mthaddon> thx
<allenap> gnuoy: Adding docs to MAAS entitles you to a bit wet kiss. Unfortunately, as the Red Squad representative in Norfolk, I would be the one authorised to deliver your reward.
<gnuoy> allenap, I'll settle for a big wet pint
<allenap> gnuoy: Deal.
<allenap> gnuoy: Or you can wait for gmb to visit the county? Take your pick.
<gnuoy> I'll wait !
<allenap> Think of all those bristles...
<allenap> gnuoy: I've just replied to your weird kernel options bug.
<allenap> gnuoy: Have a good weekend!
#maas 2014-11-03
<tz__> I added a second interface on my cluster. When commissioning a new node, it correctly gets DHCP for the new subnet, but cloud-init gives the error: url_helper.py[WARNING]: Calling 'http://169.254.168.254/2009-04-04/meta-data/instance-id failed [x/y]:  url error [[Errno 113] No route to host]' .. Where is it getting this IP/route from and how can I force it to point to the correct maas server?
<tz__> basically the same problem as the person who posted this: http://askubuntu.com/questions/486821/url-helper-py-warning-cannot-route-to-host-ubuntu-server
<tz__> apparently the 169.254.169.254 url is a default for EC2 instances in cloud-init. any ideas on how to tell it (maas, cloud-init, metadata server) that its not EC2 and it should connect to maas?
<tz__> is there a fast way to login to a commisioned node (ie for the default ubuntu user) ? I'm trying to troubleshoot a cloud-init bug but cant login to look at the log files
<jhobbs> tz__: https://maas.ubuntu.com/docs/troubleshooting.html
<jhobbs> look at the Debugging ephemeral image section
<tz__> interesting, that has one too many directory wildcards. taking it down to 5 fixes it though
<jhobbs> ah, yeah, it's for 1.7
<tz__> any suggestions on reading to make the commissioning script point to the maas metadata instead of EC2? I suspect that's what these cloud-init logs are going to say is happening
<jhobbs> EC2 shouldn't be involved in commissioning or any other time - are you sure the nodes are getting DHCP? you can ssh into them while they're commissioning?
<tz__> yes i can confirm they are getting the correct DHCP address. It's just booting up now so I'll see what the OS says when I login with the backdoor installed.
<tz__> url_helper.py just retries for 120s trying to contact the 169.254.169.254 (EC2) address for some reason though, as if EC2 was set as the default on this additional interface
<jhobbs> can you reach MAAS's metadata server from the node?
<jhobbs> tz__: can you post your cloud-init.log in a pastebin
<tz__> I can ping it, yes. It's just an additional virtual interface on the one maas controller
<tz__> ls
<tz__> http://pastebin.com/vFmfzWNY .. just reading through it now
<jhobbs> tz__: don't see much of use in there, how about /var/log/cloud-init-output.log, /var/log/dmesg
<tz__> jhobbs: http://pastebin.com/GT5EKyZn .. strange it's trying to hit a webserver on the..  gateway
<jhobbs> tz__: what is MAAS's IP?
<tz__> .16
<jhobbs> tz__: can you post dmesg?
<tz__> it has multiple vlan interfaces, 10.100.84.16, 10.100.98.16, etc
<bigjools> morning
#maas 2014-11-04
<lovea> Following instructions at http://www.ubuntu.com/download/cloud/install-ubuntu-openstack. Have installed a Utopic 14.10 server. There is no ppa:cloud-installer/testing for Utopic!
<dimitern> allenap, rvba, gmb, hey guys, any of you around?
<allenap> rvba, gmb: Please can one of you help dimitern? I have to go out very soon (taking my son for an appointment).
<rvba> Hi dimitern.
<dimitern> rvba, hey
<dimitern> allenap, thanks!
<dimitern> rvba, so a quick question for the squid proxy setup
<dimitern> rvba, IIRC there was a way to configure maas 1.7 to cache not just debs but other things, e.g. lxc/kvm images
<dimitern> rvba, I'm doing lots of testing deployments today and I already lost a good deal of time waiting for juju to download the lxc or kvm cloud image on a fresh deployment
<rvba> dimitern: in 1.7 we use a standard proxy (and not squid-deb-proxy)
<dimitern> rvba, right, but out of the box it doesn't seem it caches stuff an node wgets during cloudinit boot
<rvba> dimitern: it's configured to cache only debs by default but you can tweak the config to change that.
<dimitern> rvba, great, how to do that?
<rvba> dimitern: change /etc/maas/maas-proxy.conf and restart the maas-proxy service.
<rvba> dimitern: it's just a squid3 instance.
 * dimitern hates wifi today
<dimitern> rvba, I'm not sure if you got my question - I didn't receive a reply due to my connection dropping
<mgz> 13:58 < rvba> dimitern: change /etc/maas/maas-proxy.conf and restart the maas-proxy service.
<mgz> 13:58 < rvba> dimitern: it's just a squid3 instance.
<rvba> Thanks mgz :)
<dimitern> thanks mgz and rvba ! :)
<MasterPiece> Is exist anyway to use Ubuntu ISO Images in maas controller? How?
<jtv> MasterPiece: it might work with curtinator.  IIRC the URL is https://launchpad.net/curtinator
<MasterPiece> jtv, How can I use of curtinator?
<jtv> I don't know to be honest: only saw a mention of it today.  But AIUI it converts an install image to an image that MAAS can use.
<roaksoax> roadmr: ^^ you wrote it, right?
<roadmr> roaksoax: I did, yes
<roaksoax> MasterPiece: needs input on how to use it
<roadmr> MasterPiece: install bzr, then bzr branch lp:curtinator
<roadmr> MasterPiece: inside you'll find a README file with how to set it up and build a MAAS-compatible image from a desktop ISO
<tz_> hey, i was in here yesterday asking about a commissioning with multiple vlans question. I've narrowed the issue down to the ephem/commissioning image getting the wrong host IP for the variable 'cloud-config-url' . How can I set that variable to not be the default?
<tz_> basically the maas server has 2 vlan/subnets and it gets half of the variables pointing to one subnet (from the cluster config), and half to the other subnet (from some default settings .. somewhere?)
#maas 2014-11-05
<tz_> further tinkering tells me that $cloud-config-url is pulled from DEFAULT_MAAS_URL in maas_local_settings.py .. How do I set that variable for different networks/interfaces in a cluster?
<bigjools> DEFAULT_MAAS_URL is the address of the region and needs to be reachable from anywhere
<tz_> so you're saying the cloud-config-url can only be one IP per cluster?
<bigjools> it's the address of the app server
<bigjools> it's one IP globally
<bigjools> well not IP, a host
<bigjools> URL to a host
<bigjools> all maas networks need a routing to the region appserver
<tz_> so say that host has multiple interfaces setup, how do I set default_maas_url to 0.0.0.0 or some sort of dynamic IP? is it possible at all?
<bigjools> you need to set it to the IP of the interface that can route to and from the cluster networks
<bigjools> s/interface/region's interface/
<tz_> which can only be one at a time?
<bigjools> the URL you set is handed to the nodes when they're deploying so they know how to contact the region
<bigjools> yes, one
<bigjools> on big deployments with many appservers it is likely to be a load balancer
<tz_> this is just one server, that's why it has multiple interfaces/vlans
<tz_> its would be a waste to start throwing in extra cluster controllers and load balancers, hence why I want that one variable to work for the networks setup on the one cluster
<bigjools> you just need a route from the cluster interfaces to the relevant region interface
<bigjools> then it will work
<tz_> yeah i get that, trying to keep the networks segregated without adding a bunch of static routes and cross-vlans
<tz_> in the config file it has a comment "
<tz_> "configuration can, and probably should, override this."  .. do you know what config it's referring to?
<bigjools> maas_local_settings
<bigjools> it's set from dpkg-reconfigure
#maas 2014-11-06
<jgrassler> Hello
<jgrassler> Does anybody know how MAAS determines the IP address it hardwires into boot images as its iscsi server address?
<jgrassler> It seems to disregard the IP address configured for maas-region-controller/maas-cluster-controller
<jgrassler> Which is a bit of an issue in my case: I'm deploying MAAS in an openstack instance, and its eth0 IP address is different from its public floating IP address
<jtv> jgrassler: that sounds pretty exciting.
<jtv> I think the address you want is the one that we call the "MAAS URL."
<jtv> It's defined as the URL where cluster controllers and nodes can reach the region controller.
<jtv> So whenever we need to tell cluster controllers or nodes the address where they can reach the region controller, that's the address we give.
<jtv> You can change it through "dpkg-reconfigure maas-cluster-controller" on the cluster controller machine.
<jtv> (Usually the cluster controller and the region controller run on the same machine.)
<jgrassler> Yes, that's what I tried.
<jgrassler> After which I deleted the images and had maas rebuild them.
<jgrassler> But they still end up with the internal IP address.
<jtv> That shouldn't be needed.  Unless it was changed while I was doing other things, we don't encode the address into the images.
<jgrassler> Interesting...how is the address communicated to the PXE booted system then?
<jgrassler> Does it just use the DHCP-Server's next-address value?
<jtv> IIRC we pass it on the kernel parameter.
<jtv> Ahem.
<jtv> Kernel command line, I mean.
<jtv> The next-address (next-server?  I forget) setting is only used for the PXE boot itself.
<jgrassler> Ok, that's good to know.
<jtv> Off the top of my head, it goes something like...  Node wakes up; goes into PXE; gets DHCP address; requests PXE config from cluster controller; PXE config netboots the node into an install image.
<jgrassler> Sounds sensible, yes
<jtv> I really don't recall but presumably the iscsi address is in the PXE config there somewhere.
<jtv> Ah, maybe what we pass on the kernel command line is just for rsyslog.
<jgrassler> And that's what it looked like on the serial console back when we had our MAAS controller on a hardware node, too
<jtv> I guess you're not using MAAS as the DHCP server?
<jgrassler> No, that would take some dirty, dirty hacks in that scenario :-)
<jtv> So you serve DHCP to nodes that are also on EC2?
<jgrassler> But our hardware MAAS controller was not in charge of DHCP either
<jgrassler> No, not at all
<jtv> Ah, I misread the "no."  :)
<jtv> (My fault for asking a negative question â "no" can mean that you agree or that you disagree!)
<jgrassler> No worries :-)
<jtv> We don't normally test in that situation, so it's quite possible to run into trouble.
<jgrassler> Yeah, it is somewhat unusual
<jgrassler> But we didn't want to keep wasting a - fairly sizeable - hardware node on serving boot images
<jgrassler> Hmm, interesting: https://gist.github.com/jgrassler/6567ce9aee97f8d54918
<jgrassler> Apparently the internal IP address does show up in the root images.
<jgrassler> Never mind. I'm to close to calling it a day for now to properly escape meta characters in my regexes :-)
<jgrassler> I.e. the grep triggered on this kind of thing: /boot/System.map-3.13.0-27-generic:c1020080 t cyrix_identify
<jtv> jgrassler: the images import system _was_ recently rewritten, so I may be behind the times.  But it seems strange to me to see the IP address in there.
<jtv> Oh...
<jtv> Now I see what you mean about the regexes.  :)
<jtv> Yeah, escaping those dots might change things.  :)
<jtv> Or use fgrep I guess.
<jgrassler> At least now I know where not to look :-)
<jgrassler> So I guess I'll delve into the code then - where would I start that?
<jtv> I think in src/maasserver/pxeconfig.py.
<jtv> Or maybe it's src/maasserver/api/... hang on.
<jtv> When the node requests its PXE config from the cluster controller, the cluster controller asks the region controller "please compose a PXE config file for this node."
<jtv> Ah: src/maasserver/api/pxeconfig.py
<jgrassler> Ok...let's see where that ended up in the file system...
<jtv> In there, look for the pxeconfig function.  That's the region-controller function that ends up handling that request.
<jtv> Oh, on an installed machine:
<jtv> /usr/lib/python2.7/dist-packages/maasserver/api/
<jtv> (On older versions, the api subdir would not exist and you'd have to get... either pxeconfig.py or api.py.)
<jgrassler> Thanks
<jgrassler> I think 1.5.4 qualifies as an older version then
<jtv> Whoops yes it does
<jtv> On the bright side, fewer unfamiliar changes.  :)
<jgrassler> That's something :-)
<jtv> It may have been /usr/lib/python2.7/dist-packages/maasserver/pxeconfig.py in that version.
<jgrassler> It appears to be api.py
<jtv> That's a shame.  It was huge.
<jgrassler> It's a bit problematic though...it's invoked from somewhere else
<jtv> But come to think of it... why would 1.5 not give the MAAS_URL..?
<jtv> Yes, it's invoked from the cluster controller, as part of its handling of the node's TFTP request.
<jgrassler> And "somewhere else" supplies a local parameter. Said parameter is the cluster controller's IP address.
 * jtv looks...
<jgrassler> So apparently the cluster controller is the confused party
<jtv> That would be the party where the MAAS_URL is configured...
<jtv> The cluster controller passes the parameters through its API request.
<jtv> The cluster-controller code is, for hysterical raisins, in /usr/lib/python2.7/dist-packages/provisioningserver.
<jtv> This is where it all gets a bit hard to follow.
<jgrassler> From a quick glance it looks fairly messy, yes
<jgrassler> I think I'll put that bit off until tomorrow. Here in UTC+2istan it's getting to be the time where my wife calls asking why I'm not home, yet :-)
<jtv> Ah, I know that one.
<jtv> Anyway, it comes from /etc/maas/pserv.yaml.
<jtv> That's where it gets the URL.
<jtv> And "dpkg-reconfigure maas-cluster-controller" rewrites that bit of YAML with the URL you enter.
<jgrassler> Odd.
<jgrassler> That file contains the public IP address.
<jtv> @!#
<jtv> Old instance still running for whatever reason..?
<jtv> Anyway, run home.  :)
<jgrassler> No
<jgrassler> I configured the packages with a preseed, no chance of that happening :-)
<jtv> Oh make sure it's a URL you see there, not just an IP address.
<jgrassler> Yes, it's an URL
<jtv> Grrr
<jtv> "Have you tried turning it off and on again?"
<jgrassler> It's getting to be tricky, yes...
<jgrassler> Countless time :-)
<jgrassler> s/time/times/
<jtv> No that's fine for us binary types: zero, one, countless.
<jgrassler> That box is provisioned through heat/puppet and rebuilds from scratch in ~10 minutes
<jgrassler> Heh
<jgrassler> And I made use of that mechanism a couple of times - to no avail
<jtv> Oops, and I have to run!
<jgrassler> Alright, let's run then :-)
<jgrassler> Thanks for the help
<jtv> np
 * jtv runs
#maas 2014-11-07
<horatio> I'm having troubles with IPMI + Commissioning. I get 3 of "could not open device at /dev/ipmi0 .. no such file or directory", followed by initscript ipmievd action "start" failed. I added a backdoor and ran the maas_ipmi_autodetect.py script, which works. Then I ran IPMI tool with the credentials the python script returns, and that works too.  And when I check, /dev/ipmi0 exists.
<horatio> The IPMI commissioning failures look like they're blocking the reboot at the end of the script though.
<jtv> Who's up for a pre-imp?
<jtv> Because I've got to do some actual product improvement done this week, as opposed to specs and meetings, or I'll go mad.
<jgrassler> Good morning.
<jgrassler> jtv: I found out where yesterday's problem occurs.
<jgrassler> local_host, local_port = get("local", (None, None))   # tftp.py, line 180
<jgrassler> This retrieves the machine's IP addres from twisted, which is quite sensible in most cases, but not in mine.
<jgrassler> (since it amounts to what you'd get from `ifconfig eth0`)
<jgrassler> I fixed it in a rather messy manner by setting params['local'] manually in the next line, but that's not exactly clean.
<jgrassler> I might cobble up a patch that allows for configuring params['local'] in pserv.yaml - would that have a chance of getting accepted upstream?
<dimitern> hey guys, just a heads-up; as discussed on the x-team call, I've filed a couple of bugs - https://bugs.launchpad.net/maas/+bug/1390404 and https://bugs.launchpad.net/maas/+bug/1390411
<ubot5> Ubuntu bug 1390404 in MAAS "new API to get network interfaces information for a node" [Undecided,New]
<ubot5> Ubuntu bug 1390411 in MAAS "add docs on how to take advantage of maas-proxy to cache custom images (e.g. for LXC/KVM containers)" [Undecided,New]
<dimitern> and I've also commented on a few static ip addresses bugs I filed (mostly questions about whether what's left will be fixed in time for 1.7.1 or 1.7.2)
<jtv> Hi jgrassler â thanks for digging that up!   Let me just digest the whole thing...
<jtv> allenap, did you see jgrassler's note above?  Looks like the tftp code doesn't use MAAS_URL when parameterising a boot method, but "whatever my address is."
<allenap> jtv, jgrassler: I need to remind myself what that code is meant to do.
 * jtv mumbles his usual rant about documenting code
<jtv> The "local" parameter goes into the PXE config as the iscsi address for the boot image.
<jtv> Where do we serve iscsi now?
<jtv> If it's the cluster controller, then MAAS_URL doesn't apply of course.
<allenap> It should be the cluster controller.
<jtv> Blast.
<jtv> We have an abstraction for "address where nodes can find the region controller," but no equivalent for the cluster controller.
<allenap> Well, here we discover it from the address on which the node has contacted the cluster controller.
<allenap> That should be accurate.
<allenap> Unless NAT or something gets in the way.
<jtv> Which seems to be the case here.  :(
<jtv> Maybe this should really be the cluster interface address.
<allenap> The cluster can have multiple interfaces, right?
<jtv> (Which, I know, just raises more wrinkles)
<jtv> Yes, a single cluster can manage multiple subnets...
<roaksoax> scsi is on the cluster
<roaksoax> jtv: we cannot bind the clsuter to one single address
<jtv> Yeah.
<roaksoax> because the cluster manages different networks
<allenap> I think supporting cluster controllers behind NAT is a can of worms at the end of a rabbit hole that I really don't want to get drawn in to. jgrassler, I think it's unlikely that we'll support it.
<roaksoax> if we do NAT< then MAAS would have to inject rules for all the networks it knows about
<jtv> In this case MAAS is not involved in managing the NAT (and couldn't be).
<roaksoax> jtv:that doens't mean that we wont :)
<roaksoax> jtv: but that's not something we will be doing anytime soon
<jtv> I'm guessing if only we could determine an appropriate cluster interface, the cluster interface's address would be the right one here.
<roaksoax> jtv: we can't really know what's te right clsuter interface
<roaksoax> jtv: when It comes to know, the right clsuter interface is the interface they are being managed from
<allenap> jtv: I think MAAS would need to know its real address and the address that outsiders know it by, and relate the two.
<jtv> I think we want "the cluster interface's address from a given node's point of view."
<jtv> Which, yes, is a can of worms.  :(
<roaksoax> jtv: right, and that we *can* know
<roaksoax> jtv: with the NIC->network matching, we can know
<jtv> Not in this case, I think.
<jtv> (We're talking about a very specific scenario here)
<roaksoax> jtv: well, i think we should discuss this in Austin
<gmb> jtv, rvba, allenap: Branch needs review: https://code.launchpad.net/~gmb/maas/check-for-overlapping-cluster-networks/+merge/241061
<jtv> roaksoax: you mean supporting NAT between the node and the cluster controller?  I'm just mulling it over in hopes of finding an easy solution, but if we don't find one, is it a use-case we want to support?
<roaksoax> jtv we might want to support doing nat when both the cluster and region are in the same machine
<jtv> That is the case here.
<gmb> allenap: Thanks for the review. I've replied. After your comments, I think it's safer to just disallow overlaps altogether. Sound sane to you?
<allenap> gmb: Sounds good to me, but I'd like rvba to take a look too. rvba, can you look at Graham's last diff comment on https://code.launchpad.net/~gmb/maas/check-for-overlapping-cluster-networks/+merge/241061?
<rvba> allenap: sure
<jtv> Python wishlist item: have "import" propagate indirect ImportErrors as a different type, so we can tell "I'm trying to import something that doesn't exist" from "I'm trying to import a file with a broken import in it."
<jtv> (Because I'm tired of test runners reporting that my test doesn't exist just because my test contains an import error)
<jgrassler> jtv, allenap: Sorry, I missed the discussion (lunch o' clock got in the way)
<jtv> jgrassler: it's not good news I'm afraid â I hadn't realised yesterday that there'd be NAT between the cluster controller and the nodes.
<jgrassler> I can relate to not wanting to support the oddball scenario we've got here - I'll just fix it locally by templating the address into tftp.py with puppet
<jgrassler> It's ugly but it'll do for now
<jgrassler> These floating IP addresses are a bit of a nuisance, unfortunately.
<jgrassler> It's not the first time we've run afoul of the problem :-)
<jtv> And this is one area where even in 1.7 you won't get IPv6.  :(
<jgrassler> That'll be another can of worms at some point in the future...
<gmb> rvba: I think you missed what allenap was asking aboutâ¦ See the final comment on the *original* diff (circa line 124)
<gmb> (https://code.launchpad.net/~gmb/maas/check-for-overlapping-cluster-networks/+merge/241061)
<gmb> rvba: He's spotted a problem with the assumption that different clusters can define the same networks. And I think he's right.
<rvba> gmb: when we discussed about this yesterday, we were talking about overlapping networks *in the same cluster*.  I don't think it makes sense to have a node related to many clusters (not related in DB terms, I'm talking network here)
<gmb> rvba: Right, so the point allenap is making then â that no two interfaces *anywhere* in a region's scope should have overlapping networks - makes sense. It's not that the node is related to two clusters, its that two independent clusters can right now define interfaces with exactly the same network settings. Which looks fine on paper â they're not the *same* network on the physical level â but once you get to layer 3 and above, they're identical, w
<roaksoax> they could be yes
<rvba> gmb: what I mean it that I don't see why we would have to enforce that in MAAS.  The only problem we could see was the IP allocation and it only becomes a pb if a node is connected (network) to 2 clusters.
<rvba> s/I mean it/I mean is/
<roaksoax> rvba: rihgt, but nodes should not be connected to two cluster, should they?
<rvba> roaksoax: yeah, that's my point.
<rvba> roaksoax: but it's not something we enforce anywhere.
<gmb> rvba: So, my concern is that we're leaving a potential footgun lying around for people if we allow them to do stuff like this. OTOH, you could do some NATing at the cluster level, so maybe it's not a big deal and we should let them. I'm happy with either solution, TBH.
<gmb> And we probably should't be telling network admins what to do.
<gmb> Come to think of it :)
<rvba> gmb: yeah, I think it's the admin's job to sort out the routing.  Unless letting them configure identical network will break something in MAAS, I think we should let them do so.
<rvba> networks*
<gmb> rvba: Okay, I'm happy with that.
<roaksoax> rvba: right, but that's not something we reocmmend either
<rvba> roaksoax: probably not.  But I don't think we should forbid this (again, unless it breaks something in MAAS itself).
<roaksoax> rvba: yeah, if someone does that it is their own fault
<jesk> hi
<jesk> trying to understand maas... having problems it :-)
<jesk> s/it/with it/
<jesk> i'am not able to get informations of the boot order process
<jesk> what I could see so far was that a) server boots via PXE, b) server reboots again and boots via PXE (whyever two times) c) shuts off
<jesk> when trying to install something (only tried juju quickstart) a) server boots and installs image b) reboots and boots again from PXE
<jesk> do I need to deactivate PXE manually or is that handled by MAAS?
<gmb> allenap, rvba, jtv: 'Nother branch for all y'all. https://code.launchpad.net/~gmb/maas/fix-ipmi-wording-bug-1304518/+merge/241075
<jtv> I'll take it.
<allenap> jtv: I've done it.
<jtv> Grrr
<allenap> Sorry :)
<jtv> Bikeshed derby is ON!
<jesk> is there any real technical doc about MAAS?
<jesk> or just cloud-style powerpoint informations :D
<jtv> jesk: http://maas.ubuntu.com/docs1.5/
<jesk> jtv: those docs dont explain what happens when you really want to deploy nodes
<jtv> It's an old version...  more recent docs on maas.ubuntu.com may help.  Did you have anything specific in mind?
<jesk> its more like "type this and that"
<jesk> jtv: i dont get the overall picture of it
<jesk> jtv: concrete use case
<jesk> currently i'am only having one MAAS node and now i want to deploy more nodes. I'am not coming over the step of "booting a server from pxe" which shuts down after PXE boot
<jtv> Okay, so you're clearly beyond the part covered in the Orientation section.
<jtv> Which is good.
<jesk> even wake on lan works
<jesk> "start node" -> node starts, boots from pxe -> and down again
<jtv> Ah, wake-on-LAN is awkward because it has no way to shut down a node.
<jtv> So you already commissioned and allocated the node?
<jesk> yeah, but i'am happy with shutting down fvor the moment via ILO-manually
<jesk> yeah I did that, *but* i'am not able to get the knowledge what it even means :-)
<jtv> Right.
<jtv> I'm sure we documented this _somewhere_, but let me be lazy first and summarise.
<jesk> one node is currently in "allocated to root"-state and one in "ready"-state
<jtv> OK.  The "allocated to root" one should be either being installed, or up and installed with your system and your ssh key.
<jtv> Or, crucially, it could be waiting for you to start it.
<jesk> "allocated to root"-state because of issueing "juju quickstart" most probably, which unfortunately ended in nothing
<jtv> (This all gets much better in the 1.7 which we're currently in the process of releasing)
<jtv> Oh, this was done through juju?
<jesk> i'am not 100% sure if juju started through MAAS a node
<jesk> but i could saw with remote console that a system was installed
<jtv> Oh, an operating system was installed on that node?
<jtv> That's good.
<jesk> after automatic reboot it was booted again via PXE and juju quickstart timeout
<jtv> When you bootstrap a juju environment, it allocates a node for itself.
<jtv> It asks MAAS to allocate a node, and when it gets a node, it tells MAAS to start the node up.
<jtv> As the node starts up, it netboots off the MAAS server, and boots into an install image.
<jtv> Thus it installs an OS, and the user's SSH keys.
<jtv> Then it reboots into the OS that it just installed.
<jtv> At this point the user (which I guess here is juju) has a working node.
<jtv> I guess your node got installed, and then shut down... and did it come back up after that?
<jesk> in my case (i believe) it rebooted from PXE again :-)
<jesk> so the user has to make sure that PXE is turned off on the server when the server reboots finally after OS installation?
<jesk> or can the PXE boot image check if the server was started for normal operation after OS installation and boots from local disk?
<jesk> boot order is: (1) PXE (2) CD (3) HDD
<jtv> Once the node is deployed (as this one seems to be), it will boot off its own disk.
<jesk> to have the flexibility to always boot from PXE for new installation PXE has to be (1)
<jtv> So no need to change that order.
<jesk> if it boots from disk directly boot order must be alsways (1) HDD first
<jesk> but then iam not able to boot from PXE if i want to
<jtv> If the node tries to netboot while it's deployed, the MAAS server tells it to boot from local disk.
<jesk> and HDD gets boots always as soon as the HDD has an valid boot record
<jesk> ah! so via PXE it gets told to boot from local disk
<jtv> Yeah.  No need to change that order: just always let it netboot.
<jesk> interesting but unfortunately seemed not work
<jtv> Any symptoms?
<jesk> it booted from PXE
<jesk> saw this via console
<jesk> but maybe "juju quickstart" dont tell MAAS to handle that installation persistent and leave it as "new installed node"?
<jesk> so much magic :-)
<jesk> i'am just dumb network engineer playing with that stuff
<jtv> So the node that was "allocated to root" booted from PXE?  What happened then?
<jesk> (with a bit of linux and freebsd background)
<jesk> it shut off after that
<jtv> (Yes, far too much complexity â there's a lot less you can count on once you cross the boundaries between machines and between reboots)
<jtv> It shut off...
<jtv> That normally means it's not allocated.
<jesk> ah those server shouldnt shut off after PXE boot?
<jtv> Now, the situation as I understand it is that you have two nodes: #1 was deployed by Juju itself, and #2 is in the Ready state.
<jtv> Servers will PXE-boot rather a lot... it depends on the situation.  *During deployment* there should be one reboot, from the install image into the installed system.
<jesk> yes, but both off
<jtv> Is it possible that the wake-on-LAN simply didn't come through?  Again, things are much better in 1.7, but in 1.5 the server just wouldn't notice.
<jesk> wake on lan works, i dont get how a system is installed at all. What I saw is that servers boot two times from PXE, but dont install a full blown OS
<jesk> and doesnt matter what I do they dont come up with a plain OS boot
<jesk> they always boot something from PXE which ends in a shutdown after that
<jtv> I wonder if maybe you don't have the boot images you need...
<jesk> so the goal ist that MAAS would install the image I gave the node via MAAS frontend
<jtv> Well you wouldn't have to provide an image; MAAS downloads those by itself.
<jesk> and it would install and boot it similar to installation from CD
<jesk> ending in terminal prompot
<jtv> Well, login prompt.  :)
<jesk> yes :)
<jesk> however it finds out charsets, language, time zone, disk partitions...
<jtv> Let me just summarise what phases these pxe-boots go through:
<jtv> First you "enlist" nodes â usually simply by turning them on and letting them netboot off the MAAS server.
<jtv> They then register their existence with MAAS.
<jtv> Then, you tell MAAS that you want to "commission" them.
<jtv> MAAS boots them up, but into an ephemeral image, and builds an inventory of the node's hardware.
<jtv> After this step, a node is Ready.
<jtv> If you got to this point, that should mean that basic things like netbooting already work.
<jtv> I do believe that ILO has some IPMI quirks, but if you're using wake-on-LAN, I don't think those would affect you.
<jesk> (the wakeonlan package isnt installed by dependencies btw, had to manually do this)
<jesk> ok those steps were all done I guess, the server registered, I see their MACs, and both were already in the ready-state
<jesk> but what after that
<jtv> If you go to a Ready node's UI page, there are buttons to allocate and start the node.
<jtv> (You may want to log in as a non-admin user to hide the atypical steps for now)
<jtv> Did you upload your SSH public key?
<jesk> yes
<jtv> Then the Start button should boot the node into the installer.
<jesk> when I press "start node" what will happen?
<jesk> Ã¬t boots again from PXE?
<jtv> Yes, into an installer.
<jesk> ah ok
<jtv> That then installs the OS (which is always Ubuntu in 1.5).
<jtv> (If you edit the node you can select a different release.)
<jtv> When the installer is done, it reboots the node.
<jtv> At that point the node should come back up, into the OS that was just installed  â with your SSH keys on it.
<jesk> and this last step can also be managed by juju for example?
<jtv> Yes.
<jtv> When you ask Juju to start a unit, it allocates and starts that node.
<jtv> (It provides some custom data to install the charm you want, of course.)
<jtv> Once you have the node up and running, it's utterly yours.  You can reboot it, mess with the OS, etc.  Just don't disable PXE-boot or it will be hard for MAAS to manage after you release it.  :)
<jesk> thanks so far, jtv
<jtv> Juju has a very cloud-y view of machines, so it will tend to think of machines as things you start up once, use for as long as you need it, and then discard.
<jtv> np.
<jesk> i will play a little more
<jtv> OK.  Let me know when you want to tackle the Mystery of the Phantom Server.
<jesk> of what? :D
<jtv> Yeah, this analogy isn't working very well.  These titles usually complain about dead people/ships/animals acting as if they're alive, not the other way around.
<jesk> i would really like to install the whole openstack magic on box for now
<jesk> and have like 6 nodes for storage and computing, just for the possibilities in my lab
<jesk> one openstack box
<jtv> One small caveat: MAAS can manage VMs, but it doesn't create them.
<jesk> but unfortunately the guides install like 6 openstack servers for having 2 compute and 2 storage nodes
<jesk> ok thanks for the hint
<jesk> interesting OS was installed
<jesk> but cant login, SSH pubkey of my user doesnt work
<jesk> :D
<jesk> wonder how juju handled that...
<jesk> muha... my mistake sorry for the spam... user ubuntu ...
<jtv> :)
 * jtv steps out for a break
<jesk> it's a bit of shame that if you do an openstack installation like ubuntu guides suggest that on all edges and corners things dont work as explained
<jesk> i would expect that for foreign howtos, but from the distribution itself
<lutostag> jesk: where at? I'd like to fix that if possible?
<jesk> just take one of the manuals about installing openstack, this is really not a flame... i'am now trying for few days to install it in all kinds of variants... without success
<jesk> next error happened right now:
<jesk> 2014-11-07 14:56:27 ERROR juju.cmd supercommand.go:305 gomaasapi: got error back from server: 401 OK (Expired timestamp: given 1415372187 and now 1415381494 has a greater difference than threshold 300)
<jesk> its a mess
<roaksoax> rvba: ^^
<jesk> you need a lot of clue of all components, maybe then its possible to install that stuff, but then please no big marketing about "openstack installation from canonical step by step guides"
<roaksoax> jesk: what guides are you using?
<jesk> i tried all I could found :D
<jesk> maas guides, juju guides, openstack guides
<jesk> all from the ubuntu doc archive, and also foreign stuff
<roaksoax> jesk: like?
<jesk> what do you mean?
<jesk> like that https://insights.ubuntu.com/2014/05/21/ubuntu-cloud-documentation-14-04lts/
<jesk> or just the openstack-install package
<roaksoax> jesk: http://insights.ubuntu.com/wp-content/uploads/UCD-latest.pdf?utm_source=Ubuntu%20Cloud%20documentation%20%E2%80%93%2014.04%20LTS&utm_medium=download+link&utm_content= that's what you need to follow
<roaksoax> maybe you just run into a bug with juju
#maas 2014-11-08
<Scroll_Tro0L> Heya, if I wanted to share a log file, where do you guys prefer my pasting it?
<jesk> funny when using cloud install single installation it crashs (like every other type of installation too) with :
<jesk> Exception: There was a problem running (sudo -H -u jesk TERM=xterm256-color ssh -t -q -l ubuntu -o "StrictHostKeyChecking=no" -o "UserKnownHostsFile=/dev/null" -i /home/jesk/.ssh/id_rsa 10.0.3.90 mkdir -p ~/.cloud-install) in the container (uoi-bootstrap:10.0.3.90) Error: Command 'sudo -H -u jesk TERM=xterm256-color ssh -t -q -l ubuntu -o "StrictHostKeyChecking=no" -o "UserKnownHostsFile=/dev/null" -i /home/jesk/.ssh/id_rsa 10.0.3.90 mkdir -p ~/.cloud-install' 
<jesk> the funny thing is: why the hell 10.0.3.90? Why are there foreign IP literals hardcoded....
<jesk> and you never never can reuse openstack-install, because of there will be files which wont be overwritten but instead openstack-install crashs
<jesk> doing apt-get purge openstack wont help
<jesk> its a mess :D
<jesk> roaksoax: i just tried the guide you recommended and did really every step like described
<jesk> when doing "juju quickstart" following encounts:
<jesk> juju-quickstart: error: cannot use the maas environment:
<jesk> file not found in the specified path
<jesk> ERROR subprocess encountered error code 2
<jesk> sure, its because rsa pubkey of ssh are not in authorized_keys
<jesk> next problem: juju bootsrap --upload-tools --debug
<jesk> node boots but fails installation
<jesk> because of: < jesk> sure, its because rsa pubkey of ssh are not in authorized_keys
<jesk> argh
<jesk> because of: Stderr: u'lsblk: /dev/cciss!c0d1: not a block device\n'
<jesk> all HP servers have /dev/cciss/c0d* block devices
<jesk> again something stupid hardcoded
<jesk> i give up
<jesk> ok fixed that with patching curtin :-/
<jesk> now the ssh login fails without reason
<jesk> lulz
<jesk> pretty much this issue https://bugs.launchpad.net/juju-core/+bug/1314682
<ubot5> Ubuntu bug 1314682 in juju-core "Bootstrap fails because of virt-manager config" [Medium,Triaged]
<jesk> hm no, something different... sshd log from node says "juju error: Could not load host key: /etc/ssh/ssh_host_ed25519_key"
<jesk> strange, as when using the same ssh command it works for me
<jesk> and also 2014-11-08 11:08:23 ERROR juju.cmd supercommand.go:323 waited for 30m0s without being able to connect: /var/lib/juju/nonce.txt contents do not match machine nonce
<jesk> ok will spam you now
<jesk> just wanted to show that those guides are really silly
<jesk> *stop spam you :D
#maas 2015-11-02
<mup> Bug #1512187 opened: ability to create bridges on MAAS nodes for use without Juju <MAAS:New> <https://launchpad.net/bugs/1512187>
<mup> Bug #1512187 changed: ability to create bridges on MAAS nodes for use without Juju <MAAS:New> <https://launchpad.net/bugs/1512187>
<mup> Bug #1512187 opened: ability to create bridges on MAAS nodes for use without Juju <MAAS:New> <https://launchpad.net/bugs/1512187>
<gringo_> hello?
<mup> Bug #1512297 opened: Revision of cancel and save buttons while renaming a node <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1512297>
<mup> Bug #1512297 changed: Revision of cancel and save buttons while renaming a node <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1512297>
<mup> Bug #1512297 opened: Revision of cancel and save buttons while renaming a node <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1512297>
* You're now known as ubuntulog2
#maas 2015-11-03
<Javi123> Hello there. glad to join all of you.
<Javi123> do you know if it is possible to install Maas in LXD container?
<gor> hello
<gor> someone has a tutorial on how to install maas to virtualbox?
<mup> Bug #1512742 opened: Mouseover an item in Storage/Available Disks and Partitions causes filesystem to disappear <MAAS:New> <https://launchpad.net/bugs/1512742>
<mup> Bug #1512742 changed: Mouseover an item in Storage/Available Disks and Partitions causes filesystem to disappear <MAAS:New> <https://launchpad.net/bugs/1512742>
<mup> Bug #1512742 opened: Mouseover an item in Storage/Available Disks and Partitions causes filesystem to disappear <MAAS:New> <https://launchpad.net/bugs/1512742>
<mup> Bug #1512742 changed: Mouseover an item in Storage/Available Disks and Partitions causes filesystem to disappear <ui> <MAAS:Invalid by ricgard> <https://launchpad.net/bugs/1512742>
<test-user> So a question if I enlist a chassis (Virsh/libvirt) to maas, and it has the proer creds, can it then spin up kvmguests on said "Chassis" or do I have to always create my KVM guests manually, and set them to pxe mode to but auto discovered and then enlisted?
<SpeeR> I'm setting up a 10 blade maas setup. I hate to burn a blade for the cluster, what are the thoughts in running the cluster in qemu, or on vcenter along side the cluster?
<mup> Bug #1512742 opened: Mouseover an item in Storage/Available Disks and Partitions causes filesystem to disappear <ui> <MAAS:Invalid by ricgard> <https://launchpad.net/bugs/1512742>
<mup> Bug #1512765 opened: inconsistent edit mechanisms and various UI issues on Node details page <ui> <ux> <MAAS:New> <https://launchpad.net/bugs/1512765>
<pmatulis> in general i feel that networking stuff in the 'Clusters' tab can be consolidated with stuff under the 'Networks' tab. it's quite confusing
<pmatulis> (MAAS GUI)
<mup> Bug #1512820 opened: deployment count on images tab is double for 14.04 <MAAS:New> <https://launchpad.net/bugs/1512820>
<mup> Bug #1512825 opened: Deployment of Wily from Release Stream fails because of Cloud-Init <MAAS:New> <https://launchpad.net/bugs/1512825>
<mup> Bug #1512832 opened: maasserver.api.tests.test_fannetworks.TestFanNetworksAPI.test_read fails randomly <MAAS:Triaged> <https://launchpad.net/bugs/1512832>
<mup> Bug #1512832 changed: maasserver.api.tests.test_fannetworks.TestFanNetworksAPI.test_read fails randomly <MAAS:Triaged> <https://launchpad.net/bugs/1512832>
<mup> Bug #1512832 opened: maasserver.api.tests.test_fannetworks.TestFanNetworksAPI.test_read fails randomly <MAAS:Triaged> <https://launchpad.net/bugs/1512832>
<mup> Bug #1512857 opened: RAID 5 deployment on 3 disks fails <storage> <curtin:Confirmed> <MAAS:Triaged> <https://launchpad.net/bugs/1512857>
<mup> Bug #1512857 changed: RAID 5 deployment on 3 disks fails <storage> <curtin:Confirmed> <MAAS:Triaged> <https://launchpad.net/bugs/1512857>
<mup> Bug #1512857 opened: RAID 5 deployment on 3 disks fails <storage> <curtin:Confirmed> <MAAS:Triaged> <https://launchpad.net/bugs/1512857>
<marka13> got a question.  If you previously deployed a node.  Delete it and re-add it.  How to you remove it from DNS and the static IP pool?
<marka13> It doesn't seem like it wants to be re-commissioned.  During commissioning it boots and says operating system not found.
<mup> Bug #1512882 opened: Unable to enlist machines with 2 nics connected <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1512882>
<mup> Bug #1512885 opened: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512885 changed: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512885 opened: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512885 changed: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512885 opened: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512885 changed: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:Invalid> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512890 opened: Way too complicated workflow to remove default LVM partition scheme on 1.9 <MAAS:New> <https://launchpad.net/bugs/1512890>
<mup> Bug #1512891 opened: MAAS storage partitioning does not allow formatting as Swap <MAAS:New> <https://launchpad.net/bugs/1512891>
<mup> Bug #1512890 changed: Way too complicated workflow to remove default LVM partition scheme on 1.9 <MAAS:New> <https://launchpad.net/bugs/1512890>
<mup> Bug #1512891 changed: MAAS storage partitioning does not allow formatting as Swap <MAAS:New> <https://launchpad.net/bugs/1512891>
<mup> Bug #1512885 opened: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:Invalid> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512885 changed: Unable to complete juju env bootstrap, connection time out. <cdo-qa> <MAAS:Invalid> <https://launchpad.net/bugs/1512885>
<mup> Bug #1512890 opened: Way too complicated workflow to remove default LVM partition scheme on 1.9 <MAAS:New> <https://launchpad.net/bugs/1512890>
<mup> Bug #1512891 opened: MAAS storage partitioning does not allow formatting as Swap <MAAS:New> <https://launchpad.net/bugs/1512891>
#maas 2015-11-04
<eternal_> Hi, is there a fix for the no-such-image boot problem?  I've been looking around but can't seem to find a solution.  Each time I PXE boot my machines say Loading ubuntu/amd64/generic/trusty/no-such-image/boot-kernel which fails.
<mup> Bug #1491831 changed: maas-cli should allow to specify a cluster controller for a new node <MAAS:Expired> <https://launchpad.net/bugs/1491831>
<mup> Bug #1491831 opened: maas-cli should allow to specify a cluster controller for a new node <MAAS:Expired> <https://launchpad.net/bugs/1491831>
<mup> Bug #1491831 changed: maas-cli should allow to specify a cluster controller for a new node <MAAS:Expired> <https://launchpad.net/bugs/1491831>
<mup> Bug #1491831 opened: maas-cli should allow to specify a cluster controller for a new node <MAAS:Expired> <https://launchpad.net/bugs/1491831>
<mup> Bug #1491831 changed: maas-cli should allow to specify a cluster controller for a new node <MAAS:Expired> <https://launchpad.net/bugs/1491831>
<mup> Bug #1512959 opened: MAAS should not offer EXT3, rather VFAT, EXT2, EXT4 <MAAS:New> <https://launchpad.net/bugs/1512959>
<mup> Bug #1512959 changed: MAAS should not offer EXT3, rather VFAT, EXT2, EXT4 <MAAS:New> <https://launchpad.net/bugs/1512959>
<mup> Bug #1512959 opened: MAAS should not offer EXT3, rather VFAT, EXT2, EXT4 <MAAS:New> <https://launchpad.net/bugs/1512959>
<mup> Bug #1512891 changed: MAAS storage partitioning does not allow formatting as Swap <MAAS:Won't Fix> <https://launchpad.net/bugs/1512891>
<mup> Bug #1512891 opened: MAAS storage partitioning does not allow formatting as Swap <MAAS:Won't Fix> <https://launchpad.net/bugs/1512891>
<mup> Bug #1512891 changed: MAAS storage partitioning does not allow formatting as Swap <MAAS:Won't Fix> <https://launchpad.net/bugs/1512891>
<mup> Bug #1512891 opened: MAAS storage partitioning does not allow formatting as Swap <MAAS:Won't Fix> <https://launchpad.net/bugs/1512891>
<mup> Bug #1512891 changed: MAAS storage partitioning does not allow formatting as Swap <MAAS:Won't Fix> <https://launchpad.net/bugs/1512891>
<mup> Bug #1463176 changed:  modprobe hpvsa fails after installing hpvsa <hp> <hpvsa> <MAAS:Invalid> <https://launchpad.net/bugs/1463176>
<mup> Bug #1512825 changed: Deployment of Wily from Release Stream fails because of Cloud-Init <MAAS:Won't Fix> <maas-images:New> <https://launchpad.net/bugs/1512825>
<mup> Bug #1512890 changed: Way too complicated workflow to remove default LVM partition scheme on 1.9 <MAAS:New> <https://launchpad.net/bugs/1512890>
<mup> Bug #1501112 changed: Node Networking - Users can edit network info while commissioning <MAAS:Invalid by blake-rouse> <https://launchpad.net/bugs/1501112>
<devop> HELP
<devop> Hello
<devop> Somepeople
<devop> can help me?
<roaksoax> !ask
<roaksoax> !question
<roaksoax> !help
<mup> roaksoax: Run "help <cmdname>" for details on: bug, contrib, echo, help, infer, login, poke, register, run, sendraw, sms
<roaksoax> argh
<roaksoax> devop: plase ask your question and if someone knows the answer they'll help
<devop> help
<devop> I have one problem with the nodes
<devop> not working
<devop> i ready every documentation in internet
<devop> but my nodes not working yet
<devop> can you help me?
<roaksoax> devop: TBH, with the information you've provided I cannot help
<roaksoax> devop: if you were to expand what's actually not working, I'd be able to know whether I can help
<devop> the problem is 	Failed commissioning
<devop> i installed everything
<roaksoax> devop: there are many reasons why it could fail
<roaksoax> devop: 1. what version of MAAS
<roaksoax> 2. what type of machine is it
<devop> the last
<roaksoax> 3. can MAAS power control these machines ?
<devop> my machine is one dell power edge r220
<roaksoax> devop: i dunno what you mean with last, it could be last one in tursty, (1.7), last one stable in ppa (1.8), last one in development (1.9)
<roaksoax> devop: can MAAS power control them? as in, whne you click commission, do they actually turn on?
<devop> this is MAAS Version 1.8.3+bzr4053-0ubuntu1 (tru
<devop> not
<roaksoax> devop: so MAAS cannot power control them?
<devop> really
<devop> can not power
<roaksoax> devop: how did you add the machines to MAAS? via WebUI ?
<devop> yes via web
<devop> i created two nodes
<roaksoax> devop: ok, go to the node details page, and on the top, there's a button that says "check power"
<roaksoax> what's the result of that?
<devop> one the power type wake on lan and the other vish
<roaksoax> devop: ah!
<roaksoax> devop: then that's probably the problem
<roaksoax> devop: if it is a dell, doesn't it have IPMI ?
<devop> i used the ubuntu image personalized for dell power edge r220
<devop> ok one moment
<devop> i ll try
<devop> this the log
<devop> Powering node on	Wed, 04 Nov. 2015 13:32:03 Node changed status - From 'New' to 'Commissioning'
<devop> comissioning... but not ready
<devop> the status is commisioning always
<devop> 3 days looking a solution for this problem
<devop> I need nodes ready for install openstack and juju
<devop> failed now
<devop> Failed to power on node - Timed out	Wed, 04 Nov. 2015 13:37:03 Node changed status - From 'Commissioning' to 'Failed commissioning'	
<devop> Andres i m spanish too
<devop> where are you from
<devop> que desesperacion
<devop> somepeople can help me?
<marka13> Question: I have landscape juju and maas on one machine.  When I perform an openstack install, it gets to the very end of the installation where it is going to import images.  But something is changing the IP of eth0 and I am thinking that is what is failing the last step
<marka13> anyone seen that behavior before?
<marka13> twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone
<marka13> I can't seem to find in the logs where the ip was changed, the interfaces file is 10.0.0.9, but for some reason the ip got changed to 0.26
<marka13> weird
<mup> Bug #1511713 changed: udev rules not updated to reflect MAC change in node <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1511713>
<mup> Bug #1513085 opened: Partitioning should align for performance <curtin:New> <MAAS:New> <https://launchpad.net/bugs/1513085>
<mup> Bug #1513095 opened: MAAS should prevent deploying nodes with PXE interface 'unconfigured' (or all NIC's as 'Unconfigured') <networking> <ui> <MAAS:Triaged> <https://launchpad.net/bugs/1513095>
<marka13> no one around?
<MilesDenver> no one
<marka13> lol
<marka13> Just trying to figure out what had happened
<mup> Bug #1513111 opened: When a bond is created all IP address associated with the bond members should be removed <networking> <MAAS:Triaged> <https://launchpad.net/bugs/1513111>
<mup> Bug #1513111 changed: When a bond is created all IP address associated with the bond members should be removed <networking> <MAAS:Triaged> <https://launchpad.net/bugs/1513111>
<mup> Bug #1513111 opened: When a bond is created all IP address associated with the bond members should be removed <networking> <MAAS:Triaged> <https://launchpad.net/bugs/1513111>
<mup> Bug #1512029 changed: maasserver.models.tests.test_interface:TestReleaseAutoIPs.test__calls_update_host_maps_for_next_ip_managed_subnet randomly fails <tech-debt> <tests> <MAAS:Invalid> <https://launchpad.net/bugs/1512029>
<mup> Bug #1513198 opened: maas needs to install wsmancli as a dependency <MAAS:New> <https://launchpad.net/bugs/1513198>
<mup> Bug #1513198 changed: maas needs to install wsmancli as a dependency <MAAS:New> <https://launchpad.net/bugs/1513198>
<mup> Bug #1513198 opened: maas needs to install wsmancli as a dependency <MAAS:Won't Fix> <https://launchpad.net/bugs/1513198>
<mup> Bug #1513198 changed: maas needs to install wsmancli as a dependency <MAAS:Won't Fix> <https://launchpad.net/bugs/1513198>
<mup> Bug #1513198 opened: maas needs to install wsmancli as a dependency <MAAS:New> <https://launchpad.net/bugs/1513198>
<mup> Bug #1513198 changed: maas needs to install wsmancli as a dependency <MAAS:New> <https://launchpad.net/bugs/1513198>
<mup> Bug #1513214 opened: Unable to remove bond in the UI <networking> <ui> <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1513214>
<mup> Bug #1513224 opened: 1.9b2: node details storage displays big json blob <landscape> <MAAS:New> <https://launchpad.net/bugs/1513224>
<mup> Bug #1513214 changed: Unable to remove bond in the UI <networking> <ui> <MAAS:Invalid by ltrager> <https://launchpad.net/bugs/1513214>
<mup> Bug #1513258 opened: CSS Broken for Bond Network Device <networking> <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1513258>
#maas 2015-11-05
<mup> Bug #1513271 opened: Unable to unmount a filesystem in the UI <storage> <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1513271>
<nitin_> hello
<nitin_> i have set up a MAAS server with 2 nodes as ready
<nitin_> but failing while running "juju bootstrap"
<nitin_> can anybody help me?
<mup> Bug #1513373 opened: Cannot modify the default VLAN for a fabric <MAAS:New> <https://launchpad.net/bugs/1513373>
<mup> Bug #1513379 opened: Cannot unset class_type for a fabric once set. <MAAS:New> <https://launchpad.net/bugs/1513379>
<mup> Bug #1513391 opened: unexpected hover on subnets page <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1513391>
<mup> Bug #1513392 opened: radiobuttons should be orange <ui> <MAAS:Triaged by ricgard> <https://launchpad.net/bugs/1513392>
<mup> Bug #1513373 changed: Cannot modify the default VLAN for a fabric <MAAS:Invalid> <https://launchpad.net/bugs/1513373>
<mup> Bug #1513373 opened: Cannot modify the default VLAN for a fabric <MAAS:Invalid> <https://launchpad.net/bugs/1513373>
<mup> Bug #1513373 changed: Cannot modify the default VLAN for a fabric <MAAS:Invalid> <https://launchpad.net/bugs/1513373>
<mup> Bug #1513413 opened: IP assignment for devices inconsistent with interfaces on nodes <MAAS:New> <https://launchpad.net/bugs/1513413>
<mup> Bug #1513432 opened: Cannot unset a space for a subnet once set <MAAS:New> <https://launchpad.net/bugs/1513432>
<mup> Bug #1513432 changed: Cannot unset a space for a subnet once set <MAAS:New> <https://launchpad.net/bugs/1513432>
<mup> Bug #1513432 opened: Cannot unset a space for a subnet once set <MAAS:New> <https://launchpad.net/bugs/1513432>
<mup> Bug #1513432 changed: Cannot unset a space for a subnet once set <MAAS:Invalid> <https://launchpad.net/bugs/1513432>
<mup> Bug #1513379 changed: Cannot unset class_type for a fabric once set. <MAAS:Invalid> <https://launchpad.net/bugs/1513379>
<mup> Bug #1513485 opened: Commissioning doesn't update the IP of PXE interface UI, maybe due to lease parser failure <MAAS:Triaged> <https://launchpad.net/bugs/1513485>
<mup> Bug #1513506 opened: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513506>
<mup> Bug #1513507 opened: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513507>
<mup> Bug #1513506 changed: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513506>
<mup> Bug #1513507 changed: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513507>
<mup> Bug #1513506 opened: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513506>
<mup> Bug #1513507 opened: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513507>
<stokachu> urthmover:
<stokachu> this is what i have so far
<stokachu> [15:04] <mpontillo> stokachu: what do you mean when you say "inside" vcenter? yes, you can host the MAAS server on VMware - as long as it has the connectivity it needs to the clusters
<stokachu> [15:04] <mpontillo> stokachu: if you're talking about "add hardware > chassis", it should work with 1.8 too
<stokachu> [15:05] <mpontillo> stokachu: the thing you need to be careful about is to make sure the VMware user that MAAS uses only has permission to access the nodes you want to make into MAAS-managed nodes. (or use the prefix filter option to restrict it based on the VM name.)
<stokachu> [15:06] <mpontillo> stokachu: that should be fine
<stokachu> [15:06] <mpontillo> stokachu: I just wanted to give you that word of warning, because when you do probe-and-enlist (aka "Add Hardware > Chassis"), MAAS will reconfigure any VMs it finds through the VMware API so that they PXE boot
<stokachu> urthmover: ^
 * mpontillo didn't realize there was a discussion going on here, sorry =)
<stokachu> mpontillo: nah it started in #juju
<urthmover> stokachu: ok I read that on the docs or wiki yesterday about chassis taking over every guest
<urthmover> thank you stokachu a bunch
<stokachu> urthmover: np
<mpontillo> urthmover: ping me if you have questions.
<urthmover> mpontillo: in chan or priv?
<urthmover> mpontillo: chan is fine with me :)
<mpontillo> urthmover: yeah keep it in #maas please so stokachu or others can jump in if needed
<stokachu> urthmover: if you have other questions with the OpenStack Installer you can ping me or mmcc (he's in #ubuntu-server)
<urthmover> mpontillo: At this point I do not have cli or api access to the vmware vcenter server.  I do have vcenter access using the fatclient.  I am able to build guests, mount disks, etc.  How do I handle the power aspects of using MAAS with my seemly limited access to the hypervisor?
<mpontillo> urthmover: hm, it's weird that the fat client would work but not an API client; I thought it used the same API
<urthmover> mpontillo: currently I've built a maas/juju environment locally using vmware fusion.  I've had topower on the guests manually throughout the process of installing everything
<mpontillo> urthmover: unfortunately, API access is a requirement if you want to use the VMware power control driver. VMware fusion, unfortunately, doesn't work - it does not provide the API. the cheapest version of VMware that supports it is VMware Workstation
<mpontillo> urthmover: although I think you could convince the free version of ESXi to work (even without vSphere, etc), given enough time/effort
<urthmover> mpontillo: it is wierd, but the hosting provider has attempted to curb my permissions on the environment so that they feel they have some sort of control, to make their support easier.  The dumb part is we are the only tenants on this cluster (but that's another story)
<urthmover> mpontillo: if I am unable to gain access to the api, do you think I should build a KVM cluster inside the vmware environment, then build MAAS on top of that...etc
<mpontillo> urthmover: well, I don't understand how you could use the fat client to turn on/off machines, but the API wouldn't work. it should be making the same calls. I'd file a support ticket?
<mpontillo> urthmover: you could do it that way too. I assume this is just for demo purposes then?
<mpontillo> urthmover: if you want to try ESXi, I found this blog post helpful - http://blog.mywarwithentropy.com/2013/12/configuration-esxi-55-from-command-line.html
<urthmover> mpontillo: this is testing, that will eventually become production ....I think I'll go down the path of testing api access and demanding it before I clutter up this design with too many embedded hypervisors
<mpontillo> urthmover: also, regarding ESXi, I have these commands written in my notes: http://paste.ubuntu.com/13115403/ (I was trying to figure out how to create additional networks for the VMs to use using "bare ESXi")
<urthmover> mpontillo: thank you I'll work with those commands and the syntax to see if I have some sort of api access
<mpontillo> urthmover: the bottom line for me is, if you can point an official VMware client (such as adding it as a server in Fusion), MAAS should work with it.
<urthmover> mpontillo: I hear that and I'm keeping my fingers crossed that I can....give me a few to test and I'll update you and the chan
<urthmover> mpontillo: thank you for your time so far btw
<mpontillo> urthmover: thanks, good luck.
 * urthmover scampers off
<mup> Bug #1513506 changed: Storage table row interaction <ui> <MAAS:New for ricgard> <https://launchpad.net/bugs/1513506>
#maas 2015-11-06
<mup> Bug #1513695 opened: setting up dhcp on second interface with cidr ip/21 failed <MAAS:New> <https://launchpad.net/bugs/1513695>
<nitin> I want to import custom boot images in MAAS. How can I do it?
<nitin> Please help me with this
<mup> Bug #1513775 opened: MAAS didn't parse dnssec-validation automatically <MAAS:New> <https://launchpad.net/bugs/1513775>
<mup> Bug #1513789 opened: In regiond, several start-up tasks are done concurrently with other Twisted service starts <MAAS:New> <https://launchpad.net/bugs/1513789>
<mup> Bug #1513789 changed: In regiond, several start-up tasks are done concurrently with other Twisted service starts <MAAS:New> <https://launchpad.net/bugs/1513789>
<mup> Bug #1513789 opened: regiond gets wedged when restarted rapidly <MAAS:Triaged> <https://launchpad.net/bugs/1513789>
<lborda> hi guys I am trying maas 1.9.0 (beta2) in a development environment, do the subnets can only be added through the cli now ?
<roaksoax_> lborda: for now yes
<roaksoax_> lborda: 1.9.1 will fix that though
<lborda> cool tks roaksoax_
#maas 2015-11-07
<mup> Bug #1514045 opened: MAAS sometime install fails if mutiple DHCP are confuigued in MAAS <MAAS:New> <https://launchpad.net/bugs/1514045>
<mup> Bug #1514094 opened: bcache setup fails in gmaas <MAAS:New> <https://launchpad.net/bugs/1514094>
<mup> Bug #1514094 changed: bcache setup fails in gmaas <MAAS:New> <https://launchpad.net/bugs/1514094>
<mup> Bug #1514094 opened: bcache setup fails in gmaas <MAAS:New> <https://launchpad.net/bugs/1514094>
#maas 2016-11-07
<mup> Bug #1616201 changed: MAAS 2: database notification failure causes out-of-date DNS <landscape> <MAAS:Expired> <https://launchpad.net/bugs/1616201>
<mup> Bug #1639766 opened: [SRU] MAAS 2.1.1 <maas (Ubuntu):New> <https://launchpad.net/bugs/1639766>
<mup> Bug #1639766 changed: [SRU] MAAS 2.1.1 <maas (Ubuntu):New> <https://launchpad.net/bugs/1639766>
<mup> Bug #1639766 opened: [SRU] MAAS 2.1.1 <maas (Ubuntu):New> <https://launchpad.net/bugs/1639766>
<mup> Bug #1639769 opened: error when non-existent token_key provided to delete_authorisation_token could be clarified <error-surfacing> <MAAS:New> <https://launchpad.net/bugs/1639769>
<mup> Bug #1639769 changed: error when non-existent token_key provided to delete_authorisation_token could be clarified <error-surfacing> <MAAS:New> <https://launchpad.net/bugs/1639769>
<mup> Bug #1639769 opened: error when non-existent token_key provided to delete_authorisation_token could be clarified <error-surfacing> <MAAS:New> <https://launchpad.net/bugs/1639769>
<kiko> morning
<mup> Bug #1639839 opened: Error message when giving an invalid type as an argument to boot-resources GET is not clear <error-surfacing> <MAAS:New> <https://launchpad.net/bugs/1639839>
<UEBZJmHZgQUzuMmt> https://www.youtube.com/watch?v=3EsJLNGVJ7E & https://wikileaks.org/podesta-emails/emailid/15893, https://wikileaks.org/podesta-emails/emailid/23561, http://www.reuters.com/article/us-usa-election-foundation-idUSKBN12Z2SL & https://wikileaks.org/podesta-emails/emailid/3774 (ctrl+f qatar) - please don't let these be buried
<kiko> seriously
<brendand> thanks uuid man
<Braven> hello
<Braven> how is everyone
<roaksoax> /w/win 10
<kiko> hey Braven
<kiko> sorry for the delay
<kiko> what's cooking?
<Braven> what is different between zones and spaces
<kiko> Braven, zone are AZs
<kiko> Braven, they are only useful as a way to group nodes
<kiko> which something that uses the API can query on
<kiko> spaces are collections of subnets
<Braven> so a space would be like a datacenter
<kiko> collections of IP subnets
<kiko> if that makes it clearer
<Braven> we just updated to from 1.7 to 1.9
<Braven> so a lot of thing are different
<Braven> what is fabric?
<Braven> kiko: what is fabric
<kiko> a fabric is a collection of L2 segments
<kiko> Braven, https://insights.ubuntu.com/?p=52900 and http://maas.io/docs/en/intro-concepts
<Braven> can static IP be assign though the maas web interface
<kiko> Braven, yes, definitely -- just change from DHCP to static and specify it on a node edit page
<Braven> Since I upgrade to MAAS 1.9 a lot of nodes have network and storage info.  How can we get the info into maas
<mup> Bug #1639332 opened: Expire the DHCP lease after commissioning or make the  default-lease-time configurable <MAAS:Incomplete by andreserl> <https://launchpad.net/bugs/1639332>
<mup> Bug #1639930 opened: initramfs network configuration ignored if only ip6= on kernel command line <maas-ipv6> <cloud-init:Confirmed> <MAAS:New> <cloud-init (Ubuntu):Confirmed> <https://launchpad.net/bugs/1639930>
<jrf> Hey....Im wondering if it makes sense to use maas for multiple SMB's. I'm a small service provider with several customers in a small town. I have been doing all windows on hyper-v core, and would like to offer remote replication and DR ops as well as email from a datacenter location--but I'd like to integrate my onsite hw mgt
#maas 2016-11-08
<gaurangt> Hi, I'm using MAAS in KVM virtual environment. When I try to deploy the VMs with xenial image, the cloud-init job tries to update the kernel and it is very very slow. Any idea on what might be going wrong?
<blahdeblah> gaurangt: More than likely it's a slow connection between you and archive.ubuntu.com/security.ubuntu.com; you can specify a local mirror in the MAAS settings.
<gaurangt> yeah.. I've tried that as well. I guess the problem is not with downloading packages. Unpacking and installing them is very slow.
<gaurangt> blahdeblah: http://paste.ubuntu.com/23445514/
<gaurangt> this takes a long time
<gaurangt> Is there any way to increase deployment timeout in MAAS?
<gaurangt> one other observation - The problem occurs when I deploy more machines. Timeout doesn't occur for a single machine deployment.
<blahdeblah> gaurangt: Other ways you can speed it up are using bcache, and (if you're just using the KVM machines for testing and don't care if they break) using cache=unsafe for your VMs
<gaurangt> where should I specify these settings?
<blahdeblah> bcache is done when you install the bare metal machine - you need an SSD to cache your spinning rust HDs
<blahdeblah> cache=unsafe is done when you run virt-install for the VM
<gaurangt> ok
<gaurangt> I'm using virt-manager for creating the VMs
<gaurangt> i have set cache option as none
<blahdeblah> I have a rather old core 2 duo machine I use for testing, and it's quite speedy enough when I put both bcache and cache=unsafe
<gaurangt> the other value options are default/writeback/writethrough
<mup> Bug #1640147 opened: [2.1] rsyslog is flooded with "no link-local IPv6 address for $IFACE" on commissioning <MAAS:New> <https://launchpad.net/bugs/1640147>
<mup> Bug #1640201 opened: [API] node parameter silently ignored when creating DHCP snippets if global_snippet set to True <maas-cli> <MAAS:New> <https://launchpad.net/bugs/1640201>
<Braven> Hello
<Braven> Can I have maas manage dhcp but set the scope to 0 host available. But then allocate static IP
<mup> Bug #1640259 opened: [2.1 UI] IP Ranges table no longer showing in Subnet Details page. <MAAS:In Progress by newell-jensen> <https://launchpad.net/bugs/1640259>
<roaksoax> /win/win 6
<mup> Bug #1640298 opened: multiple subnets dhcp shared-network does not work with dhcp-relay <MAAS:New> <https://launchpad.net/bugs/1640298>
<nturner> Anyone want more info about the state of https://bugs.launchpad.net/maas/+bug/1640300 before I restart maas?
<roaksoax> nturner: the postgres logs
<nturner> roaksoax: ok, I can attach those
<roaksoax> nturner: thanks
<nturner> ok, attached
<roaksoax> nturner: ty
<roaksoax> nturner: approximately, how many machines do you have ?
<nturner> 36
<roaksoax> nturner: yeah that sounds like something is leaking
<nturner> want any other logs?
<roaksoax> nturner: i think we have an idea of what it may be. So postgres logs + all maas's should be enough
<nturner> ok, I'll tar up and attach /var/log/maas too
<roaksoax> nturner: ty
<mup> Bug #1640300 opened: django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections <MAAS:New> <https://launchpad.net/bugs/1640300>
<mup> Bug #1640301 opened: Deploying servers with multiple disks fails <MAAS:New> <https://launchpad.net/bugs/1640301>
<mup> Bug #1640333 opened: No rack controller available when trying to enable DHCP on subnet on tagged VLAN <MAAS:New> <https://launchpad.net/bugs/1640333>
#maas 2016-11-09
<xme> hi all
<mbeierl> Question: I have MAAS installed and am wanting to manually insert a single server.  Using the web ui, I added hardware, with MAC address of the PXE interface, as well as IPMI functions.  On commissioning, the server powers on and starts PXE.  On MAAS server, using tcpdump, I see the PXE boot requests coming in from MAC address, but MAAS never responds.  What did I do wrong?
<brendand> mbeierl, have you enabled the dhcp server on the right subnet?
<mbeierl> brendand: ah, probably not.  Was following maas guides on line and that was not clear.  Can you point me to a guide for that in 1.9?  Or if it's simple, what do I need to do?
<mbeierl> also, does it become a DHCP server for that subnet by default, or will it only respond to the MAC addresses listed in the hardware?  I need to be careful as this is a shared subnet
<brendand> mbeierl, hmm. no - it's not really recommend to enable dhcp on a subnet with another dhcp server running
<mbeierl> there is not another DHCP running
<mbeierl> but we might want to have two MAAS systems for testing and I want to have mine respond to my servers only.  Is that possible?
<mbeierl> I know from dnsmasq, as an example, that I can restrict replies to certain MACs only, but I don't know if that is the intent of MAAS or not.
<brendand> mbeierl, there might be a way to do that, but i don't know off the top of my head. are you using 1.9 or 2.0 > ?
<mbeierl> 1.9
<brendand> ok
<mbeierl> I found the 1.9 docs for DHCP, I think
<mbeierl> brendand: ah.  I'm acutally using MAAS 2.0.  And I cannot find the installation docs for that online: https://maas.ubuntu.com/docs2.0/
<brendand> mbeierl, http://maas.io/docs/en/?
<mbeierl> brendand: ah.  thanks!
<mbeierl> http://maas.io/docs/en/installconfig-subnets-dhcp <- I think this is what I missed
<mbeierl> brendand: ok, enabled DHCP on the VLAN, but still nothing...
<mbeierl> 07:42:12.293469 90:e2:ba:82:bc:f8 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 590: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 90:e2:ba:82:bc:f8, length 548
<brendand> mbeierl, as long as you have images synced it really ought to work
<mbeierl> brendand: :(  That's what I thought.  Going to dig further to see if I can figure out what else might be going on.  Any log files you can recommend?
<begbert> My fresh MAAS install seems to be hanging on Region importing when I try to import boot images.
<begbert> Last log entry is Nov  9 13:16:32 bme-maas maas.import-images: [INFO] Finished importing boot images, the region does not have any new images.
<Braven> Hello all. Can vlan be used to commission nodes?
<Braven> ??
<begbert> http://askubuntu.com/questions/847544/importing-images-hangs-at-importing-region
<begbert> Well, this was a very helpful use of my time.  Guess I'll go use Foreman or something.
<Braven>  I am trying to add a vip in interface on the maas server
<Braven> it keeps cutting off the :0 of bond0:0
<mbeierl> ok, I am able to start commissioning a node, but it does not appear to be successful.  I am getting HTTP 401 errors: POST /MAAS/metadata/2012-03-01/ HTTP/1.1" 401.  The systems cannot access the internet directly for NTP, so I put an ntp server into the MAAS/settings page, but still no luck
<Braven> hmm
<Braven> how long did you wait?
#maas 2016-11-10
<mup> Bug #1640671 opened: "No rack controllers can access the BMC of node: <node name>" when using OpenStack nova power driver <maas (Ubuntu):New> <https://launchpad.net/bugs/1640671>
<mup> Bug #1640780 opened: Restarting image import leaves boot-resources empty (selection is made but not saved) <MAAS:New> <https://launchpad.net/bugs/1640780>
<Braven> Good morning
<brendand> Braven, hi
<Braven> brendand:how are you
<brendand> Braven, i saw you're question last night but it was too late to help, sorry about that
<brendand> *your
<Braven> brendand: Do vlans work with PXE?
<brendand> Braven, they ought to, can you describe what you're seeing?
<Braven> brendand:The node is failing on commissioning. It is able to boot to PXE. I can see file loading but it at the helper_*.py is having issue
<brendand> Braven, do you have logs?
<Braven> brandand:you are wanting the console logs?
<brendand> Braven, wherever you see the helper_*.py issue
<Braven> brendand: let me try to set the logs up.  So i need to redirect console to a serial over network?
<brendand> Braven, if you're using 2.0 there is actually a syslog in /var/log/maas/rsyslog/
<Braven> brendand:I am still on 1.9.
<brendand> Braven, then no, you would need to get the serial console - or just give a bit more info about the error, exactly what it's saying
<Braven> brendan:How do you redirect to serial console. I am able to connect to the server ipmitool and I am getting output but is there away to save it
<brendand> Braven, you could just copy and paste, or if you wanted to put it in a file you can use tee
<Braven> brendan:okay I will give it a try
<Braven> Does anything in conmissioning script need to be changed?
<brendand> Braven, i need to see what's wrong first
<Braven> so reason it stop writing to the serial
<Braven> brendand:sorry this is taking so long. I am still trying to get you the console log.
<brendand> Braven, np
<Braven> brendand: this is last thing I can see "Loading ubuntu/amd64/generic/trusty/release/boot-initrd........................"
<Braven>  when logging console log :
<brendand> hmmm. that's interesting
<brendand> Braven, which version btw?
<brendand> oh, trusty
<brendand> yeah
<Braven> brendand:So wrote down the error. Url_helper.py[WARNING]: Calling 'http://<subnet gateway>'//
<Braven> latest-data/instance-d (caused by <socket.error'>
<brendand> Braven, and where's that error?
<Braven> errno 111
<Braven> brendand:why is it try to connect my gateway IP and not maas server"
<brendand> Braven, be good to know which log file that's in
<Braven> brendand: I know.
<Braven> brendand: What appication do use for connecting to the console
<Braven> console=tty0 console=ttyS1,57600n8  should I add this to MAAS under setting-> Global Kernel Parameters?
<brendand> Braven, you might need to, yeah
<mwenning> Has anyone seen this problem?   MaaS 2.1, I can enlist and commission OK, but deploy dumps all sorts of error messages about cannot fetch files.   I cannot ssh into the node to see what's wrong, looks like it's not configuring the NIC's
<mup> Bug #1459892 changed: User SSH Key is not editable <MAAS:Invalid by newell-jensen> <https://launchpad.net/bugs/1459892>
<Braven> brendand: I added the our server to our prod deployment network and everything is working now.
#maas 2016-11-11
<Braven> hello
<brendand> hi
<Braven> brendand: We got it working by switching to a direct access port instead of trunked port on the network switch
<Braven> brendand: I would like thank you for help.
<brendand> Braven, oh i didn't do too much
<mup> Bug #1641171 opened: maas 2.1 fails deployment <MAAS:New> <https://launchpad.net/bugs/1641171>
#maas 2016-11-12
<Jonathan___> Hello
#maas 2016-11-13
<farhad2161> When I add a new "Chassis", it shows an error that contains this "CSRF verification failed. Request aborted."
<farhad2161> I need a temperory quick fix.
<farhad2161> Tested in MAAS Version 2.1.0+bzr5480-0ubuntu1 (16.04.1)
#maas 2017-11-06
<mup> Bug #1573982 opened: LVM boot problem - volumes not activated after upgrade to Xenial <lvm2 (Ubuntu):Confirmed> <maas (Ubuntu):New> <https://launchpad.net/bugs/1573982>
<bladernr> Hey, can someone help me debug a commissioning issue w/ MAAS 2.3.0~beta2??
<bladernr> I'm trying to commission a node but commissioning fails because it's getting 401 errors from $MAAS_IP/MAAS/metadata/2012-03-01/
<bladernr> I can access that page via a browser and the message returned is: No authorization header received.
<bladernr> I've seen a few bugs via the goog that have hte same error message, but they all look like they have different causes, so not terribly helpful (that I could determine).
<roaksoax> bladernr: i'd recommend you to upgrade to beta3
<roaksoax> bladernr: the second, check that /etc/maas/rackd.conf points to the rgith IP and it is an IP that the machines can talk to
<bladernr> roaksoax, ok, upgrading first. thanks
<dnegreira> Hi everyone, I am hitting exactly the same issue as a bug that was closed: https://bugs.launchpad.net/maas/+bug/1583093. I do currently have several VLANS on the interface but mtu is 1500. I can see the nodes getting DHCP and I can ping them from the MAAS but I am unable to ssh into the nodes.
<dnegreira> current maas version is 2.2.2-6099-g8751f91-0ubuntu1~16.04.1
<dnegreira> if I  try to ssh from the maas controller towards the nodes I always get a connection reset
<dnegreira> and I am currently unable to login via console as well so its difficult to troubleshoot from a node point of view :/
<roaksoax> dnegreira: the way maas makes a machine deployed is when cloud-init, after booting for the first time to maas, contacts MAAS and gathers the user data
<roaksoax> dnegreira: what could be happening is that your amchine is not contacting maas, hence, making it failed deployment
<roaksoax> dnegreira: are you adding any user data? do you see any errors on regiond.log ?
<dnegreira> I am even embarassed to say what was wrong
<dnegreira> wrong gateway on the vlan definition :)
<dnegreira> sorry for the noise..
<roaksoax> no worries :)
<mup> Bug #1730456 opened: Cannot commission node due to 401 errors fetching metadata. <MAAS:Incomplete> <https://launchpad.net/bugs/1730456>
<mup> Bug #1730474 opened: [2.x] MAAS region startup sequence leads to race conditions <MAAS:Triaged by mpontillo> <https://launchpad.net/bugs/1730474>
<roaksoax> c/win 3
<mup> Bug #1730481 opened: [2.3, HWTv2] When 'other' test fails, node listing incorrectly shows two icons <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1730481>
<mup> Bug #1730485 opened: [2.2+, HWT] badblocks fails with LVM <internal> <MAAS:New> <https://launchpad.net/bugs/1730485>
<mup> Bug # changed: 1418044, 1696122, 1716328, 1718044, 1718776, 1718779, 1721827, 1721887, 1722665, 1723944, 1724402, 1724627, 1727360, 1727576, 1727962, 1728300, 1728302, 1729857, 1729902
<mup> Bug # opened: 1418044, 1696122, 1716328, 1718044, 1718776, 1718779, 1721827, 1721887, 1722665, 1723944, 1724402, 1724627, 1727360, 1727576, 1727962, 1728300, 1728302, 1729857, 1729902
<mup> Bug # changed: 1418044, 1696122, 1716328, 1718044, 1718776, 1718779, 1721827, 1721887, 1722665, 1723944, 1724402, 1724627, 1727360, 1727576, 1727962, 1728300, 1728302, 1729857, 1729902
<mup> Bug #1573982 opened: LVM boot problem - volumes not activated after upgrade to Xenial <MAAS:New> <lvm2 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1573982>
<mup> Bug #1573982 changed: LVM boot problem - volumes not activated after upgrade to Xenial <MAAS:New> <lvm2 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1573982>
<mup> Bug #1573982 changed: LVM boot problem - volumes not activated after upgrade to Xenial <MAAS:New> <lvm2 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1573982>
<mup> Bug #1573982 opened: LVM boot problem - volumes not activated after upgrade to Xenial <MAAS:New> <lvm2 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1573982>
<mup> Bug #1573982 opened: LVM boot problem - volumes not activated after upgrade to Xenial <MAAS:New> <lvm2 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1573982>
<mup> Bug #1730493 opened: MAAS is dropped in grub menu when booting in UEFI mode, and Secure Boot not working  <MAAS:New> <https://launchpad.net/bugs/1730493>
<mup> Bug #1573982 changed: LVM boot problem - volumes not activated after upgrade to Xenial <MAAS:New> <lvm2 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1573982>
<mup> Bug #1730493 changed: MAAS is dropped in grub menu when booting in UEFI mode, and Secure Boot not working  <MAAS:New> <https://launchpad.net/bugs/1730493>
<mup> Bug #1730493 opened: MAAS is dropped in grub menu when booting in UEFI mode, and Secure Boot not working  <MAAS:New> <https://launchpad.net/bugs/1730493>
<mup> Bug #1730524 opened: Failed commissioning leaves machine completely cut off from MAAS <hwcert-server> <MAAS:New> <https://launchpad.net/bugs/1730524>
<mup> Bug #1730525 opened: maas 2.2.2 unable to disk erase artful deployment  <MAAS:New> <https://launchpad.net/bugs/1730525>
<mup> Bug #1730525 changed: maas 2.2.2 unable to disk erase artful deployment  <MAAS:New> <https://launchpad.net/bugs/1730525>
<mup> Bug #1730525 opened: maas 2.2.2 unable to disk erase artful deployment  <MAAS:New> <https://launchpad.net/bugs/1730525>
#maas 2017-11-07
<mup> Bug #1730626 opened: bond-lacp-rate set on any bond type <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1730626>
<dnegreira> I am trying to automate my nodes installation and I have one question.. when deploying a node, if I need to copy a file after the instalation is done, what is the best way to achieve that with MAAS? Lets say its a secret file which I need on the server to run a set of scripts to be ran by curtain with late_commands, is there any way to copy the files from the MAAS controller automagically?
<dnegreira> The only way that I see it is that I need to maintain a custom image repository and have the file already inside of that image
<[Kid]> anyone used maas and juju to deploy kubernetes?
<[Kid]> i am having issues
<roaksoax> [Kid]: what issues?
<roaksoax> dnegreira: you can do that via a late_command or by passing cloud-init user-data
<mup> Bug #1711760 changed: [2.3] resolv.conf is not set (during commissioning or testing) <verification-done> <verification-done-xenial> <MAAS:Fix Released by andreserl> <resolvconf (Ubuntu):Won't Fix> <resolvconf (Ubuntu Xenial):Fix Released> <https://launchpad.net/bugs/1711760>
<mup> Bug #1728309 changed: [2.3] Comissioning didn't remove disks that no longer exist <MAAS:Triaged by blake-rouse> <https://launchpad.net/bugs/1728309>
<mup> Bug # opened: 1730690, 1730691, 1730693, 1730694, 1730697
<mup> Bug #1730703 opened: [2.3rc2, UI] Rename the section "Settings" of machine details  to Configuration <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730703>
<mup> Bug #1730704 opened: [2.3rc2, UX improvement] "Edit" is a vague link label in the Overview card <2.3qa> <ui> <MAAS:Incomplete> <https://launchpad.net/bugs/1730704>
<mup> Bug #1730708 opened: [2.3rc2] Inconsistency between icons/state in machine listing and the hardware test section <2.3qa> <MAAS:New> <https://launchpad.net/bugs/1730708>
<mup> Bug #1730708 changed: [2.3rc2] Inconsistency between icons/state in machine listing and the hardware test section <2.3qa> <MAAS:New> <https://launchpad.net/bugs/1730708>
<din> hello, does anyone have experience with running a Rack Controller in a different VLAN than the one it is supposed to be managing?
<mup> Bug #1431156 changed: When a custom image is added, multiple OS entries are created for it <MAAS:Invalid> <https://launchpad.net/bugs/1431156>
<mup> Bug #1730089 changed: IPMI power cycle on power on <MAAS:Won't Fix> <https://launchpad.net/bugs/1730089>
<roaksoax> din: yes
<roaksoax> din: can you expand into what you are trying to do ?
<din> roaksoax: I have a subnet for mixed VMs and bare metal (where MAAS lives) and a subnet dedicated for baremetal. The Rack Controller is currently set up to serve the mixed subnet, and all of MAAS is on one node
<din> I have the DHCP/PXE requests forwarding fine (tcpdump looks good on that front) but I think the subnet in MAAS is not set up with a rack controller which is why the nodes are not provisioning
<din> roaksoax: it won't let me select the current rack controller (which is probably intended) and I'm unsure exactly how to proceed. The docs seem thin here.
<din>  For the archives, I got it working. You can have multiple subnets in a VLAN. I had a VLAN per subnet. So adding the desired subnet to the VLAN that was associated with the existing Rack Controller made it automagical.
<[Kid]> roaksoax, i am doing a juju deploy and i have 7 "Ready" nodes in MAAS, but juju only spins up 4 of them
<[Kid]> i have pointed it to a YAML that has all 7 machines in it.
<roaksoax> din: they could not be provisioning because the machine may not be able to communicate to the region
<roaksoax> [Kid]: that's probably because of the constraints. What are the hardware details for your machines ? are you using any extra constraints
<roaksoax> ?
<[Kid]> hardware is plenty
<[Kid]> 10+ cores and 96GB+ of RAM
<[Kid]> no extra constraints on model or deployment
<mup> Bug #1730751 opened: LACP rate fast by default <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1730751>
<roaksoax> [Kid]: are they in the same "zone " ?
<[Kid]> yes, they are
<[Kid]> i just have the default zone
<roaksoax> [Kid]: how are youjuju deploy, what if you deploy mahcines with maas directly via the api or cli
<[Kid]> that works.
<[Kid]> it is something between juju and maas
<roaksoax> [Kid]: enable juju debug logging and see what juju is doing
<[Kid]> what is that?
<[Kid]> juju debug-log?
<[Kid]> that just shows it
<roaksoax> [Kid]: yeah, see what requests juju makes and what's going om
<[Kid]> ahh i gotcha.
<[Kid]> i will try that
<[Kid]> thankas
#maas 2017-11-08
<mup> Bug #1730783 opened: maas 2.2 api returning malformed data <MAAS:New> <https://launchpad.net/bugs/1730783>
<juanquy> hello there, guys I'm having issues with pxe booting , the box start the boot sec perfect Ubuntu 16.04 loading, I can even see the new node added on my MAAS dash but after a few sec the box shutdown
<juanquy> any ideas please?
<juanquy> hello
<mup> Bug #1730799 opened: [2.3] Traceback when viewing controller commissioning scripts <MAAS:In Progress by ltrager> <https://launchpad.net/bugs/1730799>
<mup> Bug #1730916 opened: Intermittent failure: snippets.tests.test_maas_run_remote_scripts.TestRunScripts.test_run_scripts  <MAAS:New> <https://launchpad.net/bugs/1730916>
<mup> Bug #1730916 changed: Intermittent failure: snippets.tests.test_maas_run_remote_scripts.TestRunScripts.test_run_scripts  <MAAS:New> <https://launchpad.net/bugs/1730916>
<mup> Bug #1730916 opened: Intermittent failure: snippets.tests.test_maas_run_remote_scripts.TestRunScripts.test_run_scripts  <MAAS:New> <https://launchpad.net/bugs/1730916>
<the_questioner> hello maas channel on irc
<the_questioner> i have never used maas before but it seems to be exactly what i need
<the_questioner> i have a few questions on the subject
<the_questioner> i have a firewall ssytem that serves dhcp on the network, also handles port forwarding to my current webserver
<the_questioner> would this be an issue for a maas setup
<the_questioner> also how the pods work. are the machines intalled from the shared pool or resources, shared resources across multiple systems? or are they single physical systems that are just added together in the pool.
<externalreality> Hello, I followed some web instructions I found for setting up a VMAAS. The worked for the most part, however, commissioning nodes time out after 20 minutes according to the maas logs.
<externalreality> The MAAS is able to power on the node and begin commissioning but always fails with a timeout.
<mup> Bug #1730953 opened: [2.3rc2, UI] The arrow in the  domain dropdown is covered by the text <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730953>
<mup> Bug #1730955 opened: [2.3rc2, UI] The Take action dropdown should follow the Vanilla padding guidelines <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730955>
<mup> Bug #1730955 changed: [2.3rc2, UI] The Take action dropdown should follow the Vanilla padding guidelines <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730955>
<mup> Bug # opened: 1730955, 1730960, 1730965, 1730967
<mup> Bug #1730968 opened: [2.3rc2, UI] The cards should have the available metris even if the metrics don't have any data <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730968>
<mup> Bug #1730969 opened: [2.3rc2, UI] Machine details secondary nav hidden at medium viewport <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730969>
<mup> Bug #1730972 opened: [2.3qa, UI] When the contextual menu is open and I scroll down the menu is still open and sits on top of the header <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730972>
<mup> Bug #1730972 changed: [2.3qa, UI] When the contextual menu is open and I scroll down the menu is still open and sits on top of the header <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730972>
<mup> Bug #1730972 opened: [2.3qa, UI] When the contextual menu is open and I scroll down the menu is still open and sits on top of the header <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730972>
<mup> Bug #1730973 opened: [2.3rc2, UX improvement]  "See results" links should link to specific results row <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730973>
<mup> Bug #1730976 opened: [2.3rc2] The Volume groups have health information  <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730976>
<mup> Bug #1730977 opened: [2.3rc2, UI] When removing a Volume group and RAIDs the menu item says remove disk which might be confusing  <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730977>
<mup> Bug #1730979 opened: [2.3rc2, UI] In the Settings Section when I click Edit the edit button remains on the screen <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730979>
<mup> Bug #1730977 changed: [2.3rc2, UI] When removing a Volume group and RAIDs the menu item says remove disk which might be confusing  <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730977>
<mup> Bug #1730979 changed: [2.3rc2, UI] In the Settings Section when I click Edit the edit button remains on the screen <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730979>
<mup> Bug #1730977 opened: [2.3rc2, UI] When removing a Volume group and RAIDs the menu item says remove disk which might be confusing  <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730977>
<mup> Bug #1730979 opened: [2.3rc2, UI] In the Settings Section when I click Edit the edit button remains on the screen <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730979>
<mup> Bug #1730983 opened: [2.3rc2, UI] In the storage card the type iof disk should come before the number and size <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730983>
<mup> Bug #1730983 changed: [2.3rc2, UI] In the storage card the type iof disk should come before the number and size <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730983>
<mup> Bug #1730983 opened: [2.3rc2, UI] In the storage card the type iof disk should come before the number and size <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730983>
<mup> Bug #1730985 opened: [2.3rc2, UI] When failed testing has been overwritten for a machine use the notification pattern to notify the user   <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1730985>
<mup> Bug #1730991 opened: Bond mode active/backup garp config <canonical-bootstack> <internal> <MAAS:Triaged> <https://launchpad.net/bugs/1730991>
<mup> Bug #1731009 opened: [2.3rc2, UI] The column spacing in the storage tables needs to be specified <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731009>
<mup> Bug #1731009 changed: [2.3rc2, UI] The column spacing in the storage tables needs to be specified <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731009>
<mup> Bug #1731009 opened: [2.3rc2, UI] The column spacing in the storage tables needs to be specified <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731009>
<mup> Bug #1731011 opened: [2.3rc2, UI] The storage tags link you to the network discovery page <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731011>
<mup> Bug #1731011 changed: [2.3rc2, UI] The storage tags link you to the network discovery page <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731011>
<mup> Bug #1731011 opened: [2.3rc2, UI] The storage tags link you to the network discovery page <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731011>
<mup> Bug #1731017 opened: [2.3rc2, UI] When a property is Unconfigured it should be consistently presented in the UI <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731017>
<dfin> Using MAAS I'm able to half-commission a machine on a different subnet that the one that the rack controller is sitting on (same VLAN in MAAS). The PXE process seems to go fine until I get to cloud-init, where I get a "Request Time Out" when talking to the MAAS server (attempting to the the preseed via http). Is there a limitation where there needs to be a Rack Controller in the subnet that is being managed?
<dfin> Also, it seems like SSH is misconfigured after the failed preseed, so I cannot get into the box for debugging
<roaksoax> dfin: check /etc/maas/rackd.conf doesn't point to localhost and points to the right IP the machine can communicate with
<kukacz> hi, is there a way, how to tell MAAS to use a specific block device (not the first one) as root device yet before commissioning starts?
<roaksoax> kukacz: no it is not unfortunately. May I ask what are you trying to achieve?
<kukacz> roaksoax: sure, I'm setting up a series of servers which have the system drive presented as the last one. I'm using CLI for that
<kukacz> anytime I create a machine, commissioning starts using the defaults values
<kukacz> then I need to go and delete all the LVs, VGs, partitions and set the correct drive and re-run commissioning
<kukacz> I'd like to have the work done without these unnecessary steps
<roaksoax> kukacz: why do you need to re-run commissioning ? it doesn't seem like you need to
<roaksoax> kukacz: commissioning will only discover disks. After that the machine is ready and you can configure your storage configuration
<roaksoax> kukacz: if you want to re-run commissioning and want it to not mess with your storage configuration, there's an option for that which will prevent MAAS from overwriting your storage config
<kukacz> ok, that was my wrong understanding then
<kukacz> but how about those volumes placed on wrong drive?
<roaksoax> kukacz: so this is what I did. 1. added a new machine, which was commissioned. After successfully doing that, the machine was in 'Ready' state
<roaksoax> 2. the machine listed a root disk mounted on /, after which I removed it, and configured LVM on it
<roaksoax> so at this point I have a machine in ready state with LVM configured as I need to
<roaksoax> if I deploy the machine, it works just fine
<roaksoax> I then release it, it goes back into 'ready', and the lvm configuration remains the same
<roaksoax> kukacz: so, now, I 'commission' the machine *again*, and commissionning resets my LVM configurate
<roaksoax> kukacz: so, now, I 'commission' the machine *again*, and commissionning resets my LVM configuration
<roaksoax> kukacz: which is correct approach
<roaksoax> kukacz: however, if I were to have selected "retain storage configuration" option, it wouldn't have reset the storage config. It would have kept it
<kukacz> roaksoax: well, I'm defining a procedure for a group of operators and writing the docs. it seems really strange to define precisely all the steps to delete all objects which were created uselessly by a step (commissioning) which can not be prevented from run before correct parameters are set
<kukacz> it would be better then if the comissioning had not created any disk layout. we would saved 3 steps at minimum
<roaksoax> kukacz: maas just works the way it has worked way before the ability to define custom storage, this is what most users of maas use by default, hence maas takes a simple default
<roaksoax> if you want to change the default, they you have the flexibility to do so when the machine is 'ready'
<kukacz> hmm, perhaps we are a rare usecase, when using multiple disk drives with system volume being presented the last. it's common Dell server though
<roaksoax> kukacz: you mean, the bios has the last disk as the disk that the machine will boot from ?
<kukacz> roaksoax: yes
<kukacz> roaksoax: that's because there are non-raid and raid-disks used at the same time. while the raid-disk (used for OS install) is presented last
<mup> Bug #1722578 changed: Controller <> is running an older version of MAAS (less than 2.3.0) <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1722578>
<mup> Bug #1731059 opened: [2.3, HWTv2] Machines changing states due to aborted operations may leave machines with 'pending' tests <MAAS:New> <https://launchpad.net/bugs/1731059>
<mup> Bug #1731059 changed: [2.3, HWTv2] Machines changing states due to aborted operations may leave machines with 'pending' tests <MAAS:New> <https://launchpad.net/bugs/1731059>
<mup> Bug #1731059 opened: [2.3, HWTv2] Machines changing states due to aborted operations may leave machines with 'pending' tests <MAAS:New> <https://launchpad.net/bugs/1731059>
<mup> Bug #1731062 opened: [2.3, HWTv2] Incorrect overall status messages about failed tests <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1731062>
<shadoxx> Hmm, running into a weird issue with Pods.
<shadoxx> If I compose a machine, and leave it in the default zone, with the default domain (maas), the machine will comission fine. However, if I change either the domain or zone its in, it fails to PXE boot
#maas 2017-11-09
<mup> Bug #1731075 opened: [2.3, HWTv2] Rogue test results when machine fails to commission for the first time <MAAS:Triaged> <https://launchpad.net/bugs/1731075>
<shadoxx> Narrowed it down. I can't get the cloud-init to succeed. The MaaS boot keeps saying No datasource found
<mup> Bug #1731162 opened: Unable to deploy Artful on maas 2.2.1 <MAAS:New> <https://launchpad.net/bugs/1731162>
<mup> Bug #1731162 changed: Unable to deploy Artful on maas 2.2.1 <MAAS:New> <https://launchpad.net/bugs/1731162>
<mup> Bug #1731162 opened: Unable to deploy Artful on maas 2.2.1 <MAAS:New> <https://launchpad.net/bugs/1731162>
<mup> Bug #1731188 opened: [2.3rc2, UI] When a test has failed the expanded panel should have information about the failure (instead of metric info) <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731188>
<mup> Bug #1731191 opened: [2.3rc2, UI] The padding of the icons in hardware testing is not right <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731191>
<mup> Bug #1731188 changed: [2.3rc2, UI] When a test has failed the expanded panel should have information about the failure (instead of metric info) <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731188>
<mup> Bug #1731191 changed: [2.3rc2, UI] The padding of the icons in hardware testing is not right <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731191>
<mup> Bug #1731188 opened: [2.3rc2, UI] When a test has failed the expanded panel should have information about the failure (instead of metric info) <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731188>
<mup> Bug #1731191 opened: [2.3rc2, UI] The padding of the icons in hardware testing is not right <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731191>
<mup> Bug #1731206 opened: [2.3rc2]  The hardware tests list doesn't update without a refresh when running tests again <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731206>
<kukacz> hi, can a storage layout be applied on a "Ready" machine without re-running commissioning?
<kukacz> forgot to say I'm thinking of CLI
<kukacz> just found it: maas ${PROFILE} machine set-storage-layout ${SYSTEM_ID} storage_layout=lvm
<kukacz> solved
<mup> Bug #1731210 opened: [2.3rc2, UI] The padding of the metrics/ error text in the expanded hardware testing row is smaller than guidelines <2.3qa> <ui> <MAAS:New for deadlight> <https://launchpad.net/bugs/1731210>
<mup> Bug #1731214 opened: [2.3rc2, UI] Consider removing the day from the standard date format <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731214>
<mup> Bug #1731214 changed: [2.3rc2, UI] Consider removing the day from the standard date format <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731214>
<mup> Bug #1731214 opened: [2.3rc2, UI] Consider removing the day from the standard date format <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731214>
<mup> Bug #1731243 opened: [2.3rc2, UI] I clicked map subnet (while Network discovery was disabled) and I didn't get a warning or information message <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731243>
<mup> Bug #1731244 opened: - [ ] [2.3rc2] Subnet details; the Save button doesn't save my changes or change the state of the section from Edit mode to Read mode <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731244>
<mup> Bug #1731243 changed: [2.3rc2, UI] I clicked map subnet (while Network discovery was disabled) and I didn't get a warning or information message <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731243>
<mup> Bug #1731244 changed: - [ ] [2.3rc2] Subnet details; the Save button doesn't save my changes or change the state of the section from Edit mode to Read mode <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731244>
<mup> Bug #1731243 opened: [2.3rc2, UI] I clicked map subnet (while Network discovery was disabled) and I didn't get a warning or information message <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731243>
<mup> Bug #1731244 opened: - [ ] [2.3rc2] Subnet details; the Save button doesn't save my changes or change the state of the section from Edit mode to Read mode <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731244>
<mup> Bug #1731162 changed: Unable to deploy Artful on maas 2.2.1 <MAAS:Invalid> <https://launchpad.net/bugs/1731162>
<mup> Bug #1731162 opened: Unable to deploy Artful on maas 2.2.1 <MAAS:Invalid> <https://launchpad.net/bugs/1731162>
<mup> Bug #1731162 changed: Unable to deploy Artful on maas 2.2.1 <MAAS:Invalid> <https://launchpad.net/bugs/1731162>
<mup> Bug #1731243 changed: [2.3rc2, UI] I clicked map subnet doesn't work (while Network discovery was disabled) <2.3qa> <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1731243>
<mup> Bug #1731244 changed: [2.3rc2] Save button doesn't work in Subnet Details page. <2.3qa> <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1731244>
<mup> Bug #1731243 opened: [2.3rc2, UI] I clicked map subnet doesn't work (while Network discovery was disabled) <2.3qa> <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1731243>
<mup> Bug #1731244 opened: [2.3rc2] Save button doesn't work in Subnet Details page. <2.3qa> <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1731244>
<mup> Bug # changed: 1722903, 1727548, 1731243, 1731244
<mup> Bug #1731284 opened: [2.x, UI] Subnet details page takes a long too lon when multiple IP addressed (observed) <performance> <ui> <MAAS:Triaged> <https://launchpad.net/bugs/1731284>
<mup> Bug #1731292 opened: [2.3, UI, regression] Hardware testing / commissioning table doesn't fit in a small screen but there's a lot of whitespace <MAAS:Triaged> <https://launchpad.net/bugs/1731292>
<mup> Bug #1731293 opened: [2.3rc2, UI] Previous results should be sorted by most recent first <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731293>
<mup> Bug #1731292 changed: [2.3, UI, regression] Hardware testing / commissioning table doesn't fit in a small screen but there's a lot of whitespace <MAAS:Triaged> <https://launchpad.net/bugs/1731292>
<mup> Bug #1731293 changed: [2.3rc2, UI] Previous results should be sorted by most recent first <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731293>
<mup> Bug #1731292 opened: [2.3, UI, regression] Hardware testing / commissioning table doesn't fit in a small screen but there's a lot of whitespace <MAAS:Triaged> <https://launchpad.net/bugs/1731292>
<mup> Bug #1731293 opened: [2.3rc2, UI] Previous results should be sorted by most recent first <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731293>
<mup> Bug #1731298 opened: [2.3rc2, UI] When I see the log of a test and I click on Back to machine details, back doesn't take me to the tab I came from <2.3qa> <ui> <MAAS:New> <https://launchpad.net/bugs/1731298>
<mup> Bug #1731350 opened: [2.3, HWTv2, UI] Aborting commissioning (+ testing) of a machine never commissioned before, leaves 'pending' icons in the UI <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1731350>
<mup> Bug #1731353 opened: [2.3, HWTv2] Re-running commissioning + testing of a NEW machine that was previously manually aborted results on tests not run <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1731353>
<mup> Bug #1731353 changed: [2.3, HWTv2] Re-running commissioning + testing of a NEW machine that was previously manually aborted results on tests not run <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1731353>
<mup> Bug #1731353 opened: [2.3, HWTv2] Re-running commissioning + testing of a NEW machine that was previously manually aborted results on tests not run <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1731353>
#maas 2017-11-10
<jac_cplane> I'm running into bug 1628337 - cloud init install NTP before configuring archives.   I see from github that it was fixed 20 days ago.  can someone tell me how I can get this fix?   i'm running MAAS 2.1.5 - I deleted my current xenial image and downloaded it again via maas.
<jac_cplane> which didn't fix it
<jac_cplane> do I need to edit MAAS config ?
<jac_cplane> I also did a apt-get update on my maas server
<jac_cplane> trusty deploys fine but xenial fails
<jac_cplane> Do I need to edit cloudinit in maas server /usr/lib/python3/dist-packages/cloudinit/config/cc_ntp.py and patch myself?
<jac_cplane> please disregard.  figured it out.
<roaksoax> jac_cplane: what was it ?
<jac_cplane> apt-get update followed by apt-get upgrade
<jac_cplane> coming from rh/centos  yum update does both.
<mup> Bug #1731405 opened: disk names in gui don't transfer through to node device names <MAAS:New> <https://launchpad.net/bugs/1731405>
<mup> Bug # changed: 1662343, 1721825, 1722646, 1723425, 1728304, 1730474, 1730481, 1730485, 1730703, 1730799, 1731075, 1731292, 1731350
<mup> Bug #1721743 changed: [2.3b2] Rack and region controller versions still not updated <MAAS:Fix Released> <https://launchpad.net/bugs/1721743>
<mup> Bug #1731490 opened: Can't boot MAAS with a UEFI Node and a Ubuntu custom storage layout <MAAS:New> <https://launchpad.net/bugs/1731490>
<jamesbenson> roaksoax: ping
<mup> Bug #1618466 changed: [2.1] Fail to add PPA behind maas-proxy <MAAS:Triaged> <https://launchpad.net/bugs/1618466>
<pzingg> Newbie, working my way through first deployment. Got node through commissioning but getting "No filesystems." and "No available disks or partitions."
<pzingg> I can see the disk in the "07-block-devices" output. Is it because I already have Ubuntu installed on these disks/partitions?
<pzingg> Can't find any documentation on this.
<shadoxx_> pzingg: more than likely, yes, that's why it's failing
<shadoxx_> ideally the node you have will have blank disks when commissioning and deployed
<pzingg> I'm going to see if there's a way to either release a yet-to-be-deployed node, or command line into it to run gparted and blow away the partitions.
<shadoxx_> pzingg: generally i just used gparted or do something fun like dd if=/dev/zero of=/dev/drive bs=512 count=8
#maas 2017-11-11
<mup> Bug #1713695 changed: QueryDict error when creating an iprange via the API <MAAS:Expired> <https://launchpad.net/bugs/1713695>
<mup> Bug #1716494 changed: Admin certificate upload <MAAS:Expired> <https://launchpad.net/bugs/1716494>
<mup> Bug #1713695 opened: QueryDict error when creating an iprange via the API <MAAS:Expired> <https://launchpad.net/bugs/1713695>
<mup> Bug #1716494 opened: Admin certificate upload <MAAS:Expired> <https://launchpad.net/bugs/1716494>
<mup> Bug #1713695 changed: QueryDict error when creating an iprange via the API <MAAS:Expired> <https://launchpad.net/bugs/1713695>
<mup> Bug #1716494 changed: Admin certificate upload <MAAS:Expired> <https://launchpad.net/bugs/1716494>
<if_gaga1> hello guys,i intalled maas on server with 2 network interfaces (eth0, eth1, for example), how i can bind all MAAS *network activity* to eth1
<if_gaga1> ,
<if_gaga1> ?
<jac_cplane> I just upgraded to maas 2.2.2  and it seem custom images are no longer supported through this command.    maas admin boot-resource create name=custom/centos7-netfix title="Centos 7 - netfix" architecture=amd64/generic content@=centos7-root-netfix.tgz
<jac_cplane> can any help
<jac_cplane> Hi narinder
<jac_cplane> are you on?
<jac_cplane> I just upgraded to maas 2.2.2  and it seem custom images are no longer supported through this command.    maas admin boot-resource create name=custom/centos7-netfix title="Centos 7 - netfix" architecture=amd64/generic content@=centos7-root-netfix.tgz
<jac_cplane> was is the latest way to install a custom image into maas
<jac_cplane> anyone?
<jac_cplane> how can i import a custom image into maas 2.2.2 since boot-resource create is no longer supported
<jac_cplane> is read now the replacement command for create
<jac_cplane> it seems that create is no longer available but read or delete is
<jac_cplane> any place this is documented
<mup> Bug #1731671 opened: Commissioning Composable Machine Fails <MAAS:New> <https://launchpad.net/bugs/1731671>
#maas 2017-11-12
<jac_cplane> custom image no longer boots in 2.2.2 - works in 2.2.0
<jac_cplane> i have two maas servers side by side.  one is 2.2.2 and one is 2.2.0.     I load the same custom image into both.   I can boot fine with 2.2.0 but get a image-not-found in 2.2.2
<jac_cplane> any help?
<roaksoax> jac_cplane: boot-resources instead of boot-resource
<roaksoax> jac_cplane: and check that your machine doesn't have a minimum kernel set that's no longer available
<jac_cplane> yes I figured out the typo
<jac_cplane> where can I check min kernel
<jac_cplane> no minimum kernel is selected
<jac_cplane> i'm filling out a bug-report now
<roaksoax> jac_cplane: two places, 1, the machine itsself, 2 the settings page
<jac_cplane> please see Bug #1731709 r
<jac_cplane> setting page is set to no min kernel
<jac_cplane> nothing on the machine
<jac_cplane> custom image works on 2.2.0  but not on 2.2.2
<roaksoax> jac_cplane: i dont think this is an issue with the custom iamge. MAAS doens't pxe boot cusotm images. MAAS boots into ubuntu to deploy custom images
<roaksoax> jac_cplane: i've seen that issue before i can't remember when
<roaksoax> jac_cplane: are there no logs in maas.log or rackd.log that show why it failed to boot
<roaksoax> ?
<roaksoax> i think that what it shows is just a side effect
<jac_cplane> i cannot see anhything in the logs
<jac_cplane> I do this this error in maas.log
<jac_cplane> Nov 11 16:44:10 MAAS maas.preseed: [info] MACH2: custom network and storage options are only supported on Ubuntu. Using flat storage layout.
<roaksoax> that's not related though
<jac_cplane> yes - i agree
<jac_cplane> I have a 2.2.0 side by side with 2.2.2 - works fine with 2.2.0
<jac_cplane> cannot find any other error in the logs.  but the boot console does show the error.  in the but report
<jac_cplane> this appears after the error in maas.log
<jac_cplane> Nov 11 13:46:56 MAAS maas.node: [error] MACH2: Marking node failed: Installation failed (refer to the installation log for more information).
<jac_cplane> not sure where installation log is
<jac_cplane> found it.
<jac_cplane> Traceback (most recent call last):
<jac_cplane>   File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 595, in <module>
<jac_cplane>     main()
<jac_cplane>   File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 537, in main
<jac_cplane>     curthooks.write_files(cfg, target)
<jac_cplane> AttributeError: module 'curtin.commands.curthooks' has no attribute 'write_files'
<jac_cplane> Unexpected error while running command.
<jac_cplane> this is the error
<jac_cplane>   File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 595, in <module>
<mup> Bug #1731709 opened: centos custom images won't boot MAAS 2.2.2 <MAAS:Incomplete> <https://launchpad.net/bugs/1731709>
<roaksoax> jac_cplane: i've asked more questions on the bug
<roaksoax> jac_cplane: when you provide all the info please mark it as 'new' again
<roaksoax> jac_cplane: also, I think this is an issue with curtin, maybe they regressed somethign
<roaksoax> jac_cplane: it seems it has, how did you create that custom image ?
<jac_cplane> thanks  i will provide more details
<jac_cplane> dont remember how we created the image.  it was created when we had maas 2.0
<jac_cplane> can anyone help with Bug #1731709
<jac_cplane>   AttributeError: module 'curtin.commands.curthooks' has no attribute 'write_files'
<jac_cplane> File "/tmp/tmpngh0fprp/target/curtin/curtin-hooks.py", line 537, in main
<jac_cplane> please see https://bugs.launchpad.net/ubuntu/+source/curtin/+bug/1731805
<jac_cplane> and  https://bugs.launchpad.net/maas/+bug/1731709
<jac_cplane> who can help?
<roaksoax> jac_cplane: where did you get that custom image?
<jac_cplane> it was made from instructions from canonical
<jac_cplane> the image is not the problem
<jac_cplane> the image works with curtin 505 but not on curtin 532
<roaksoax> jac_cplane: i see that now, but i need to understand where that image came from
<roaksoax> jac_cplane: what version of the inage generator is the one you used to create the new image
<jac_cplane> from our own repo.  I can put the instructions we used in the maas bug
<roaksoax> ?
<jac_cplane> maas 2.1
<jac_cplane> it was created using maas 2.1
<roaksoax> jac_cplane: right but did you use maas inage builder with instructions provided by a support engineer or did you manaully build that image yourself?
<jac_cplane> the image was done based on this bug https://bugs.launchpad.net/juju/+bug/1646847
<jac_cplane>     wget https://people.canonical.com/~iatrou/curtin-hooks-20170110.py.gz -O - | gunzip > curtin-hooks.py
<jac_cplane>     wget http://images.maas.io/ephemeral-v2/daily/centos70/amd64/20160801_01/root-tgz -O centos7-root.tgz
<jac_cplane>     mkdir centos7-root && cd centos7-root
<jac_cplane>     sudo tar -zxvf ../centos7-root.tgz
<jac_cplane>     sudo cp ../curtin-hooks.py curtin/
<jac_cplane>     sudo chown root: curtin/curtin-hooks.py
<jac_cplane>     sudo chmod 755 curtin/curtin-hooks.py
<jac_cplane>     sudo tar -zcvf ../centos7-root-netfix.tgz .
<jac_cplane> we created the image on the instruction from a canonical support engineer.  dont rmembe who.
<roaksoax> jac_cplane: right so it seems you created an image from an unsupported path, which is a problem, that said curtin seem to have changed which is whats brraking you
<roaksoax> jac_cplane: the wuickest fix is to change your image to ise futil from where it has been moved rather than where it is being imported
<jac_cplane> those instructions were given to us by Michael Iatrou
<jac_cplane> can you repeat your last comment. I don't understand
<roaksoax> jac_cplane: please paste those instructions in yhe bug and ill deal with it tomorrow
<jac_cplane> ok
<jac_cplane> thanks
<jac_cplane> done
